Cron Job is Getting Killed - Inconsistent Execution
I have a cron job that runs every 24 hours and scrapes information from 5 different websites and stores it in a database.
The cron job is really inconsistent - one week it would work without any issues. The cron job would execute and complete its task without any errors.
Other weeks, the cron job would almost fail immediately or after 20 minutes into scraping the data. I get no errors, just an email saying that the cron job was started.
When I try to run the script from PuTTY, it works but gets killed a couple of times. So I have to re-do the operations.
What could be the reason for this inconsistency? 1 week the cron job runs without any issues, the 2nd week cron job fails almost every single day after 10-20 minutes + it gets killed on PuTTY automatically.
The scripts are really simple.
Get project IDs and Website addresses from the database.
Give the website addresses to a function
- Get data from the website
- Clean the data
- Open database connection
- Insert new data
- Close database connection.
And then the next website address is passed. Rinse and repeat.
So, any advice on how to solve the issue of inconsistent cron jobs?
P.S. Website is built using the Codeigniter MVP framework.