Runaway scheduled job
A runaway cron job in SysAdmin Crisis: a backup starts around three in the morning and pins the CPU at 100% until stopped. Learn the log signature.
Last updated:
A scheduled job fires, does not finish, and holds the CPU at 100%. It is the only incident that comes back after you have dealt with it — because dealing with the process is not the same as dealing with the schedule.
What it does
On the target machine, CPU goes into saturation: WARN, then CRIT, and uptime
slides along with it. The job is a named process and shows up in the
top output with its PID; so the hard part is not the
diagnosis.
The hard part is that the scheduler is still alive. You stop the process and the same incident instance starts it again shortly after. It does this a limited number of times, until the job's own window closes by itself — so the night you spend dealing with it again and again does end, but it ends because the job ran out of patience, not because you solved something; the closing log line says as much.
The log signature
In the cron log a job starts, and then monit shows the same name above the CPU threshold:
03:02:15 db-01 CRON[24310]: (root) CMD (/opt/jobs/backup.sh)
03:02:51 db-01 monit[588]: 'backup.sh' cpu usage of 100% matches resource limit [cpu usage > 80.0%]
03:03:40 db-01 monit[588]: 'db-01' cpu usage of 98% matches resource limit [cpu usage > 90.0%]
Timing is part of the signature. This incident tends to land in the middle of the
shift, around 03:00 — which is exactly when a real backup window would open. You
find the process with top; who started it is written in
ps and the cron records.
If you leave it alone
Sustained CPU saturation on a single machine, and the
SLA debt that an unresolved incident
brings. It is survivable — the machine does not go DOWN on its own — but it
bleeds uptime for as long as it runs.
Telling it apart
| Looks similar | What separates it |
|---|---|
| DDoS flood | Both lock up the CPU. A flood arrives across the web machines with SYN warnings from the kernel; a runaway job names a single process on a single machine with no traffic story. |
| Memory leak | Both name a process. Look at which metric is moving: CPU here, RAM there. |
When it shows up
| Earliest night | 4 |
| Needs | a db machine |
| Weight | 85 |
| At once | one instance |