Runaway scheduled job
A backup that starts around three in the morning and pins the CPU until something stops it.
Last updated:
A scheduled job fires, does not finish, and holds the CPU at 100%. It is the one incident that comes back after you deal with it — because dealing with the process is not the same as dealing with the schedule.
What it does
CPU on the target host saturates: WARN, then CRIT, and uptime slides with
it. The job is a named process, visible in top with its
PID, so identification is not the hard part.
The hard part is that the scheduler is still running. Stop the process and the same incident instance starts it again a short while later. It will do that a limited number of times before the job's window closes on its own — so a night spent handling it repeatedly does end, but it ends because the job ran out of patience, not because you solved anything, and the closing log line says so.
The log signature
A job name that sounds like night work, a start line, and CPU pinned right after it:
03:02:15 [cron] backup.sh started
03:02:51 [db-01] backup.sh (PID 2261) at 100% cpu
03:03:40 [db-01] cpu saturation warning (98%)
The timing is part of the signature. This one tends to land in the middle of the shift, around 03:00, which is exactly when a real backup window would open.
If you leave it alone
Sustained CPU saturation on one host and the SLA debt
that comes with an unresolved incident. It is survivable — the host does not go
DOWN on its own — but it bleeds uptime for as long as it runs.
Telling it apart
| Looks similar | What separates it |
|---|---|
| DDoS flood | Both pin CPU. The flood arrives with [fw] traffic lines across web machines; the runaway names one process on one host with no traffic story. |
| Memory leak | Both name a process. Watch which metric moves: CPU here, RAM there. |
When it shows up
| Earliest night | 4 |
| Needs | a db machine |
| Weight | 85 |
| At once | one at a time |