Memory leak
A memory leak in SysAdmin Crisis: one process on one machine that never gives memory back. Learn its log signature and how to tell it apart.
Last updated:
The leak is the quietest incident in the catalogue. Nothing spikes, nothing overflows — a single process on a single machine just stops giving memory back.
What it does
A machine's RAM climbs over minutes, not seconds. When it hits the ceiling, the
OOM killer starts taking processes, which is itself a service outage. When
there is nothing left to take, the machine goes DOWN.
The important detail is that this is local. It affects one machine and it affects RAM — not bandwidth, not disk, not the fleet.
The log signature
Memory percentage climbs in monit first; when it hits the ceiling, the kernel brings in the OOM killer. The OOM record says who invoked the killer, not the PID of the leaking process:
02:41:08 web-02 monit[601]: 'web-02' mem usage of 80% matches resource limit [mem usage > 75.0%]
02:43:52 web-02 monit[601]: 'api-worker' mem usage of 38% matches resource limit [mem usage > 25.0%]
02:45:10 web-02 kernel: nginx invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
free shows the overall picture, and top shows which
process is growing. Sort the process list by memory use; the leaking process may
not be the one named in the OOM record.
If you leave it alone
The OOM killer works through the process list in order, so you lose function
piece by piece before you lose the machine. Because the machine goes CRIT and
then DOWN, the uptime cost comes not as a separate penalty but through server
state — plus SLA debt once the grace
window passes.
Telling it apart
| Looks similar | What separates it |
|---|---|
| DDoS flood | A flood raises CPU and bandwidth across the web machines; a leak raises RAM on exactly one. |
| Disk full | Neither spikes, both creep; but one is RAM and the other is a mount point. df settles it in one line. |
| Runaway job | Both name a process in the top output. A runaway job locks up the CPU; a leak grows memory. |
When it shows up
| Earliest night | 2 |
| Needs | a web machine |
| Weight | 100 — tied with the most likely incidents in the game |
| At once | one instance |