Memory leak
One process on one host whose resident memory only ever goes up.
Last updated:
A leak is the quietest incident in the catalogue. Nothing spikes, nothing floods — a single process on a single host simply stops giving memory back.
What it does
RAM on one machine climbs over minutes rather than seconds. When it reaches the
ceiling the OOM killer starts taking processes, which is a service interruption
in itself. Once there is nothing left to take, the machine goes DOWN.
The important detail is that this is local. It affects one host, and it affects RAM — not bandwidth, not disk, not the fleet.
The log signature
A named process, a PID, and a resident set that is described as climbing:
02:41:08 [web-02] api-worker (PID 4312) RSS climbing — RAM 61%
02:43:52 [web-02] memory pressure (88%) — oom killer armed
02:45:10 [web-02] oom killer took PID 4390 (nginx)
The oom killer armed line is the last comfortable warning you get.
top on that host names the process and gives you the PID.
If you leave it alone
The OOM killer works through the process list, so you lose function in pieces
before you lose the machine. Because the host reaches CRIT and then DOWN,
the uptime cost arrives through server status rather than as a separate
penalty — plus the SLA debt once the
grace window passes.
Telling it apart
| Looks similar | What separates it |
|---|---|
| DDoS flood | A flood raises CPU and bandwidth across web machines; a leak raises RAM on exactly one. |
| Disk full | Both creep rather than spike, but one is RAM and the other is a mount point. df settles it in one line. |
| Runaway job | Both name a process in top. The runaway pins CPU; the leak grows RSS. |
When it shows up
| Earliest night | 2 |
| Needs | a web machine |
| Weight | 100 — tied for the most likely incident in the game |
| At once | one at a time |