Disk full
A mount point filling up, and writes that start failing at the top.
Last updated:
The most mechanical incident in the game, and the one with the cleanest signature. A disk fills, and at 100% the machine can no longer write.
What it does
Disk usage on one host climbs steadily. It crosses WARN at 90% and CRIT at
98%. At full the machine starts failing writes, and a sustained write failure
takes it DOWN.
There is no cleverness to it — no cascade, no second-order effect elsewhere in the fleet. What makes it dangerous is that it is undramatic enough to scroll past while you are dealing with something louder.
The log signature
A percentage and a path. /var/log is almost always the culprit, which is a
joke at the game's own expense:
01:52:19 [web-02] disk 91% — /var/log
02:07:44 [web-02] disk full — writes failing
df confirms it and shows usage per mount point across the
fleet, so it is one command to go from "something is wrong" to "it is web-02".
If you leave it alone
Writes fail, then the machine goes DOWN, then the SLA debt
starts on top of that. Because the ramp is slow, a disk that reaches 91% during
a busy stretch will often reach 100% during the next one.
Telling it apart
| Looks similar | What separates it |
|---|---|
| Memory leak | Both creep. One names a process and grows RAM; this one names a mount point and grows disk. |
| Overheating | Both are single-host and slow, but heat shows a temperature and a fan line. |
Two ways out, and they are not equal
This is the one incident where the choice of remedy is visibly economic: there is an instant free answer and a permanent paid one, and the closing log line tells you which route you took. Which is which is worth working out yourself — but note that the free route does not raise the ceiling, so the same disk can fill again later in the same night.
When it shows up
| Earliest night | 2 |
| Needs | a web machine |
| Weight | 100 |
| At once | one at a time |