Disk full

The most mechanical incident in the game, and the one with the cleanest signature. A disk fills up, and at 100% the machine can no longer write.

What it does

On one machine, disk usage climbs steadily. It crosses WARN at 90% and CRIT at 98%. When it is completely full, the machine starts throwing write errors, and sustained write errors take it DOWN.

There is no trick to it — no cascade, no secondary consequence elsewhere in the fleet. What makes it dangerous is that it is undramatic enough to be read past while you deal with something louder.

The log signature

The first sign is monit's disk warning; once the disk is full, rsyslogd says it cannot write:

01:52:19 web-02 monit[601]: 'rootfs' space usage 91% matches resource limit [space usage > 80.0%]
02:07:44 web-02 rsyslogd[633]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2312.0]
02:07:44 web-02 monit[601]: 'rootfs' space usage 100.0% matches resource limit [space usage > 95.0%]

df shows which mount point has filled up. Who consumed the space is something ls, lsof, and the process list will tell you; it is almost always a log file or a process writing to one.

If you leave it alone

Writes fail, then the machine goes DOWN, then SLA debt piles on top. Because the ramp is slow, a disk that reaches 91% during one busy stretch often reaches 100% by the next.

Telling it apart

The ways out are not equal

On this incident the path you pick also has an economic consequence: some paths are instant and free but do not raise the ceiling, others are permanent but cost money. The closing log line tells you which way you went. If the space looks freed but df does not change, a process may still be holding the file open.

When it shows up