Certificate expiry

The smallest incident in the game and the one most often lost to inattention. It announces itself once, clearly, and then does nothing for a while.

What it does

It arrives in two phases. On injection, a single warning line says the load balancer's TLS certificate is about to expire — and nothing else happens. No metric moves, no bar changes colour, no machine changes status.

After the grace period the certificate expires and two things happen at once: legitimate traffic drops because clients can no longer connect, and the retry storm from those clients burns CPU on the load balancers until they reach CRIT. Uptime falls from both directions.

The log signature

The [lb] host and the word cert. The first line is the one that matters:

00:14:22 [lb-01] TLS handshake failures ↑ — cert for lb-01 expires soon
00:19:40 [lb-01] TLS cert expired — traffic dropping

There is no diagnostic command that surfaces a pending expiry. The warning goes past in the log stream and that is your only notice, which is the design: this incident tests whether you are reading, not whether you can investigate.

If you leave it alone

The impact persists for the rest of the shift. The warning does not repeat. This is the one incident where the gap between "resolved in time" and "noticed too late" is a single line of text you either read or did not.

Telling it apart

Nothing else in the game produces TLS handshake failures, and nothing else warns you before it does damage. If you see cert in a log line, there is no diagnostic work to do — only a decision about when to act.

When it shows up

The remedy is the cheapest paid action in the game and takes ten seconds to take effect, so the resolution is not instant even once you have acted.