Certificate expiry

The smallest incident in the game, and the one most often claimed by inattention. It announces itself once, plainly, and then does nothing for a while.

What it does

It arrives in two stages. When injected, a single warning line says the load balancer's TLS certificate is about to expire — and nothing else happens. No metric moves, no bar changes colour, no machine changes state.

Once the grace period runs out, the certificate expires and two things happen at once: legitimate traffic drops, because clients can no longer connect; and those clients' retry storm burns the load balancers' CPU until it reaches CRIT. Uptime falls from both directions at once.

The log signature

First a single warning from monit on the load balancer; once the certificate has expired, handshake errors land in the nginx error log:

00:14:22 lb-01 monit[540]: 'shop-tls' certificate expiry test failed -- certificate expires in less than 1 day
00:19:40 lb-01 nginx: [crit] SSL_do_handshake() failed (SSL: error:0A000415:SSL routines::sslv3 alert certificate expired:SSL alert number 45) while SSL handshaking, client: 198.51.100.23, server: 0.0.0.0:443

The warning slips past in the log stream and does not repeat. You can read the certificate's real expiry date with openssl x509 -enddate -noout -in <cert>; the openssl command shows that date exactly as it is in the certificate.

If you leave it alone

The impact lasts for the rest of the shift. The warning does not repeat. This is the only incident where the difference between "resolved in time" and "noticed far too late" is a single line of text you either read or did not.

Telling it apart

Nothing else in the game produces a TLS handshake error, and nothing else warns you before it does harm. If you see certificate in a log line, the diagnosis is short; the real question is when you take the warning seriously.

When it shows up

The intervention does not show its effect immediately; even after you act, the fix is not instant.