Database lock pile-up
Healthy web servers, a latency graph going vertical, and queries waiting on each other.
Last updated:
The incident that punishes pattern-matching. Everything on the rack looks fine, the traffic bar is normal, and the night is still falling apart.
What it does
Queries on a database host start waiting on locks instead of completing. Nothing crashes and no machine saturates — but requests queue behind the database, so latency climbs and uptime follows it down.
The web tier is healthy throughout. That is the whole character of this incident: the symptom is fleet-wide slowness and the cause is on one host that never turns red.
The log signature
Query ids, wait times in the tens of seconds, and the phrase LOCK WAIT with a
transaction name:
04:03:11 [db-01] slow query #9412: 34000ms — LOCK WAIT (checkout_tx)
04:03:40 [db-01] 62% of connections waiting on locks
04:04:02 average latency 2100ms — above target
db_stat lists active queries with their ids, wait times
and names, and marks the ones stuck on locks. It is the only command that can
see inside this, which makes it the fastest way to confirm or rule the incident
out.
If you leave it alone
The lock queue grows and latency stays elevated for the rest of the shift.
Because no machine goes DOWN, the uptime loss arrives entirely through latency
and then through SLA debt. It is
possible to lose a night to this with a fully green rack.
Telling it apart
| Looks similar | What separates it |
|---|---|
| DDoS flood | Both raise latency. A flood also raises the bandwidth bar and CPU on web machines; a lock pile-up raises neither. |
| Runaway job | Both can involve a database host, but a runaway pins CPU and names a process; this names queries. |
When it shows up
| Earliest night | 3 |
| Needs | a db machine |
| Weight | 80 |
| At once | one at a time |
There is also a partial way out that resolves it at a real uptime cost — the closing log line distinguishes it from the clean one.