Skip to content
Play

Log signatures

The diagnostic lines, grouped by subsystem tag, with the shape each one takes.

Last updated:

Every log line is templated, which means the shapes are stable and learnable. This page lists the diagnostic lines — the ones that tell you something is wrong. Confirmation lines that appear after you act are left out on purpose.

Reading a line

03:11:42 [fw]     203.0.113.7 — 4200 req/s (UA: "-")
│        │        └─ the payload: what is unusual
│        └─ subsystem tag, or a hostname speaking for itself
└─ in-game timestamp

Subsystem tags

TagWho is speaking
[fw]The firewall — traffic and addresses
[hw]Hardware — temperature and fans
[cron]The job scheduler
[db-01] and other hostnamesA machine reporting about itself
Deniz:The mentor, on night one and at the start of each campaign night

Network

LineMeans
{ip} — {rps} req/s (UA: "-")An address at an implausible rate with no user agent
bandwidth at {pct}% of capacityThe fleet is approaching its traffic ceiling
upstream {host} marked unhealthyThe load balancer has stopped trusting a backend
{host} not responding — request timed outA ping that got nothing back

Metrics

LineMeans
{host} cpu saturation warning ({pct}%)CPU past its warning threshold
{host} memory pressure ({pct}%) — oom killer armedRAM near the ceiling; processes are about to be taken
{host} disk {pct}% — /var/logA partition filling, almost always with logs
{host} disk full — writes failingPast 100%; the machine can no longer write
{host} temp {temp}°C — fan RPM maxHeat with cooling already at maximum
{host} thermal shutdown at {temp}°C95 °C reached; the machine is DOWN for a minute
average latency {ms}ms — above targetFleet-wide slowness, cause unstated

Processes and jobs

LineMeans
{name} (PID {pid}) RSS climbing — RAM {pct}%A leak, named and located
{name} (PID {pid}) at {pct}% cpuA process holding the CPU
oom killer took PID {pid} ({name})Memory ran out; the kernel chose a victim
{name} startedA scheduled job beginning — innocuous unless CPU follows

Database

LineMeans
{host} slow query #{id}: {ms}ms — LOCK WAIT ({name})A query blocked on a lock, with its id
{pct}% of connections waiting on locksThe pile-up has spread across the pool

TLS

LineMeans
TLS handshake failures ↑ — cert for {host} expires soonThe one and only early warning
TLS cert expired — traffic droppingToo late; clients can no longer connect

Related