Reference of the diagnostic log lines in SysAdmin Crisis, grouped by subsystem tag and shown in their fixed formats so you can recognise a problem.
Last updated:
Every log line is templated, so the formats are fixed and learnable. This page
lists the diagnostic lines — the ones that tell you something is wrong. The
confirmation lines that appear after you act are deliberately left out.
How to read a line
Logs take the format a real Linux machine would write: time, hostname, the
service that wrote it, and the message. In the game you read the files under
/var/log with cat, tail, grep, and awk; there is also journalctl.
03:11:42 web-02 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
│ │ │ └─ message: the thing that is out of the ordinary
│ │ └─ the service that wrote it (kernel, monit, nginx, postgres, CRON ...)
│ └─ the machine reporting about itself
└─ in-game timestamp
Who is speaking
Service
What it writes
kernel
Network tables, the OOM killer, temperature and fan alarms, the network card
monit
CPU, memory, and disk thresholds, health checks, certificate expiry
nginx
Configuration and TLS errors, upstream status
postgres
Slow queries, lock waits, connection exhaustion
CRON
Scheduled jobs starting
sshd / sudo
Login attempts and administrator commands
Deniz:
The mentor; on night one and at the start of every campaign night
Network
Line
Meaning
kernel: TCP: request_sock_TCP: Possible SYN flooding on port 443
The number of connections reaching the edge is far above normal
kernel: nf_conntrack: table full, dropping packet
The connection table is full; packets are being dropped
nginx: upstream timed out (110: Connection timed out) while reading response header from upstream
The load balancer is getting no response from a backend
Metrics
Line
Meaning
monit: '{host}' cpu usage of {pct}% matches resource limit
CPU crossed its threshold
monit: '{host}' mem usage of {pct}% matches resource limit
RAM is near the ceiling
monit: 'rootfs' space usage {pct}% matches resource limit
A partition is filling up
rsyslogd: ... No space left on device
The partition is full; writes fail
kernel: CPU0: Core temperature above threshold, cpu clock throttled
The CPU throttled because of heat
kernel: nct6775 ... fan1: alarm, input 0 RPM
A fan stopped
kernel: thermal thermal_zone0: critical temperature reached
Thermal shutdown
monit: 'shop-api' failed protocol test ... response time
Slowness across the fleet; no cause stated
Processes and jobs
Line
Meaning
monit: '{name}' mem usage of {pct}%
A process whose memory is growing
monit: '{name}' cpu usage of {pct}%
A process holding the CPU
kernel: {name} invoked oom-killer
Memory ran out; the name written is whoever invoked the killer, not whoever leaked
CRON: (root) CMD (/opt/jobs/{name})
A scheduled job starting
Database
Line
Meaning
postgres: process N still waiting for ShareLock ... Process holding the lock: M
N is waiting for M's lock
postgres: duration: {ms} ms statement: ...
The duration of the query holding the lock
postgres: FATAL: remaining connection slots are reserved ...
The connection pool is exhausted
TLS
Line
Meaning
monit: 'shop-tls' certificate expiry test failed -- certificate expires in less than 1 day
