The alert says "API p99 latency above 2 s on api-07". The dashboard shows a CPU graph that looks busy but not alarming. You SSH in at 03:10 and have a few minutes before someone asks what is going on. Rebooting would make the symptom go away and destroy every clue; guessing wastes the time you have.

This guide is the sequence we teach new on-call engineers. It uses tools present on almost every Linux server (the sysstat package adds iostat, mpstat, pidstat and sar), and each step answers one question: is the bottleneck CPU, memory, disk, network, or one process misbehaving? The approach owes a lot to Brendan Gregg's USE method (utilisation, saturation, errors) and the Netflix "60,000 milliseconds" checklist; we have adapted it to what we see most in practice.

Minute 1: load, kernel messages and failed units

uptime
dmesg -T | tail -30
systemctl --failed
journalctl -p err -b --since "-30 min" --no-pager | tail -40

uptime gives three load averages over 1, 5 and 15 minutes. On Linux, load counts tasks that are running, waiting for CPU, and in uninterruptible sleep (usually waiting on disk or NFS). Compare it with the number of CPUs (nproc): a load of 30 on 32 cores is busy, on 4 cores it is a queue. The trend matters more than the value. A 1-minute figure far above the 15-minute one means something started recently.

dmesg catches what applications never log: the OOM killer (Out of memory: Killed process), disk errors (I/O error, dev nvme0n1), dropped packets from a full connection-tracking table (nf_conntrack: table full, dropping packet), and CPU throttling on overheating hardware. On a modern kernel you may need sudo to read it. If you see any of those lines, you may already have your answer.

systemctl --failed and the error-priority journal show crashed services and restart loops, which often explain a latency spike behind a load balancer.

Minute 2: CPU and run queue with vmstat

vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
14  0      0 412300  91204 9120448    0    0     3   180 9120 18233 78 15  7  0  0
16  0      0 410876  91204 9120612    0    0     0   212 9433 19015 81 14  5  0  0

Ignore the first line (it averages since boot) and read the rest:

Column What it tells you
r Tasks running or waiting for CPU. Persistently above the CPU count means CPU saturation
b Tasks blocked in uninterruptible sleep, usually on I/O
si, so Swap in and out. Any sustained non-zero value means memory pressure
us, sy User and kernel CPU time. High sy hints at syscall storms, locking or networking overhead
wa CPU idle while waiting for I/O. A clue, not a measurement of disk load
st Time stolen by the hypervisor. High on noisy or burstable cloud instances

In the example, r around 15 on an 8-core box with us near 80 says "CPU bound, and it is application code". Then find out which process:

mpstat -P ALL 1 3
pidstat 1 5

mpstat shows whether all cores are busy or one core is pinned at 100 percent, the signature of a single-threaded hot loop or one interrupt-heavy NIC queue. pidstat lists CPU per process every second, which is easier to read than a constantly reshuffling top.

Minute 3: memory, and whether it really is short

free -m
cat /proc/pressure/memory

               total        used        free      shared  buff/cache   available
Mem:           15935       13802         402          88        2090        1831
Swap:              0           0           0

Look at available, not free. Linux uses spare memory as page cache, so free is low on any healthy server. available estimates how much can be handed to applications without swapping. When it falls to a few percent of total, the kernel starts reclaiming cache aggressively, disk reads rise, and eventually the OOM killer chooses a victim.

The pressure stall information file (PSI, kernel 4.20 and later) is the clearest single signal:

some avg10=12.40 avg60=8.91 avg300=3.02 total=91828112
full avg10=4.10 avg60=2.77 avg300=0.88 total=21093320

some avg10=12.40 means that in the last ten seconds, at least one task was stalled waiting for memory 12.4 percent of the time. Near zero is healthy. The same files exist for cpu and io under /proc/pressure/, and they answer "is this resource actually making work wait?" directly.

Minute 4: disks, space and inodes

iostat -xz 1 3
df -h
df -i

iostat -x gives per-device statistics; -z hides idle devices. The useful columns:

Column Meaning Worry when
r/s, w/s Read and write operations per second Near the volume's provisioned IOPS
rkB/s, wkB/s Throughput Near the volume's throughput limit
r_await, w_await Average time per request in ms, including queueing Many times higher than normal for the device
aqu-sz Average queue length Consistently well above 1 on a single disk
%util Share of time the device was busy Close to 100 on a single spindle

Be careful with %util on SSDs, NVMe and cloud block storage. Those devices serve many requests in parallel, so 100 percent busy does not mean saturated. Latency (await) and the provider's IOPS limits are better signals.

df -h finds full filesystems. A full /var stops logging and can make databases refuse writes. df -i finds the quieter cousin: a filesystem out of inodes because something wrote millions of tiny files, which gives "No space left on device" while df -h shows free space.

If one process is hammering the disk, pidstat -d 1 shows reads and writes per process.

Minute 5: network and connections

ss -s
ss -tanp state established '( sport = :443 )' | wc -l
sar -n DEV 1 3
sar -n TCP,ETCP 1 3

ss -s summarises sockets by state. Thousands of TIME-WAIT sockets are normal on a busy client. Thousands of CLOSE-WAIT sockets are a bug: the remote side closed and our process never closed its end, which usually means a connection leak in the application.

sar -n DEV shows throughput per interface, so you can compare against the instance's bandwidth limit. sar -n ETCP reports retrans/s, TCP retransmissions per second; a sudden rise points at packet loss between this host and its peers rather than at the host itself.

Then: look at the process you suspect

By now you usually have a suspect. A few commands confirm it without stopping it:

top -o %CPU            # or htop: per-thread view with H
cat /proc/<pid>/status | grep -E 'Threads|VmRSS'
ls /proc/<pid>/fd | wc -l
journalctl -u myapp.service --since "-15 min" --no-pager | tail -50

The descriptor count compared with /proc/<pid>/limits catches "too many open files" before it happens, and the service journal shows what the application itself was complaining about.

Write it down as you go

Paste every command and its output into the incident channel or ticket while you work. It costs seconds and gives the postmortem a timeline, lets a teammate join without re-running everything, and keeps you honest about what you actually observed. A short note like "vmstat r=15 on 8 cores, us 80, no swap; pidstat shows report-worker at 640 percent CPU" is worth more than any later reconstruction.

Common mistakes

Mistake Why it hurts Do this instead
Rebooting first Destroys the evidence and the problem often returns Capture the outputs above, then decide
Reading free instead of available Every healthy server looks out of memory Use available and memory PSI
Trusting %util on NVMe or cloud disks Shows 100 percent while the device has headroom Look at await, queue size and provider limits
Reading the first line of vmstat or iostat It is an average since boot Use interval output only
Skipping dmesg Misses OOM kills, disk and conntrack errors Check it in the first minute
Only checking df -h Inode exhaustion looks like free space Run df -i too

Key takeaways

  • Work through CPU, memory, disk and network in a fixed order instead of jumping to a theory.
  • Load average includes tasks waiting on I/O; compare it with the CPU count and watch its trend.
  • available memory and /proc/pressure/* tell you whether a resource is actually causing waits.
  • Kernel messages often contain the answer outright; read them early.
  • On modern storage, latency beats %util as a saturation signal.
  • Record every command and output as you go; the postmortem will need it.

References