When a server is slow and the dashboard is vague, a dozen standard commands narrow the problem to CPU, memory, disk, network or a single process in about five minutes. This is the order we run them in and how we read each output.
Our MySQL primary hung and then aborted with the data volume at 100%. Binary logs had grown to 410 GB because a backfill tripled write volume while retention had been raised for a lagging replica. How we recovered, purged safely and stopped it recurring.
Our nightly report export ran perfectly when we typed it, and failed every night as a systemd service with node: command not found. Services get no login shell, no .bashrc and a working directory of /. Here is how we made the unit self-contained.
Our thumbnail service hit Errno 24 at about 1,000 concurrent connections even though ulimit -n showed 65535. systemd services never read limits.conf. We set LimitNOFILE in a drop-in and fixed a client leak that was eating descriptors.
Our single-host production server stopped accepting writes at 03:12 with no space left on device. Docker had quietly filled the disk with a 31 GB container log, old images and build cache. Here is how we found each piece and stopped it from coming back.
After moving our PHP app to a new Ubuntu 24.04 server, every page returned 502 Bad Gateway. Fixing socket ownership got us online, then traffic spikes brought the 502s back from an undersized FPM pool. Here is how we fixed both.