Every few minutes, all API requests stalled at the same moment and then recovered. The cause was an admin feature running KEYS over 40 million keys. Why KEYS blocks everything in Redis, how SLOWLOG and LATENCY found it, and the fixes: SCAN, sets as indexes, and disabling the command.
The order we apply scaling techniques to a growing web service, the signal that tells us each step is needed, what each one costs, and the mistakes we made by skipping ahead. From measuring the real bottleneck to partitioning data, which we treat as the last resort.
We added a large in-memory cache to a Java service to save Redis round trips. Average latency improved and p99 doubled. The cache pushed the heap into long GC pauses. How we found it in GC logs, why our benchmark missed it, and the measurement method we use now.
A pre-sale load test reported steady 80 ms latency all the way to the target rate. On sale day the API fell over at half that traffic. The test used a closed model with too few virtual users, so it slowed itself down instead of measuring the system. Open models, arrival-rate executors and generator checks.