Memory leaks, slow queries, caching, high-traffic incidents.
Our main pipeline took 34 minutes, and engineers batched changes to avoid waiting, which made reviews worse. We cut it to 14 minutes without new hardware magic: measure the critical path, parallelise, cache the right layers, split tests by timing and build only what changed.
Our image model trained on an expensive GPU that sat idle 70 percent of the time. Profiling showed the GPU waiting on CPU-side JPEG decoding and augmentation in too few DataLoader workers. More workers, pinned memory, GPU-side augmentation and preprocessed shards tripled throughput.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
Scoring 80 million rows with a 600 MB model took over 9 hours on a 20-node Spark cluster. The pandas UDF loaded the model from storage on every Arrow batch, thousands of times per executor. An iterator UDF that loads once per task cut the job to 28 minutes.
Our LLM bill tripled in one month and nobody could say which feature caused it. This guide covers what we built since: per-feature token accounting, prompt and response caching, model routing, output limits, batch processing and budgets with alerts.
Our quoting bot stayed far below request weight limits and still got -1015 Too many new orders at the worst moments. Order placement has its own per-account count limits, and our cancel-and-replace loop doubled order churn when prices moved fast. An order rate limiter and order amendment fixed it.
During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
Page 1 of our activity feed loaded in 9 ms, page 4,000 in 3.2 seconds, and a partner's sync job was walking every page. OFFSET makes Postgres read and discard every earlier row. We switched to keyset pagination with a tie-breaker and a matching index.
Enabling dynamic batching in Triton Inference Server doubled throughput for our ranking model and pushed p99 latency from 38 to 85 ms, breaking our SLO. A large queue delay and preferred batch sizes made requests wait for batches that rarely filled at low traffic. Measured tuning fixed both.
We once spent two weeks optimizing SQL for an endpoint whose real cost was a logging call. Since then every performance task starts with a profile. Here is how we capture flame graphs in PHP, Node.js and Python, how to read them, and how to measure the win.
Our Ray preprocessing job for image embeddings failed after an hour with OutOfMemoryError. Ray schedules tasks by CPU count, so it packed 16 tasks that each needed 3 GB onto nodes with 30 GB. Declaring memory per task, smaller blocks and a right-sized object store fixed it.
Requests decide where a pod is scheduled and how much CPU it is guaranteed; limits decide when it is throttled or killed. This guide explains QoS classes, CFS throttling and OOM kills, and shows how we size containers from real metrics with HPA and VPA.
We added a semantic cache to our help assistant to cut LLM cost. Within days a user asking how to cancel a subscription got instructions for cancelling an order. The two questions were 0.92 similar. We added exact-match first, scoped keys, intent gating and a measured threshold.
During a traffic spike our checkout service stayed at 3 pods while CPU was pegged. kubectl get hpa showed <unknown>/70% for weeks. A sidecar container added by a platform change had no CPU request, so utilisation could not be computed. Requests on every container fixed it.
Our Laravel export workers started at 70 MB and reached 1 GB after six hours, getting slower until Kubernetes killed them mid-job. An always-on query log, a static memo cache and whole-table loads were to blame. We fixed each and added worker limits.
One slow partner API call made our whole FastAPI service stop answering, health checks included, until Kubernetes restarted the pods. A synchronous requests call and a time.sleep inside async def were blocking the event loop.
We moved all EBS volumes from gp2 to gp3 to save 20 percent, as every cost guide recommends. Two days later, nightly reports on our self-managed Postgres took twice as long. The 2 TB gp2 volume had 6,000 IOPS and 250 MiB/s; gp3 defaults are 3,000 and 125. Provisioning both fixed it.
Every deploy and every traffic burst, the first requests to our booking API timed out and Postgres started refusing clients. The handler did heavy setup and opened a fresh connection per invocation. Moving init out of the handler and adding RDS Proxy fixed it.
A routine database failover lasted 20 seconds, but our API stayed down for 41 minutes. Three layers each made up to three attempts with no delay, multiplying load on the database by 27. Here is the math, the fix and how we proved it.
We enabled BuildKit's GitHub Actions cache and our multi-stage Go build still took 9 minutes on every run. The default cache export mode only stores layers of the final image, not the builder stage where all the time is spent. mode=max and per-branch scopes cut it to 2 minutes.
A shipment timeline query had a compound index yet still took 2.4 seconds. explain() showed a blocking SORT stage and 180,000 documents examined for 50 results. Reordering the index by Equality, Sort, Range fixed it.
After moving our Symfony app to the official php-fpm image, median response time went from 38 ms to 121 ms with no code change. OPcache was never enabled in the new image, and checking it from the CLI hid the problem. Here is how we found it and tuned it.
Our document tagger hit HTTP 429 constantly while sending only 40 requests a minute, far below the request limit. The token-per-minute limit was the real ceiling, inflated by an oversized max_tokens and minute-aligned bursts. How we read the headers and built a client-side limiter.
Users were logged out at random, mostly in the afternoon. Sessions and a fast-growing cache shared one Redis with allkeys-lru, so every hourly cache warm-up evicted idle sessions. Separate DB numbers would not have helped; separate instances did.
EXPLAIN ANALYZE prints a wall of nodes, costs and loops that most of us skim for the word Seq Scan. This guide walks a real slow query node by node: estimates vs actuals, loops, buffers, and the fixes each pattern points to.