Our document tagger hit HTTP 429 constantly while sending only 40 requests a minute, far below the request limit. The token-per-minute limit was the real ceiling, inflated by an oversized max_tokens and minute-aligned bursts. How we read the headers and built a client-side limiter.
Users were logged out at random, mostly in the afternoon. Sessions and a fast-growing cache shared one Redis with allkeys-lru, so every hourly cache warm-up evicted idle sessions. Separate DB numbers would not have helped; separate instances did.
Our live support chat dropped every connection that stayed quiet for a minute, with close code 1006 and no server error. Nginx's 60-second proxy_read_timeout and the load balancer idle timeout were closing idle sockets. A heartbeat plus proxy tuning fixed it.
After a scheduler outage, our daily orders DAG caught up two missed runs and both loaded the same day while another day was skipped. The task computed its date from the wall clock instead of the run's data interval.
EXPLAIN ANALYZE prints a wall of nodes, costs and loops that most of us skim for the word Seq Scan. This guide walks a real slow query node by node: estimates vs actuals, loops, buffers, and the fixes each pattern points to.
A practical guide to blameless postmortems: how to build a timeline, measure impact, find contributing factors instead of a culprit, write action items that get done, and run the review meeting so people tell the truth.
E0502 stopped us from updating parent totals inside a HashMap loop, so someone cloned the whole map to make it compile. That clone ran every ten seconds over 1.2 million entries. Here is why the borrow checker was right and the fixes that kept memory flat.
One CI run in eight failed somewhere in our checkout E2E suite, never in the same test twice. The causes were fixed sleeps, URL checks that raced navigation and text selectors that matched two buttons. Web-first assertions, role locators and traces fixed it.
Receipt uploads to S3 failed with 403 SignatureDoesNotMatch for about 4 percent of users, mostly iPhone photos uploaded from Windows. The URL was signed with one Content-Type and the browser sent another. Reading the CanonicalRequest in the error pinpointed it.
Every POST to our Django admin returned 403 after we moved TLS termination to a Traefik ingress. Django compared the browser's https Origin with a request it believed was plain http. Two settings fixed it without weakening CSRF.
A slow accounting API pushed our PayPal webhook handler past the nginx timeout. PayPal saw a 504 and retried, while the first request was still running, and 41 wallets were topped up twice. Postmortem plus the idempotent design we moved to.
Our chat assistant streamed tokens smoothly on localhost but in production the whole answer appeared after 14 seconds. nginx was buffering the event stream and gzip was holding bytes. How we proved it with curl and fixed it without breaking other routes.
After importing 410,000 tickets with their original ids, most new tickets failed with duplicate key value violates unique constraint tickets_pkey, while a few succeeded. The id sequence was still at 1. Here is why, and how we resynced every sequence safely.
Every ten minutes our catalog API spiked to 4-second responses and the database hit 100% CPU. A hot Redis key expired and 600 requests rebuilt it at once. Single-flight, a short Redis lock, stale-while-revalidate and TTL jitter fixed it.
A cancelled GitLab pipeline killed terraform apply mid-run and left the S3 backend locked. Every plan after that failed with Error acquiring the state lock. How we proved no run was active, unlocked safely, checked for drift, and stopped CI from cancelling applies.
Uploading a PDF larger than 1 MB failed with a vague error in our claims portal. Nginx returned 413, then PHP silently dropped files, then Laravel rejected them. Four layers each had their own size limit. Here is how we aligned them and tested every boundary.
A practical guide to time handling that survives DST switches: when to store UTC instants, when to keep local time plus an IANA zone, how gaps and overlaps behave, which database types to use, and how to test with fixed clocks.
Login and checkout forms started failing with 419 Page Expired the day we put two Laravel nodes behind a TLS-terminating load balancer. Untrusted proxy headers, secure cookies and file sessions on each node combined into one confusing error.
Our PayPal checkout button rendered in development and staging but was an empty box in production. A new Content-Security-Policy set in Nginx blocked the PayPal SDK script, its iframes and its API calls. We fixed it per directive, with a nonce and a report-only rollout.
Every engineer who downloaded our internal deploy helper got a "damaged" dialog. The app was fine; a CI step modified the bundle after signing and Gatekeeper checked the quarantined copy. We now sign, notarize and staple in the right order.
Our deploy ran composer update on a 2 GB server until the resolver ran out of memory. Raising the limit let the OOM killer take MySQL instead. Committing the lock file exposed a second bug: it had been resolved on PHP 8.4 for a PHP 8.2 server.
Bumping targetSdk to 35 put our toolbar under the status bar and the Pay button under the navigation bar on Android 15 devices. Edge-to-edge is now enforced, so we handled window insets properly in both Views and Compose screens.
A good bug report lets someone who has never seen your screen reproduce the problem in minutes. The anatomy we use: environment, exact steps, expected versus actual, evidence, a minimal reproduction, and severity kept separate from priority.
Every deploy of our Go pricing API produced 30 to 60 seconds of 502 and 503 errors. New pods got traffic before they were ready, and old pods exited while the ingress still routed to them. A readiness probe, a preStop sleep and real graceful shutdown made rollouts clean.
Our PDF invoices and the amounts charged through Stripe disagreed by a cent on 214 invoices in one month. Floats, toFixed and rounding at every step were to blame. We moved to integer minor units with one explicit rounding point.