A routine database failover lasted 20 seconds, but our API stayed down for 41 minutes. Three layers each made up to three attempts with no delay, multiplying load on the database by 27. Here is the math, the fix and how we proved it.
Running the test suite inside our app container dropped every table in the development database. Docker Compose injected DB_DATABASE as a real environment variable, and phpunit.xml's env entries do not override existing variables. Here is the full precedence chain and three layers of defence.
A low but stubborn rate of 502s at the load balancer and socket hang up errors between services turned out to be one race: Node closed idle keep-alive connections after 5 seconds while its callers kept reusing them.
Order dates rendered fine locally but threw hydration errors for users in Asia and Europe. The server formatted dates in UTC and en-US, the browser in local time. Here is how we found it and fixed it for good.
After a harmless-looking Express refactor, every Stripe webhook failed verification and paid bookings stayed pending. A global JSON parser consumed the body before constructEvent saw it. How we found it, fixed the route and replayed the missed events.
Our fraud model scored 0.91 AUC on holdout data and close to random in production. The serving service re-implemented preprocessing in another language: the scaler was refit on each batch, one-hot categories came in a different order and text was not lowercased. Shipping the pipeline with the model fixed it.
A practitioner's reading of AWS's official summary of the October 19-20, 2025 us-east-1 disruption: a race in DynamoDB's DNS automation left an empty record, and recovery cascaded through EC2, NLB and Lambda. What happened per the report, and what we changed.
Our support agent started failing with a 400 on roughly one conversation in twelve. The model had begun returning several tool calls in one turn and our loop only answered the first. Answering every tool_call_id in order fixed it.
We switched our semantic search to a cheaper 1024-dimension embedding model and every insert and query failed against the 1536-dimension column. Worse, the half-finished fix would have mixed incompatible vectors. How we migrated with versioned columns, a backfill and dual reads.
A new model that wins offline can still lose in production, through skew, latency, or effects on users that offline data cannot show. This guide covers a staged rollout for models: offline gates, shadow traffic, canaries on business metrics, A/B tests when needed, and fast rollback.
Page 1 of our activity feed loaded in 9 ms, page 4,000 in 3.2 seconds, and a partner's sync job was walking every page. OFFSET makes Postgres read and discard every earlier row. We switched to keyset pagination with a tie-breaker and a matching index.
Bumping targetSdk to 35 put our toolbar under the status bar and the Pay button under the navigation bar on Android 15 devices. Edge-to-edge is now enforced, so we handled window insets properly in both Views and Compose screens.
In a red-team exercise our support agent read a customer-uploaded PDF with hidden instructions and drafted an email with another customer's order details. No data left the building, but the flaw was real. How we contained it with least privilege, confirmation and output checks.
Once more than two services call LLM providers, each one reinvents timeouts, retries and key handling, usually badly. This guide covers what a small internal LLM gateway should do, the timeout and retry rules that work, and how to fail over without making outages worse.
Our PDF invoices and the amounts charged through Stripe disagreed by a cent on 214 invoices in one month. Floats, toFixed and rounding at every step were to blame. We moved to integer minor units with one explicit rounding point.
Receipt uploads to S3 failed with 403 SignatureDoesNotMatch for about 4 percent of users, mostly iPhone photos uploaded from Windows. The URL was signed with one Content-Type and the browser sent another. Reading the CanonicalRequest in the error pinpointed it.
Small tenants in our RAG product got 2 or 3 chunks back instead of 10, and sometimes none. HNSW finds ef_search candidates first and applies the tenant filter afterwards. Iterative index scans in pgvector 0.8 plus partial indexes for big tenants fixed recall.
Our partner OAuth server matched redirect_uri by prefix, and our login page followed any next URL. Chained together with a legacy implicit flow, an access token could land on an outside domain. We moved to exact matching, relative-only next and code plus PKCE.
Preview environment namespaces piled up in Terminating and our cleanup job kept failing. The namespace conditions pointed at two causes: a broken metrics APIService that blocked discovery, and custom resources whose operator had been uninstalled. Fixing the causes, not stripping finalizers, cleared them.
After we turned on streaming for our assistant, tool calls failed with JSON parse errors and sometimes ran with another call's arguments. We were parsing argument fragments per chunk and keying them by arrival order. Accumulating by index fixed it.
Our Laravel export workers started at 70 MB and reached 1 GB after six hours, getting slower until Kubernetes killed them mid-job. An always-on query log, a static memo cache and whole-table loads were to blame. We fixed each and added worker limits.
Our managed node group upgrade stalled for two hours on one node. A PodDisruptionBudget with minAvailable equal to the replica count meant no pod could ever be evicted. We switched to maxUnavailable, fixed replica counts and added a pre-upgrade PDB check.
Automated trading fails fast: a bug can place hundreds of orders before a human notices. This guide covers the monitoring and stopping mechanisms we run for every bot: heartbeats, position and loss limits, cancel-all on disconnect, a tested kill switch, and runbooks for the first five minutes.
A practical guide to time handling that survives DST switches: when to store UTC instants, when to keep local time plus an IANA zone, how gaps and overlaps behave, which database types to use, and how to test with fixed clocks.
Sending teams a monthly cost spreadsheet changes nothing. Charging their budgets before they trust the numbers starts arguments. This guide covers a showback program that works: ownership mapping, a short report with actions, budgets per service, and when to move to chargeback.