A practitioner's reading of AWS's official summary of the October 19-20, 2025 us-east-1 disruption: a race in DynamoDB's DNS automation left an empty record, and recovery cascaded through EC2, NLB and Lambda. What happened per the report, and what we changed.
We switched our semantic search to a cheaper 1024-dimension embedding model and every insert and query failed against the 1536-dimension column. Worse, the half-finished fix would have mixed incompatible vectors. How we migrated with versioned columns, a backfill and dual reads.
Page 1 of our activity feed loaded in 9 ms, page 4,000 in 3.2 seconds, and a partner's sync job was walking every page. OFFSET makes Postgres read and discard every earlier row. We switched to keyset pagination with a tie-breaker and a matching index.
In a red-team exercise our support agent read a customer-uploaded PDF with hidden instructions and drafted an email with another customer's order details. No data left the building, but the flaw was real. How we contained it with least privilege, confirmation and output checks.
Once more than two services call LLM providers, each one reinvents timeouts, retries and key handling, usually badly. This guide covers what a small internal LLM gateway should do, the timeout and retry rules that work, and how to fail over without making outages worse.
Preview environment namespaces piled up in Terminating and our cleanup job kept failing. The namespace conditions pointed at two causes: a broken metrics APIService that blocked discovery, and custom resources whose operator had been uninstalled. Fixing the causes, not stripping finalizers, cleared them.
Our managed node group upgrade stalled for two hours on one node. A PodDisruptionBudget with minAvailable equal to the replica count meant no pod could ever be evicted. We switched to maxUnavailable, fixed replica counts and added a pre-upgrade PDB check.
Automated trading fails fast: a bug can place hundreds of orders before a human notices. This guide covers the monitoring and stopping mechanisms we run for every bot: heartbeats, position and loss limits, cancel-all on disconnect, a tested kill switch, and runbooks for the first five minutes.
Sending teams a monthly cost spreadsheet changes nothing. Charging their budgets before they trust the numbers starts arguments. This guide covers a showback program that works: ownership mapping, a short report with actions, budgets per service, and when to move to chargeback.
Our support agent started opening refund requests when customers only asked where their order was. Tool names and descriptions overlapped, and the agent had 23 tools to choose from. Clear tool contracts, fewer tools per step and a tool-choice eval fixed it.
During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
A cloud bill that grows 30 percent can be great news or a disaster, depending on what grew with it. Unit costs, such as cost per active customer or per thousand requests, tell you which. This guide shows how to choose units, allocate shared costs and use the numbers in decisions.
When we finally split our bill by environment, development, staging and preview environments were 38 percent of total cloud spend. Most of it ran 24 hours a day for people who work 8. This guide covers schedules, ephemeral environments, right-sizing, data copies and guardrails.
A routine dependency update in our scoring service upgraded scikit-learn, and the logs filled with InconsistentVersionWarning when loading models pickled with an older version. One model's predictions shifted slightly. We pinned versions with each model, added load-time checks and moved to portable formats.
After a configuration refactor, a bot meant for the testnet connected to the production exchange with test parameters: order sizes ten times larger than production limits. A risk check caught it after four orders. This postmortem covers how environment selection failed and the guardrails we added.
A pipeline you are afraid to re-run is a pipeline that will eventually corrupt data. This guide covers partition overwrite, MERGE upserts, dedup keys, watermarks and why exactly-once is a property you build, not a setting you enable.
Teams often reach for fine-tuning when they need retrieval, or build a retrieval pipeline when the whole corpus fits in the prompt. This guide compares RAG, fine-tuning and long context on freshness, cost, latency, citations and effort, with a decision table and examples.
The pre-launch security checklist we run on every new web app, organised by the OWASP Top 10:2025 categories: authorization tests, injection, secrets, headers and CSP, supply chain, error handling and logging, each with a concrete check you can automate.
Software CI checks code. ML systems also change when data changes, when a model is retrained, or when a dependency shifts numerics. This guide lays out a CI/CD pipeline for ML: versioning, tests for data and models, reproducible training, promotion gates and deployment.
Our LoRA adapter for ticket summarisation worked in evaluation and produced repetitive nonsense in production. Serving loaded the base model by name without a revision, and the hub repository had been updated with a new tokenizer and weights. Pinning base revisions and a load-time smoke test fixed it.
Our image model trained on an expensive GPU that sat idle 70 percent of the time. Profiling showed the GPU waiting on CPU-side JPEG decoding and augmentation in too few DataLoader workers. More workers, pinned memory, GPU-side augmentation and preprocessed shards tripled throughput.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
A security group rule added in the console during an incident survived for eight months and was wiped out by an unrelated apply. Drift is normal; unmanaged drift is the problem. This guide covers scheduled drift detection, triage, import blocks and guardrails against click-ops.
A canary that a human watches on a dashboard is only as good as that human's attention. This guide covers automated canary analysis: which metrics to gate on, how to size steps and durations, how to handle low traffic, and the traps that make canaries pass when they should fail.
A refactor changed one output prefix in an image processing Lambda, and the function started triggering itself. Invocations went from 40,000 a day to 9 million in six hours. We stopped it with reserved concurrency, then separated input and output and added loop guards.