Our order manager occasionally raised KeyError: unknown order when a fill event arrived on the websocket before the REST place-order call returned. It keyed orders by exchange order id, which it only learned from the REST response. Keying by client order id and buffering unknown events fixed it.
A canary that a human watches on a dashboard is only as good as that human's attention. This guide covers automated canary analysis: which metrics to gate on, how to size steps and durations, how to handle low traffic, and the traps that make canaries pass when they should fail.
A routine dependency update in our scoring service upgraded scikit-learn, and the logs filled with InconsistentVersionWarning when loading models pickled with an older version. One model's predictions shifted slightly. We pinned versions with each model, added load-time checks and moved to portable formats.
When we finally split our bill by environment, development, staging and preview environments were 38 percent of total cloud spend. Most of it ran 24 hours a day for people who work 8. This guide covers schedules, ephemeral environments, right-sizing, data copies and guardrails.
Teams often reach for fine-tuning when they need retrieval, or build a retrieval pipeline when the whole corpus fits in the prompt. This guide compares RAG, fine-tuning and long context on freshness, cost, latency, citations and effort, with a decision table and examples.
Automated trading fails fast: a bug can place hundreds of orders before a human notices. This guide covers the monitoring and stopping mechanisms we run for every bot: heartbeats, position and loss limits, cancel-all on disconnect, a tested kill switch, and runbooks for the first five minutes.
Our vLLM deployment had an HPA targeting 70 percent CPU. Under a load spike, requests queued for 40 seconds while CPU sat at 15 percent, so nothing scaled. GPU-bound servers need to scale on queue depth. We switched to KEDA on vLLM's waiting-requests metric and fixed warm-up.
Our managed node group upgrade stalled for two hours on one node. A PodDisruptionBudget with minAvailable equal to the replica count meant no pod could ever be evicted. We switched to maxUnavailable, fixed replica counts and added a pre-upgrade PDB check.
Most expensive infrastructure decisions are made in a pull request where nobody sees the price. Adding a cost diff to Terraform PRs moves the conversation to the moment it is cheapest to change. This guide covers setup, usage estimates, thresholds and the limits of estimates.
Four engineers, one invoicing product, no QA team. This is the risk-based testing strategy we settled on: integration tests over mocks, contract tests at service edges, a handful of E2E happy paths and test data that does not rot.
Our support agent started opening refund requests when customers only asked where their order was. Tool names and descriptions overlapped, and the agent had 23 tools to choose from. Clear tool contracts, fewer tools per step and a tool-choice eval fixed it.
After a configuration refactor, a bot meant for the testnet connected to the production exchange with test parameters: order sizes ten times larger than production limits. A risk check caught it after four orders. This postmortem covers how environment selection failed and the guardrails we added.
Software CI checks code. ML systems also change when data changes, when a model is retrained, or when a dependency shifts numerics. This guide lays out a CI/CD pipeline for ML: versioning, tests for data and models, reproducible training, promotion gates and deployment.
Our player character stuttered against the background whenever the camera followed it, worst on 144 Hz monitors. Movement wrote the transform in Update, the camera read physics positions, and interpolation was off. How we measured the jitter and fixed all three.
Our main pipeline took 34 minutes, and engineers batched changes to avoid waiting, which made reviews worse. We cut it to 14 minutes without new hardware magic: measure the critical path, parallelise, cache the right layers, split tests by timing and build only what changed.
After a framework upgrade, our CDN cache hit ratio for static assets dropped from 97 to 41 percent and origin data transfer doubled. The new session middleware attached Set-Cookie and Cache-Control private to every response, including JavaScript bundles. Scoping middleware fixed it.
When a server is slow and the dashboard is vague, a dozen standard commands narrow the problem to CPU, memory, disk, network or a single process in about five minutes. This is the order we run them in and how we read each output.
Our settlement bot sends several transactions per minute from one address. After we parallelised it, sends failed with nonce too low and replacement transaction underpriced, and some transactions sat stuck for hours. A single nonce manager per address and correct fee bumping fixed it.
Our dosing controllers ran for weeks, then locked themselves out on day 49. The interval check compared now >= lastRun + interval, which breaks when the sum wraps past 2^32 milliseconds. How we proved it with a simulated clock and fixed it with unsigned subtraction.
Our expense app sends receipt photos to a vision model. The feature cost six times our estimate and p95 latency was 9 seconds. Phones upload 12-megapixel photos and we sent them as-is. Resizing and cropping before upload cut cost by 78 percent with no accuracy loss.
Our treasury rebalancing service swapped on a DEX and about a quarter of swaps reverted on volatile days with Too little received, wasting gas. Quotes were computed seconds before submission and transactions waited in the mempool. Fresh quotes, simulation and a deadline fixed it without loosening slippage.
Sending teams a monthly cost spreadsheet changes nothing. Charging their budgets before they trust the numbers starts arguments. This guide covers a showback program that works: ownership mapping, a short report with actions, budgets per service, and when to move to chargeback.
A pipeline you are afraid to re-run is a pipeline that will eventually corrupt data. This guide covers partition overwrite, MERGE upserts, dedup keys, watermarks and why exactly-once is a property you build, not a setting you enable.
Our image model trained on an expensive GPU that sat idle 70 percent of the time. Profiling showed the GPU waiting on CPU-side JPEG decoding and augmentation in too few DataLoader workers. More workers, pinned memory, GPU-side augmentation and preprocessed shards tripled throughput.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.