Our order manager occasionally raised KeyError: unknown order when a fill event arrived on the websocket before the REST place-order call returned. It keyed orders by exchange order id, which it only learned from the REST response. Keying by client order id and buffering unknown events fixed it.
A canary that a human watches on a dashboard is only as good as that human's attention. This guide covers automated canary analysis: which metrics to gate on, how to size steps and durations, how to handle low traffic, and the traps that make canaries pass when they should fail.
A routine dependency update in our scoring service upgraded scikit-learn, and the logs filled with InconsistentVersionWarning when loading models pickled with an older version. One model's predictions shifted slightly. We pinned versions with each model, added load-time checks and moved to portable formats.
When we finally split our bill by environment, development, staging and preview environments were 38 percent of total cloud spend. Most of it ran 24 hours a day for people who work 8. This guide covers schedules, ephemeral environments, right-sizing, data copies and guardrails.
Teams often reach for fine-tuning when they need retrieval, or build a retrieval pipeline when the whole corpus fits in the prompt. This guide compares RAG, fine-tuning and long context on freshness, cost, latency, citations and effort, with a decision table and examples.
Automated trading fails fast: a bug can place hundreds of orders before a human notices. This guide covers the monitoring and stopping mechanisms we run for every bot: heartbeats, position and loss limits, cancel-all on disconnect, a tested kill switch, and runbooks for the first five minutes.
Our vLLM deployment had an HPA targeting 70 percent CPU. Under a load spike, requests queued for 40 seconds while CPU sat at 15 percent, so nothing scaled. GPU-bound servers need to scale on queue depth. We switched to KEDA on vLLM's waiting-requests metric and fixed warm-up.
Our managed node group upgrade stalled for two hours on one node. A PodDisruptionBudget with minAvailable equal to the replica count meant no pod could ever be evicted. We switched to maxUnavailable, fixed replica counts and added a pre-upgrade PDB check.
Most expensive infrastructure decisions are made in a pull request where nobody sees the price. Adding a cost diff to Terraform PRs moves the conversation to the moment it is cheapest to change. This guide covers setup, usage estimates, thresholds and the limits of estimates.
Our support agent started opening refund requests when customers only asked where their order was. Tool names and descriptions overlapped, and the agent had 23 tools to choose from. Clear tool contracts, fewer tools per step and a tool-choice eval fixed it.
After a configuration refactor, a bot meant for the testnet connected to the production exchange with test parameters: order sizes ten times larger than production limits. A risk check caught it after four orders. This postmortem covers how environment selection failed and the guardrails we added.
Software CI checks code. ML systems also change when data changes, when a model is retrained, or when a dependency shifts numerics. This guide lays out a CI/CD pipeline for ML: versioning, tests for data and models, reproducible training, promotion gates and deployment.
Our main pipeline took 34 minutes, and engineers batched changes to avoid waiting, which made reviews worse. We cut it to 14 minutes without new hardware magic: measure the critical path, parallelise, cache the right layers, split tests by timing and build only what changed.
After a framework upgrade, our CDN cache hit ratio for static assets dropped from 97 to 41 percent and origin data transfer doubled. The new session middleware attached Set-Cookie and Cache-Control private to every response, including JavaScript bundles. Scoping middleware fixed it.
Our settlement bot sends several transactions per minute from one address. After we parallelised it, sends failed with nonce too low and replacement transaction underpriced, and some transactions sat stuck for hours. A single nonce manager per address and correct fee bumping fixed it.
Our expense app sends receipt photos to a vision model. The feature cost six times our estimate and p95 latency was 9 seconds. Phones upload 12-megapixel photos and we sent them as-is. Resizing and cropping before upload cut cost by 78 percent with no accuracy loss.
Our treasury rebalancing service swapped on a DEX and about a quarter of swaps reverted on volatile days with Too little received, wasting gas. Quotes were computed seconds before submission and transactions waited in the mempool. Fresh quotes, simulation and a deadline fixed it without loosening slippage.
Sending teams a monthly cost spreadsheet changes nothing. Charging their budgets before they trust the numbers starts arguments. This guide covers a showback program that works: ownership mapping, a short report with actions, budgets per service, and when to move to chargeback.
Our image model trained on an expensive GPU that sat idle 70 percent of the time. Profiling showed the GPU waiting on CPU-side JPEG decoding and augmentation in too few DataLoader workers. More workers, pinned memory, GPU-side augmentation and preprocessed shards tripled throughput.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
When an agent answer is slow or wrong, logs rarely tell you which step caused it. This guide shows how to trace LLM calls, tool executions and agent runs with OpenTelemetry's GenAI semantic conventions, which attributes to record, and how to handle prompts and privacy.
Scoring 80 million rows with a 600 MB model took over 9 hours on a 20-node Spark cluster. The pandas UDF loaded the model from storage on every Arrow batch, thousands of times per executor. An iterator UDF that loads once per task cut the job to 28 minutes.
A one-off data backfill filled our Postgres RDS instance, storage autoscaling grew it from 400 GB to 2 TB, and after we deleted the temporary tables the bill stayed. RDS allocated storage cannot be reduced in place. We shrank it with a blue/green switchover and capped autoscaling.
A trading system that cannot explain its balances to the cent will eventually make decisions on wrong numbers. This guide shows how we keep an internal double-entry ledger for trades, fees, funding and transfers, reconcile it with exchange balances, and handle the differences we find.
A security group rule added in the console during an incident survived for eight months and was wiped out by an unrelated apply. Drift is normal; unmanaged drift is the problem. This guide covers scheduled drift detection, triage, import blocks and guardrails against click-ops.