When an agent answer is slow or wrong, logs rarely tell you which step caused it. This guide shows how to trace LLM calls, tool executions and agent runs with OpenTelemetry's GenAI semantic conventions, which attributes to record, and how to handle prompts and privacy.
Scoring 80 million rows with a 600 MB model took over 9 hours on a 20-node Spark cluster. The pandas UDF loaded the model from storage on every Arrow batch, thousands of times per executor. An iterator UDF that loads once per task cut the job to 28 minutes.
We switched our semantic search to a cheaper 1024-dimension embedding model and every insert and query failed against the 1536-dimension column. Worse, the half-finished fix would have mixed incompatible vectors. How we migrated with versioned columns, a backfill and dual reads.
A one-off data backfill filled our Postgres RDS instance, storage autoscaling grew it from 400 GB to 2 TB, and after we deleted the temporary tables the bill stayed. RDS allocated storage cannot be reduced in place. We shrank it with a blue/green switchover and capped autoscaling.
A trading system that cannot explain its balances to the cent will eventually make decisions on wrong numbers. This guide shows how we keep an internal double-entry ledger for trades, fees, funding and transfers, reconcile it with exchange balances, and handle the differences we find.
A security group rule added in the console during an incident survived for eight months and was wiped out by an unrelated apply. Drift is normal; unmanaged drift is the problem. This guide covers scheduled drift detection, triage, import blocks and guardrails against click-ops.
Our LoRA adapter for ticket summarisation worked in evaluation and produced repetitive nonsense in production. Serving loaded the base model by name without a revision, and the hub repository had been updated with a new tokenizer and weights. Pinning base revisions and a load-time smoke test fixed it.
A field guide to receiving webhooks reliably: verify signatures on the raw body, respond fast, queue the work, deduplicate by event ID, tolerate out-of-order delivery, replay safely and monitor the whole pipeline. With code you can adapt.
A developer added user_id as a label on a request latency histogram to debug slow accounts. Active time series went from 1.2 million to 19 million, our hosted metrics bill quadrupled and the query layer slowed down. We found the culprit, dropped the label and set cardinality limits.
Our LLM bill tripled in one month and nobody could say which feature caused it. This guide covers what we built since: per-feature token accounting, prompt and response caching, model routing, output limits, batch processing and budgets with alerts.
Our nightly invoice export stopped running and nobody noticed for nine days. One Job had hung on a network call with no timeout, and with concurrencyPolicy Forbid every later schedule was skipped. Deadlines on the Job, timeouts in code and an alert on last success fixed it.
Once more than two services call LLM providers, each one reinvents timeouts, retries and key handling, usually badly. This guide covers what a small internal LLM gateway should do, the timeout and retry rules that work, and how to fail over without making outages worse.
A protective stop on a perpetual futures position did not trigger during a sharp wick that clearly crossed the stop on the last-price chart. The order was set to trigger on mark price, which moved much less. We now choose the trigger price type explicitly per order and test both behaviours.
We enabled fp16 autocast to speed up training and the loss turned NaN within 300 steps. A custom attention score and a loss computed in half precision overflowed fp16's range, and we had not used a gradient scaler. bf16 on our GPUs, fp32 for the sensitive ops and a NaN guard fixed it.
In a red-team exercise our support agent read a customer-uploaded PDF with hidden instructions and drafted an email with another customer's order details. No data left the building, but the flaw was real. How we contained it with least privilege, confirmation and output checks.
After we added merge request rules to a few jobs, every push to an open merge request started two pipelines, doubling runner usage. Job-level rules matched both the branch push and the merge request event. A workflow:rules block that picks one pipeline type fixed it.
A refactor changed one output prefix in an image processing Lambda, and the function started triggering itself. Invocations went from 40,000 a day to 9 million in six hours. We stopped it with reserved concurrency, then separated input and output and added loop guards.
The pre-launch security checklist we run on every new web app, organised by the OWASP Top 10:2025 categories: authorization tests, injection, secrets, headers and CSP, supply chain, error handling and logging, each with a concrete check you can automate.
Our nightly product description job used the Batch API to save cost. One night the batch expired with 30 percent unfinished, and our importer joined results by line position, so descriptions landed on the wrong products. We now join by custom_id and handle every terminal status.
Our quoting bot stayed far below request weight limits and still got -1015 Too many new orders at the worst moments. Order placement has its own per-account count limits, and our cancel-and-replace loop doubled order churn when prices moved fast. An order rate limiter and order amendment fixed it.
Our queue workers took exactly 30 seconds to stop on every deploy and lost the job they were processing. Exit code 137 in every pod. The Dockerfile used shell-form CMD, so /bin/sh was PID 1 and never forwarded SIGTERM to Node. Exec form plus a real shutdown handler fixed it.
During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
Kubernetes announced the retirement of ingress-nginx, with best-effort maintenance ending in March 2026. This is the plan we used to inventory annotations, map them to Gateway API, run both side by side and cut over host by host without downtime.
We launched local pricing in JPY, KRW and VND and a customer in Osaka was charged 398,000 yen for a 3,980 yen course. Our code multiplied every price by 100 before sending it to Stripe. How we found it, refunded it and built a per-currency exponent table with tests.
Our GPU node pool stayed at eight nodes from Friday night to Monday morning with almost no GPU usage. The cluster autoscaler refused to remove them: notebook pods with local storage, a bare debug pod and a kube-system pod without a PDB. Fixing each blocker cut GPU spend by 38 percent.