A routine dependency update in our scoring service upgraded scikit-learn, and the logs filled with InconsistentVersionWarning when loading models pickled with an older version. One model's predictions shifted slightly. We pinned versions with each model, added load-time checks and moved to portable formats.
Our expense app sends receipt photos to a vision model. The feature cost six times our estimate and p95 latency was 9 seconds. Phones upload 12-megapixel photos and we sent them as-is. Resizing and cropping before upload cut cost by 78 percent with no accuracy loss.
Our nightly product description job used the Batch API to save cost. One night the batch expired with 30 percent unfinished, and our importer joined results by line position, so descriptions landed on the wrong products. We now join by custom_id and handle every terminal status.
We launched local pricing in JPY, KRW and VND and a customer in Osaka was charged 398,000 yen for a 3,980 yen course. Our code multiplied every price by 100 before sending it to Stripe. How we found it, refunded it and built a per-currency exponent table with tests.
A customer saw a snippet from another company's document in an AI answer. A new 'related questions' feature queried the vector store through a helper that did not apply the tenant filter. We moved tenant scoping into the repository layer so no code path can skip it.
A new bot received order updates for one hour after start, then nothing, while the websocket stayed connected. The listenKey expires after 60 minutes without a keepalive. We fixed keepalives and expiry handling, then moved to the WebSocket API subscription before listenKey retirement.
Our support assistant quoted an old refund policy that had been removed from the help center weeks earlier. The ingestion pipeline only ever upserted chunks and never deleted them. We added source-of-truth sync, document versions and a nightly reconciliation job.
A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.
Our Ray preprocessing job for image embeddings failed after an hour with OutOfMemoryError. Ray schedules tasks by CPU count, so it packed 16 tasks that each needed 3 GB onto nodes with 30 GB. Declaring memory per task, smaller blocks and a right-sized object store fixed it.
Prompt and response logs are the fastest way to build an accidental PII database. This guide shows where personal data enters an LLM feature, how to redact it before logging and before the provider, and how to keep retention and access under control.
Our daily signals disagreed with the exchange chart, and intraday signals appeared and disappeared within the same bar. We resampled UTC millisecond timestamps in local time and included the still-forming last candle. UTC everywhere and using only closed candles fixed both problems.
A hedging bot meant to short perpetual futures placed spot sell orders instead, selling inventory it was supposed to hedge. In ccxt's unified symbols, BTC/USDT is spot and BTC/USDT:USDT is the linear perpetual. Explicit market types and assertions on market metadata fixed it.
A research agent called the same search tool 412 times in one run and spent more overnight than the whole feature spent in a normal week. The tool returned an empty string on errors, the loop had no step limit and nothing tracked spend per run.
Every value in our orders API was a bound parameter, except the sort column, which was pasted into ORDER BY from the query string. A pentest proved it was injectable. We replaced it with an allowlist map and added tests that fuzz the sort input.
Our batch classification job with a small open model produced good output for the longest prompt in each batch and nonsense for the rest. Transformers warned that right-padding was detected for a decoder-only model. Left padding, a proper pad token and the attention mask fixed it.
One slow partner API call made our whole FastAPI service stop answering, health checks included, until Kubernetes restarted the pods. A synchronous requests call and a time.sleep inside async def were blocking the event loop.
About 2% of our LLM extraction calls crashed on JSON.parse: markdown code fences, answers cut off at max_tokens and chatty text after the object. We classified every failure, switched to schema-constrained output, checked stop reasons and added validation with one repair attempt.
Customer CSV uploads failed with KeyError 'email' even though the header clearly said email. Excel had written a UTF-8 byte order mark, and other files arrived in Windows-1252. We now detect the encoding explicitly and normalise headers.
Our bot believed two orders were still open when they had filled hours earlier. The exchange closes websocket connections after 24 hours; our reconnect logic resubscribed but never checked what happened while disconnected. Reconnect with backoff plus REST reconciliation fixed the state drift.
A pull request added a dependency that an AI assistant had suggested. The package name did not exist when the code was written, and a few weeks later someone else published it. We caught it in review by luck, then added registry checks so we do not rely on luck.
Accuracy is the metric everyone wants, and for many models the labels arrive weeks or months after the prediction. This guide covers what to monitor in the meantime: data quality, input drift with PSI and KS tests, prediction drift, and alert design that does not cry wolf.
Our internal MCP server passed every test in the inspector but kept failing in the desktop client with an 'is not valid JSON' error. A startup banner and a debug print went to stdout, which is the protocol channel in stdio transport. Logging to stderr fixed it.
Our order router ran fine as one process. When we scaled to two workers, a third of private API calls failed with EAPI:Invalid nonce. Kraken requires an increasing nonce per API key, and two processes generating millisecond timestamps raced each other. One key per worker fixed it.
Our document tagger hit HTTP 429 constantly while sending only 40 requests a minute, far below the request limit. The token-per-minute limit was the real ceiling, inflated by an oversized max_tokens and minute-aligned bursts. How we read the headers and built a client-side limiter.
Our meeting transcripts contained sentences nobody said, usually during silence or hold music, plus the same line repeated a dozen times. Whisper conditions on its own previous output and has no idea what silence is. A VAD pre-filter and stricter decoding thresholds removed most of it.