After a scheduler outage, our daily orders DAG caught up two missed runs and both loaded the same day while another day was skipped. The task computed its date from the wall clock instead of the run's data interval.
A moving-average strategy showed strong backtest returns and nothing like it in paper trading. The backtest computed the signal from a candle's close and filled the order at that same close, a price nobody could trade at once the signal was known. Shifting signals to the next bar fixed the simulation.
We added cache_control to a 6,000-token system prompt and the bill went up instead of down. Every request wrote the cache and none read it, because the prompt prefix changed on every call. A stable prefix brought cache reads to 92 percent.
Every POST to our Django admin returned 403 after we moved TLS termination to a Traefik ingress. Django compared the browser's https Origin with a request it believed was plain http. Two settings fixed it without weakening CSRF.
A practical guide to time handling that survives DST switches: when to store UTC instants, when to keep local time plus an IANA zone, how gaps and overlaps behave, which database types to use, and how to test with fixed clocks.
Our monitoring service polled depth and klines for 40 symbols every two seconds. During a volatile hour it hit 429 Too much request weight, kept retrying, and earned an HTTP 418 IP ban that also blocked our trading bot on the same IP. Weight-aware limiting and websockets fixed it.
Our thumbnail service hit Errno 24 at about 1,000 concurrent connections even though ulimit -n showed 65535. systemd services never read limits.conf. We set LimitNOFILE in a drop-in and fixed a client leak that was eating descriptors.
Our nightly embedding backfill died on the same few documents every night. A handful of chunks were far over the embedding model's token limit because the chunker split by characters and some files were minified JSON. Token-aware chunking and batching fixed it.
Our support agent started failing with a 400 on roughly one conversation in twelve. The model had begun returning several tool calls in one turn and our loop only answered the first. Answering every tool_call_id in order fixed it.
Our local copy of the BTCUSDT order book slowly diverged from the exchange: crossed books, levels that never disappeared, and a best bid above the best ask. We applied depth diffs without buffering, skipped the snapshot alignment rules and never detected gaps. Following the sync procedure fixed it.
Switching our extraction endpoint to strict structured outputs failed with an invalid schema error. In strict mode every property must be listed in required and additionalProperties must be false. Optional fields became nullable fields and the errors stopped.
Our fraud model scored 0.91 AUC on holdout data and close to random in production. The serving service re-implemented preprocessing in another language: the scaler was refit on each batch, one-hot categories came in a different order and text was not lowercased. Shipping the pipeline with the model fixed it.
Our embedding API logged a tokenizers fork warning thousands of times a day and tokenisation got slower after we enabled Gunicorn preload. The tokenizer was used in the master process before workers forked. Loading it after fork fixed both.
Our rebalancing bot computed order sizes as fractions of a portfolio and about one order in five was rejected with Filter failure: LOT_SIZE or PRICE_FILTER. Quantities and prices must sit on each symbol's step and tick grid. Reading exchangeInfo filters and rounding with Decimal fixed it.
On November 3, 2024, our 01:30 billing run fired at 01:30 CDT and again at 01:30 CST, charging 1,284 subscriptions twice. The in-house scheduler matched local wall-clock time. We moved schedules to UTC and added idempotency keys, a run ledger and a unique constraint.
Customers started seeing checkout warnings that belonged to other people's orders. The cause was a list used as a default argument, created once per worker process and appended to on every request. Here is how we traced it and made it impossible to reintroduce.
Long chats in our internal assistant failed with a context length error even though we kept only the last 20 messages. One pasted log file was bigger than the whole window. We moved to token-based trimming with a reserved output budget.
Data scientists promoted a new churn model by setting the champion alias, as the newer MLflow docs recommend. Our deploy script still loaded models:/churn/Production, which pointed at the old version. Stages are deprecated; we moved the whole pipeline to aliases and added a promotion check.
Every push to our FastAPI service spent seven minutes reinstalling the same Python packages. COPY . . ran before pip install, and a missing .dockerignore let .git changes bust the cache. Reordering layers and a cache mount cut builds to 50 seconds.
Our trading bot's signed requests started failing with error -1021 a few times an hour, then constantly. The VM's clock had drifted almost two seconds because time sync was not running. Fixing NTP, measuring the offset against server time and handling -1021 explicitly ended it.