Semantic search in our docs assistant took 1.9 seconds and returned odd rankings. The query sorted by L2 distance while the HNSW index was built for cosine, so Postgres scanned every row. Matching the operator to the opclass brought it to 14 ms.
After rebuilding our inference image, models silently ran on CPU on half the GPU fleet. The new PyTorch wheel targets CUDA 12, and older nodes still had a driver that only supports CUDA 11. We aligned drivers, added a startup check that fails without a GPU, and pinned images.
A grid trading bot written in TypeScript sent quantities like 0.30000000000000004 and got error -1111. Binary floating point cannot represent most decimal fractions, and serialising numbers to strings exposed it. A decimal library end to end and string serialisation fixed it.
Our nightly embedding backfill died on the same few documents every night. A handful of chunks were far over the embedding model's token limit because the chunker split by characters and some files were minified JSON. Token-aware chunking and batching fixed it.
Every deploy of one service failed with another operation is in progress after someone cancelled a pipeline mid-upgrade. The release was stuck in pending-upgrade. We recovered with helm rollback, then added CI concurrency, timeouts and atomic upgrades so it cannot recur.
Our support agent started failing with a 400 on roughly one conversation in twelve. The model had begun returning several tool calls in one turn and our loop only answered the first. Answering every tool_call_id in order fixed it.
Our app said the media bucket held 4 TB, while S3 billed us for 31 TB. Versioning kept every overwritten and deleted object as a noncurrent version, and failed multipart uploads left invisible parts behind. Lifecycle rules brought billed storage down to 5.2 TB.
Training jobs that ran fine on a workstation crashed in containers with a bus error in DataLoader workers. Workers pass batches through shared memory, and containers get only 64 MB of /dev/shm by default. A memory-backed emptyDir in Kubernetes and --shm-size in Docker fixed it.
Our local copy of the BTCUSDT order book slowly diverged from the exchange: crossed books, levels that never disappeared, and a best bid above the best ask. We applied depth diffs without buffering, skipped the snapshot alignment rules and never detected gaps. Following the sync procedure fixed it.
A VPC line item we had never paid attention to reached 2,400 USD a month. Since February 2024, AWS charges for every public IPv4 address, in use or idle. Our EKS nodes, idle Elastic IPs and load balancers in many subnets added up. Inventory and cleanup cut it by 70 percent.
Switching our extraction endpoint to strict structured outputs failed with an invalid schema error. In strict mode every property must be listed in required and additionalProperties must be false. Optional fields became nullable fields and the errors stopped.
Half our Argo CD applications showed OutOfSync seconds after every successful sync, and auto-sync kept fighting the HPA over replica counts. The live objects had fields set by controllers and webhooks that Git never mentioned. Targeted ignoreDifferences and server-side diff fixed it.
Our fraud model scored 0.91 AUC on holdout data and close to random in production. The serving service re-implemented preprocessing in another language: the scaler was refit on each batch, one-hot categories came in a different order and text was not lowercased. Shipping the pipeline with the model fixed it.
Our CloudWatch line went from 2,100 to 6,900 USD in a month without new services. Debug logging enabled during an incident was never turned off, and every load balancer health check produced a log line. Finding the top log groups and fixing levels cut ingestion by 81 percent.
Our embedding API logged a tokenizers fork warning thousands of times a day and tokenisation got slower after we enabled Gunicorn preload. The tokenizer was used in the master process before workers forked. Loading it after fork fixed both.
Our rebalancing bot computed order sizes as fractions of a portfolio and about one order in five was rejected with Filter failure: LOT_SIZE or PRICE_FILTER. Quantities and prices must sit on each symbol's step and tick grid. Reading exchangeInfo filters and rounding with Decimal fixed it.
Our deploy job worked on main and failed on the production environment with Not authorized to perform sts:AssumeRoleWithWebIdentity. Using a GitHub environment changes the OIDC subject claim. We rewrote the trust policy around exact subjects instead of wildcards.
Long chats in our internal assistant failed with a context length error even though we kept only the last 20 messages. One pasted log file was bigger than the whole window. We moved to token-based trimming with a reserved output budget.
Data scientists promoted a new churn model by setting the champion alias, as the newer MLflow docs recommend. Our deploy script still loaded models:/churn/Production, which pointed at the old version. Stages are deprecated; we moved the whole pipeline to aliases and added a promotion check.
Our trading bot's signed requests started failing with error -1021 a few times an hour, then constantly. The VM's clock had drifted almost two seconds because time sync was not running. Fixing NTP, measuring the offset against server time and handling -1021 explicitly ended it.