A worker image built fine but every container exited at once with "no such file or directory" for a file that clearly existed. A Windows checkout had turned the shebang into /bin/bash with a carriage return. We fixed it with .gitattributes and a CI check.
Commitment discounts are the biggest lever on a steady AWS bill and the easiest way to lock in waste. This guide explains Compute and EC2 Instance Savings Plans versus Reserved Instances, coverage versus utilisation, laddering purchases, and when not to commit.
Our React app called https://undefined/v1/... in production while dev worked fine. Vite replaces import.meta.env at build time, and our Docker image set the variable only at runtime. We fixed the build and then moved to runtime config.
After moving our Symfony app to the official php-fpm image, median response time went from 38 ms to 121 ms with no code change. OPcache was never enabled in the new image, and checking it from the CLI hid the problem. Here is how we found it and tuned it.
Our internal MCP server passed every test in the inspector but kept failing in the desktop client with an 'is not valid JSON' error. A startup banner and a debug print went to stdout, which is the protocol channel in stdio transport. Logging to stderr fixed it.
Our order router ran fine as one process. When we scaled to two workers, a third of private API calls failed with EAPI:Invalid nonce. Kraken requires an increasing nonce per API key, and two processes generating millisecond timestamps raced each other. One key per worker fixed it.
Our webhook dispatcher threw ConcurrentModificationException a few times a day and silently skipped deliveries the rest of the time. Two bugs shared one list: removal inside a for-each loop and an admin thread editing it concurrently.
Our document tagger hit HTTP 429 constantly while sending only 40 requests a minute, far below the request limit. The token-per-minute limit was the real ceiling, inflated by an oversized max_tokens and minute-aligned bursts. How we read the headers and built a client-side limiter.
An internal audit found a workflow that used pull_request_target to post preview links and also checked out the pull request's code and ran npm install. Any fork PR could have run code with our deploy token. We split it into an unprivileged build and a privileged comment step.
Users were logged out at random, mostly in the afternoon. Sessions and a fast-growing cache shared one Redis with allkeys-lru, so every hourly cache warm-up evicted idle sessions. Separate DB numbers would not have helped; separate instances did.
We added a new GPU node pool and training pods stayed Pending with Insufficient nvidia.com/gpu, while the nodes sat idle. The NVIDIA device plugin was not running on the new nodes because its DaemonSet did not tolerate their taint, so the nodes never advertised GPUs.
Our live support chat dropped every connection that stayed quiet for a minute, with close code 1006 and no server error. Nginx's 60-second proxy_read_timeout and the load balancer idle timeout were closing idle sockets. A heartbeat plus proxy tuning fixed it.
Our meeting transcripts contained sentences nobody said, usually during silence or hold music, plus the same line repeated a dozen times. Whisper conditions on its own previous output and has no idea what silence is. A VAD pre-filter and stricter decoding thresholds removed most of it.
After a scheduler outage, our daily orders DAG caught up two missed runs and both loaded the same day while another day was skipped. The task computed its date from the wall clock instead of the run's data interval.
A moving-average strategy showed strong backtest returns and nothing like it in paper trading. The backtest computed the signal from a candle's close and filled the order at that same close, a price nobody could trade at once the signal was known. Shifting signals to the next bar fixed the simulation.
EXPLAIN ANALYZE prints a wall of nodes, costs and loops that most of us skim for the word Seq Scan. This guide walks a real slow query node by node: estimates vs actuals, loops, buffers, and the fixes each pattern points to.
A practical guide to blameless postmortems: how to build a timeline, measure impact, find contributing factors instead of a culprit, write action items that get done, and run the review meeting so people tell the truth.
EBS storage kept rising while our clusters stayed the same size. Preview environments created PersistentVolumes with a Retain policy, deleted namespaces left the volumes behind, and a snapshot job never expired anything. We found, verified and deleted 612 volumes and 9,400 snapshots.
E0502 stopped us from updating parent totals inside a HashMap loop, so someone cloned the whole map to make it compile. That clone ran every ten seconds over 1.2 million entries. Here is why the borrow checker was right and the fixes that kept memory flat.
One CI run in eight failed somewhere in our checkout E2E suite, never in the same test twice. The causes were fixed sleeps, URL checks that raced navigation and text selectors that matched two buttons. Web-first assertions, role locators and traces fixed it.
A new module for DNS validation of ACM certificates failed at plan time with an Invalid for_each argument error on fresh environments. The for_each keys came from attributes only known after apply. Static keys with apply-time values made the plan work in one pass.
Receipt uploads to S3 failed with 403 SignatureDoesNotMatch for about 4 percent of users, mostly iPhone photos uploaded from Windows. The URL was signed with one Content-Type and the browser sent another. Reading the CanonicalRequest in the error pinpointed it.
We added cache_control to a 6,000-token system prompt and the bill went up instead of down. Every request wrote the cache and none read it, because the prompt prefix changed on every call. A stable prefix brought cache reads to 92 percent.
An exchange API key with withdrawal rights on a bot host is a hot wallet with extra steps. This guide covers the controls we use for trading bot keys: least-privilege permissions, IP allowlists, sub-accounts, secret storage, rotation, logging without leaking signatures, and incident response.
Every POST to our Django admin returned 403 after we moved TLS termination to a Traefik ingress. Django compared the browser's https Origin with a request it believed was plain http. Two settings fixed it without weakening CSRF.