A new bot received order updates for one hour after start, then nothing, while the websocket stayed connected. The listenKey expires after 60 minutes without a keepalive. We fixed keepalives and expiry handling, then moved to the WebSocket API subscription before listenKey retirement.
Our onboarding page created two draft workspaces per visit in development. StrictMode was blamed and nearly disabled, but the real bug was a non-idempotent POST inside an effect with no cleanup, and it was also happening in production.
Product reviews containing emoji failed with error 1366 after we upgraded MySQL to 8.0, while a nightly job quietly turned them into question marks. Legacy utf8mb3 tables, a wrong connection charset and an index length limit were all involved.
Our job board listing took 1.8 seconds and ran 312 queries per page. Blade was lazy loading company, tags and application counts for every row. We fixed it with with(), withCount() and made lazy loading throw in development.
Every deploy of our Go pricing API produced 30 to 60 seconds of 502 and 503 errors. New pods got traffic before they were ready, and old pods exited while the ingress still routed to them. A readiness probe, a preStop sleep and real graceful shutdown made rollouts clean.
Every value in our orders API was a bound parameter, except the sort column, which was pasted into ORDER BY from the query string. A pentest proved it was injectable. We replaced it with an allowlist map and added tests that fuzz the sort input.
Every ten minutes our catalog API spiked to 4-second responses and the database hit 100% CPU. A hot Redis key expired and 600 requests rebuilt it at once. Single-flight, a short Redis lock, stale-while-revalidate and TTL jitter fixed it.
Our text classification service crashed with CUDA out of memory every few hours, always on a big batch. Logits kept for drift monitoring were holding autograd graphs alive, batches were sized by count not tokens, and the allocator was fragmented. How we measured each one and fixed it.
Most review comments are about naming and style, while the bugs walk past. How we review for correctness instead: follow the data, probe edge cases and error paths, look for races and trust boundaries, keep PRs small and hand style to machines.
Our embedding API logged a tokenizers fork warning thousands of times a day and tokenisation got slower after we enabled Gunicorn preload. The tokenizer was used in the master process before workers forked. Loading it after fork fixed both.
A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.
After a provider upgrade done on a MacBook, terraform init failed on Linux CI runners with a checksum mismatch against the lock file. The lock file only had hashes for darwin_arm64 packages. Locking all platforms we use fixed it without weakening verification.
Our versioning workflow created a tag and pushed it, and the release workflow that listens for tags never started. Events created with the default GITHUB_TOKEN do not trigger new workflow runs. We switched to a GitHub App token for the push and made the release explicit with workflow_call.
After we added merge request rules to a few jobs, every push to an open merge request started two pipelines, doubling runner usage. Job-level rules matched both the branch push and the merge request event. A workflow:rules block that picks one pipeline type fixed it.
EBS storage kept rising while our clusters stayed the same size. Preview environments created PersistentVolumes with a Retain policy, deleted namespaces left the volumes behind, and a snapshot job never expired anything. We found, verified and deleted 612 volumes and 9,400 snapshots.
Commitment discounts are the biggest lever on a steady AWS bill and the easiest way to lock in waste. This guide explains Compute and EC2 Instance Savings Plans versus Reserved Instances, coverage versus utilisation, laddering purchases, and when not to commit.
After rebuilding our inference image, models silently ran on CPU on half the GPU fleet. The new PyTorch wheel targets CUDA 12, and older nodes still had a driver that only supports CUDA 11. We aligned drivers, added a startup check that fails without a GPU, and pinned images.
A cancelled GitLab pipeline killed terraform apply mid-run and left the S3 backend locked. Every plan after that failed with Error acquiring the state lock. How we proved no run was active, unlocked safely, checked for drift, and stopped CI from cancelling applies.
Minutes after we proxied our marketing site through Cloudflare, every visitor hit ERR_TOO_MANY_REDIRECTS. Cloudflare's Flexible mode spoke plain HTTP to an origin that redirected all HTTP to HTTPS. Full (strict) with an origin certificate ended the loop.
Most HTTP bugs can be pinned down with one tool you already have. This guide shows the curl options we use daily: verbose output, header-only requests, --resolve and --connect-to for testing servers before DNS changes, timing breakdowns, protocol versions and TLS checks.
A developer committed a .env file with a live AWS access key, then deleted it in the next commit. The key stayed in history. We revoked it within 40 minutes, checked CloudTrail, rewrote history with git filter-repo and added scanning so it cannot happen again.
Our customer portal went dark at 07:12 when its Let's Encrypt certificate expired. Certbot's timer had run twice a day for a month and failed every time, because a firewall change blocked port 80. Here is how we traced it, fixed it and made expiry impossible to miss.
After a scheduler outage, our daily orders DAG caught up two missed runs and both loaded the same day while another day was skipped. The task computed its date from the wall clock instead of the run's data interval.
Our thumbnail service hit Errno 24 at about 1,000 concurrent connections even though ulimit -n showed 65535. systemd services never read limits.conf. We set LimitNOFILE in a drop-in and fixed a client leak that was eating descriptors.
Our nightly report export ran perfectly when we typed it, and failed every night as a systemd service with node: command not found. Services get no login shell, no .bashrc and a working directory of /. Here is how we made the unit self-contained.