A slow accounting API pushed our PayPal webhook handler past the nginx timeout. PayPal saw a 504 and retried, while the first request was still running, and 41 wallets were topped up twice. Postmortem plus the idempotent design we moved to.
Our chat assistant streamed tokens smoothly on localhost but in production the whole answer appeared after 14 seconds. nginx was buffering the event stream and gzip was holding bytes. How we proved it with curl and fixed it without breaking other routes.
Serving Llama 3.1 8B with vLLM on a 24 GB GPU crashed at startup with a KV cache error. By default vLLM sizes for the model's full 128k context, and after loading weights there was room for about 45k tokens of KV cache. Setting max_model_len to what we need fixed it.
In March 2025 the popular tj-actions/changed-files action was compromised and its version tags were repointed to code that printed CI secrets into workflow logs. Three of our repositories used it by tag. This is how we checked exposure, rotated secrets and moved to SHA pinning.
After importing 410,000 tickets with their original ids, most new tickets failed with duplicate key value violates unique constraint tickets_pkey, while a few succeeded. The id sequence was still at 1. Here is why, and how we resynced every sequence safely.
Every ten minutes our catalog API spiked to 4-second responses and the database hit 100% CPU. A hot Redis key expired and 600 requests rebuilt it at once. Single-flight, a short Redis lock, stale-while-revalidate and TTL jitter fixed it.
DataTransfer-Regional-Bytes grew to 11,000 USD a month, behind only EC2 and RDS. Services spread across three zones talked to each other at random, and Kafka consumers read from leaders in other zones. Topology-aware routing and follower fetching cut it by 68 percent.
A cancelled GitLab pipeline killed terraform apply mid-run and left the S3 backend locked. Every plan after that failed with Error acquiring the state lock. How we proved no run was active, unlocked safely, checked for drift, and stopped CI from cancelling applies.
Uploading a PDF larger than 1 MB failed with a vague error in our claims portal. Nginx returned 413, then PHP silently dropped files, then Laravel rejected them. Four layers each had their own size limit. Here is how we aligned them and tested every boundary.
Small tenants in our RAG product got 2 or 3 chunks back instead of 10, and sometimes none. HNSW finds ef_search candidates first and applies the tenant filter afterwards. Iterative index scans in pgvector 0.8 plus partial indexes for big tenants fixed recall.
A practical guide to time handling that survives DST switches: when to store UTC instants, when to keep local time plus an IANA zone, how gaps and overlaps behave, which database types to use, and how to test with fixed clocks.
Login and checkout forms started failing with 419 Page Expired the day we put two Laravel nodes behind a TLS-terminating load balancer. Untrusted proxy headers, secure cookies and file sessions on each node combined into one confusing error.
After upgrading to PyTorch 2.6, every checkpoint load in our evaluation pipeline failed with Weights only load failed. The default for weights_only in torch.load flipped to True, and our checkpoints contained a NumPy scalar. We fixed the checkpoints instead of turning the safety off.
After a provider upgrade done on a MacBook, terraform init failed on Linux CI runners with a checksum mismatch against the lock file. The lock file only had hashes for darwin_arm64 packages. Locking all platforms we use fixed it without weakening verification.
Our PayPal checkout button rendered in development and staging but was an empty box in production. A new Content-Security-Policy set in Nginx blocked the PayPal SDK script, its iframes and its API calls. We fixed it per directive, with a nonce and a report-only rollout.
Every engineer who downloaded our internal deploy helper got a "damaged" dialog. The app was fine; a CI step modified the bundle after signing and Gatekeeper checked the quarantined copy. We now sign, notarize and staple in the right order.
Our monitoring service polled depth and klines for 40 symbols every two seconds. During a volatile hour it hit 429 Too much request weight, kept retrying, and earned an HTTP 418 IP ban that also blocked our trading bot on the same IP. Weight-aware limiting and websockets fixed it.
Our deploy ran composer update on a 2 GB server until the resolver ran out of memory. Raising the limit let the OOM killer take MySQL instead. Committing the lock file exposed a second bug: it had been resolved on PHP 8.4 for a PHP 8.2 server.
A shared Kubernetes cluster shows up on the cloud bill as a few lines of EC2 and EBS. Turning that into cost per team needs decisions: allocate by requests or usage, what to do with idle capacity, and how to split shared services. This guide covers the model we use and why.
Bumping targetSdk to 35 put our toolbar under the status bar and the Pay button under the navigation bar on Android 15 devices. Edge-to-edge is now enforced, so we handled window insets properly in both Views and Compose screens.
A good bug report lets someone who has never seen your screen reproduce the problem in minutes. The anatomy we use: environment, exact steps, expected versus actual, evidence, a minimal reproduction, and severity kept separate from priority.
Every deploy of our Go pricing API produced 30 to 60 seconds of 502 and 503 errors. New pods got traffic before they were ready, and old pods exited while the ingress still routed to them. A readiness probe, a preStop sleep and real graceful shutdown made rollouts clean.
Our self-hosted LLM box produced three tokens per second on a server with an RTX 4090. ollama ps reported 100% CPU: the container had no access to the GPU. Installing the NVIDIA Container Toolkit and requesting the GPU in Compose brought it to 120 tokens per second.
Our PDF invoices and the amounts charged through Stripe disagreed by a cent on 214 invoices in one month. Floats, toFixed and rounding at every step were to blame. We moved to integer minor units with one explicit rounding point.
Our enrichment consumers rebalanced every few minutes, lag climbed past two million messages and some records were processed twice. A slow external API pushed batch processing past max.poll.interval.ms. How we proved it, fixed the poll loop and made rebalances cheap.