DNS, TLS/SSL, proxies, load balancers, CDN, timeouts.
After a framework upgrade, our CDN cache hit ratio for static assets dropped from 97 to 41 percent and origin data transfer doubled. The new session middleware attached Set-Cookie and Cache-Control private to every response, including JavaScript bundles. Scoping middleware fixed it.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
Kubernetes announced the retirement of ingress-nginx, with best-effort maintenance ending in March 2026. This is the plan we used to inventory annotations, map them to Gateway API, run both side by side and cut over host by host without downtime.
A summary of Cloudflare's official postmortem for the November 18, 2025 outage: a database permission change doubled a generated feature file, which crossed a hard limit and crashed the proxy. Plus the lessons we are applying to our own config pipelines.
Most HTTP bugs can be pinned down with one tool you already have. This guide shows the curl options we use daily: verbose output, header-only requests, --resolve and --connect-to for testing servers before DNS changes, timing breakdowns, protocol versions and TLS checks.
A new subdomain's certificate sat in pending for hours with a self check error: wrong status code 404, expected 200. The challenge solver created its Ingress with an ingress class our controller ignored. Setting the class on the solver and checking the redirect path fixed issuance.
A low but stubborn rate of 502s at the load balancer and socket hang up errors between services turned out to be one race: Node closed idle keep-alive connections after 5 seconds while its callers kept reusing them.
We moved our app to a new host and switched DNS on a Saturday. A day later almost half of requests still reached the old server. A 24-hour TTL, a nameserver change with a 48-hour delegation TTL and an old zone left online kept stale answers alive. This is the postmortem.
Our bot believed two orders were still open when they had filled hours earlier. The exchange closes websocket connections after 24 hours; our reconnect logic resubscribed but never checked what happened while disconnected. Reconnect with backoff plus REST reconciliation fixed the state drift.
After we tightened DMARC to quarantine, Gmail started filing our password resets and receipts as spam. Both SPF and DKIM passed, but for amazonses.com, not our domain. Here is how we read the headers, aligned both, and kept it that way.
Our live support chat dropped every connection that stayed quiet for a minute, with close code 1006 and no server error. Nginx's 60-second proxy_read_timeout and the load balancer idle timeout were closing idle sockets. A heartbeat plus proxy tuning fixed it.
Our chat assistant streamed tokens smoothly on localhost but in production the whole answer appeared after 14 seconds. nginx was buffering the event stream and gzip was holding bytes. How we proved it with curl and fixed it without breaking other routes.
DataTransfer-Regional-Bytes grew to 11,000 USD a month, behind only EC2 and RDS. Services spread across three zones talked to each other at random, and Kafka consumers read from leaders in other zones. Topology-aware routing and follower fetching cut it by 68 percent.
Uploading a PDF larger than 1 MB failed with a vague error in our claims portal. Nginx returned 413, then PHP silently dropped files, then Laravel rejected them. Four layers each had their own size limit. Here is how we aligned them and tested every boundary.
Every deploy of our Go pricing API produced 30 to 60 seconds of 502 and 503 errors. New pods got traffic before they were ready, and old pods exited while the ingress still routed to them. A readiness probe, a preStop sleep and real graceful shutdown made rollouts clean.
Moving our SPA from cookie auth to bearer tokens broke every API call with a CORS error. The Authorization header triggered an OPTIONS preflight, our nginx auth check rejected it, and error responses carried no CORS headers.
Minutes after we proxied our marketing site through Cloudflare, every visitor hit ERR_TOO_MANY_REDIRECTS. Cloudflare's Flexible mode spoke plain HTTP to an origin that redirected all HTTP to HTTPS. Full (strict) with an origin certificate ended the loop.
Our customer portal went dark at 07:12 when its Let's Encrypt certificate expired. Certbot's timer had run twice a day for a month and failed every time, because a firewall change blocked port 80. Here is how we traced it, fixed it and made expiry impossible to miss.
Our API certificate renewal looked perfect in desktop Chrome, yet the Android app, curl and a partner's Java service all rejected it. The server sent only the leaf certificate without the intermediate. Serving the full chain fixed every client at once.
A VPC line item we had never paid attention to reached 2,400 USD a month. Since February 2024, AWS charges for every public IPv4 address, in use or idle. Our EKS nodes, idle Elastic IPs and load balancers in many subnets added up. Inventory and cleanup cut it by 70 percent.
After moving our PHP app to a new Ubuntu 24.04 server, every page returned 502 Bad Gateway. Fixing socket ownership got us online, then traffic spikes brought the 502s back from an undersized FPM pool. Here is how we fixed both.
Our trading bot's signed requests started failing with error -1021 a few times an hour, then constantly. The VM's clock had drifted almost two seconds because time sync was not running. Fixing NTP, measuring the offset against server time and handling -1021 explicitly ended it.