Our vLLM deployment had an HPA targeting 70 percent CPU. Under a load spike, requests queued for 40 seconds while CPU sat at 15 percent, so nothing scaled. GPU-bound servers need to scale on queue depth. We switched to KEDA on vLLM's waiting-requests metric and fixed warm-up.
Our managed node group upgrade stalled for two hours on one node. A PodDisruptionBudget with minAvailable equal to the replica count meant no pod could ever be evicted. We switched to maxUnavailable, fixed replica counts and added a pre-upgrade PDB check.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
Our nightly invoice export stopped running and nobody noticed for nine days. One Job had hung on a network call with no timeout, and with concurrencyPolicy Forbid every later schedule was skipped. Deadlines on the Job, timeouts in code and an alert on last success fixed it.
Our queue workers took exactly 30 seconds to stop on every deploy and lost the job they were processing. Exit code 137 in every pod. The Dockerfile used shell-form CMD, so /bin/sh was PID 1 and never forwarded SIGTERM to Node. Exec form plus a real shutdown handler fixed it.
During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
Kubernetes announced the retirement of ingress-nginx, with best-effort maintenance ending in March 2026. This is the plan we used to inventory annotations, map them to Gateway API, run both side by side and cut over host by host without downtime.
Our GPU node pool stayed at eight nodes from Friday night to Monday morning with almost no GPU usage. The cluster autoscaler refused to remove them: notebook pods with local storage, a bare debug pod and a kube-system pod without a PDB. Fixing each blocker cut GPU spend by 38 percent.
Preview environment namespaces piled up in Terminating and our cleanup job kept failing. The namespace conditions pointed at two causes: a broken metrics APIService that blocked discovery, and custom resources whose operator had been uninstalled. Fixing the causes, not stripping finalizers, cleared them.
Our Ray preprocessing job for image embeddings failed after an hour with OutOfMemoryError. Ray schedules tasks by CPU count, so it packed 16 tasks that each needed 3 GB onto nodes with 30 GB. Declaring memory per task, smaller blocks and a right-sized object store fixed it.
We moved our EKS batch node pool to Spot and cut its cost by 60 percent. Within a week, nightly jobs failed most nights: a single instance type, no interruption handling and jobs with no checkpoints. Diversified instances, graceful draining and checkpointing made Spot reliable.
A new subdomain's certificate sat in pending for hours with a self check error: wrong status code 404, expected 200. The challenge solver created its Ingress with an ingress class our controller ignored. Setting the class on the solver and checking the redirect path fixed issuance.
Requests decide where a pod is scheduled and how much CPU it is guaranteed; limits decide when it is throttled or killed. This guide explains QoS classes, CFS throttling and OOM kills, and shows how we size containers from real metrics with HPA and VPA.
GitOps says everything lives in Git, and secrets are the exception that breaks the rule. This guide compares encrypting secrets in Git with SOPS or Sealed Secrets against referencing them from a secret manager with External Secrets Operator or the CSI driver, including rotation and bootstrap.
After we redeployed our MLflow tracking server on Kubernetes, every run older than the deploy showed metrics but no artifacts, and model loading failed. The server wrote artifacts to a local directory in the pod. We moved to proxied artifact storage on S3 and recovered what we could.
During a traffic spike our checkout service stayed at 3 pods while CPU was pegged. kubectl get hpa showed <unknown>/70% for weeks. A sidecar container added by a platform change had no CPU request, so utilisation could not be computed. Requests on every container fixed it.
Our Laravel export workers started at 70 MB and reached 1 GB after six hours, getting slower until Kubernetes killed them mid-job. An always-on query log, a static memo cache and whole-table loads were to blame. We fixed each and added worker limits.
We lowered a rate limit in a ConfigMap, Argo CD reported Synced, and the service kept using the old limit for two days. Environment variables from ConfigMaps are read once at container start, and our file mount used subPath, which never updates. A checksum annotation fixed rollouts.
Most deploy outages come from the same handful of gaps: no readiness check, killed in-flight requests, a migration the old code cannot survive, or no fast way back. This is the checklist we run before every release, with the reasoning behind each item.
After Docker Hub tightened limits for unauthenticated pulls in April 2025, our self-hosted CI runners started failing a third of builds with toomanyrequests. All runners shared one NAT IP. We authenticated pulls, added a pull-through cache and moved base images into our own registry.
We added a new GPU node pool and training pods stayed Pending with Insufficient nvidia.com/gpu, while the nodes sat idle. The NVIDIA device plugin was not running on the new nodes because its DaemonSet did not tolerate their taint, so the nodes never advertised GPUs.
EBS storage kept rising while our clusters stayed the same size. Preview environments created PersistentVolumes with a Retain policy, deleted namespaces left the volumes behind, and a snapshot job never expired anything. We found, verified and deleted 612 volumes and 9,400 snapshots.
DataTransfer-Regional-Bytes grew to 11,000 USD a month, behind only EC2 and RDS. Services spread across three zones talked to each other at random, and Kafka consumers read from leaders in other zones. Topology-aware routing and follower fetching cut it by 68 percent.
A shared Kubernetes cluster shows up on the cloud bill as a few lines of EC2 and EBS. Turning that into cost per team needs decisions: allocate by requests or usage, what to do with idle capacity, and how to split shared services. This guide covers the model we use and why.
Every deploy of our Go pricing API produced 30 to 60 seconds of 502 and 503 errors. New pods got traffic before they were ready, and old pods exited while the ingress still routed to them. A readiness probe, a preStop sleep and real graceful shutdown made rollouts clean.