Cloud services, Kubernetes, servers, storage, cost surprises.
When we finally split our bill by environment, development, staging and preview environments were 38 percent of total cloud spend. Most of it ran 24 hours a day for people who work 8. This guide covers schedules, ephemeral environments, right-sizing, data copies and guardrails.
Our managed node group upgrade stalled for two hours on one node. A PodDisruptionBudget with minAvailable equal to the replica count meant no pod could ever be evicted. We switched to maxUnavailable, fixed replica counts and added a pre-upgrade PDB check.
A security group rule added in the console during an incident survived for eight months and was wiped out by an unrelated apply. Drift is normal; unmanaged drift is the problem. This guide covers scheduled drift detection, triage, import blocks and guardrails against click-ops.
A refactor changed one output prefix in an image processing Lambda, and the function started triggering itself. Invocations went from 40,000 a day to 9 million in six hours. We stopped it with reserved concurrency, then separated input and output and added loop guards.
Preview environment namespaces piled up in Terminating and our cleanup job kept failing. The namespace conditions pointed at two causes: a broken metrics APIService that blocked discovery, and custom resources whose operator had been uninstalled. Fixing the causes, not stripping finalizers, cleared them.
A practitioner's reading of AWS's official summary of the October 19-20, 2025 us-east-1 disruption: a race in DynamoDB's DNS automation left an empty record, and recovery cascaded through EC2, NLB and Lambda. What happened per the report, and what we changed.
We moved our EKS batch node pool to Spot and cut its cost by 60 percent. Within a week, nightly jobs failed most nights: a single instance type, no interruption handling and jobs with no checkpoints. Diversified instances, graceful draining and checkpointing made Spot reliable.
Requests decide where a pod is scheduled and how much CPU it is guaranteed; limits decide when it is throttled or killed. This guide explains QoS classes, CFS throttling and OOM kills, and shows how we size containers from real metrics with HPA and VPA.
Our AWS bill jumped by over 9,000 USD in one month and nothing looked broken. The cause was NAT Gateway data processing: batch jobs in private subnets were reading S3 and pulling ECR images through NAT. VPC endpoints cut that line item by 94 percent.
A misconfigured batch job launched 60 GPU instances on a Friday evening. Our monthly budget alert fired on Sunday afternoon, after 14,000 USD was spent. Monthly thresholds and billing data lag make budgets slow. We added daily budgets, anomaly detection and real-time usage guardrails.
We spent a quarter tagging every resource with team and service, then grouped Cost Explorer by tag and saw 92 percent under No tag key. User-defined tags must be activated as cost allocation tags, they apply from activation onward, and some spend cannot be tagged at all.
We moved all EBS volumes from gp2 to gp3 to save 20 percent, as every cost guide recommends. Two days later, nightly reports on our self-managed Postgres took twice as long. The 2 TB gp2 volume had 6,000 IOPS and 250 MiB/s; gp3 defaults are 3,000 and 125. Provisioning both fixed it.
Every deploy and every traffic burst, the first requests to our booking API timed out and Postgres started refusing clients. The handler did heavy setup and opened a fresh connection per invocation. Moving init out of the handler and adding RDS Proxy fixed it.
Commitment discounts are the biggest lever on a steady AWS bill and the easiest way to lock in waste. This guide explains Compute and EC2 Instance Savings Plans versus Reserved Instances, coverage versus utilisation, laddering purchases, and when not to commit.
We added a new GPU node pool and training pods stayed Pending with Insufficient nvidia.com/gpu, while the nodes sat idle. The NVIDIA device plugin was not running on the new nodes because its DaemonSet did not tolerate their taint, so the nodes never advertised GPUs.
EBS storage kept rising while our clusters stayed the same size. Preview environments created PersistentVolumes with a Retain policy, deleted namespaces left the volumes behind, and a snapshot job never expired anything. We found, verified and deleted 612 volumes and 9,400 snapshots.
A new module for DNS validation of ACM certificates failed at plan time with an Invalid for_each argument error on fresh environments. The for_each keys came from attributes only known after apply. Static keys with apply-time values made the plan work in one pass.
Receipt uploads to S3 failed with 403 SignatureDoesNotMatch for about 4 percent of users, mostly iPhone photos uploaded from Windows. The URL was signed with one Content-Type and the browser sent another. Reading the CanonicalRequest in the error pinpointed it.
A cancelled GitLab pipeline killed terraform apply mid-run and left the S3 backend locked. Every plan after that failed with Error acquiring the state lock. How we proved no run was active, unlocked safely, checked for drift, and stopped CI from cancelling applies.
A shared Kubernetes cluster shows up on the cloud bill as a few lines of EC2 and EBS. Turning that into cost per team needs decisions: allocate by requests or usage, what to do with idle capacity, and how to split shared services. This guide covers the model we use and why.
Every deploy of our Go pricing API produced 30 to 60 seconds of 502 and 503 errors. New pods got traffic before they were ready, and old pods exited while the ingress still routed to them. A readiness probe, a preStop sleep and real graceful shutdown made rollouts clean.
Our self-managed cluster pulled images from ECR fine after setup and failed every morning with no basic auth credentials. The imagePullSecret held an ECR token that expires after 12 hours. We switched to the kubelet credential provider so nodes fetch fresh tokens themselves.
Our single-host production server stopped accepting writes at 03:12 with no space left on device. Docker had quietly filled the disk with a 31 GB container log, old images and build cache. Here is how we found each piece and stopped it from coming back.
Our app said the media bucket held 4 TB, while S3 billed us for 31 TB. Versioning kept every overwritten and deleted object as a noncurrent version, and failed multipart uploads left invisible parts behind. Lifecycle rules brought billed storage down to 5.2 TB.
A hotfix image pushed from an M3 laptop crash-looped on our x86 servers with exec format error. The image was linux/arm64 only. We moved builds to CI with buildx multi-platform manifests and stopped laptop pushes to the registry.