Cloud services, Kubernetes, servers, storage, cost surprises.
Our storefront BFF pods restarted every few hours with OOMKilled and exit code 137, then fell into CrashLoopBackOff during the holiday peak. A max-old-space-size inherited from the VM days was double the container memory limit. Sizing the heap to the limit and bounding a cache fixed it.
Our CloudWatch line went from 2,100 to 6,900 USD in a month without new services. Debug logging enabled during an incident was never turned off, and every load balancer health check produced a log line. Finding the top log groups and fixing levels cut ingestion by 81 percent.
Our deploy job worked on main and failed on the production environment with Not authorized to perform sts:AssumeRoleWithWebIdentity. Using a GitHub environment changes the OIDC subject claim. We rewrote the trust policy around exact subjects instead of wildcards.