A VPC line item we had never paid attention to reached 2,400 USD a month. Since February 2024, AWS charges for every public IPv4 address, in use or idle. Our EKS nodes, idle Elastic IPs and load balancers in many subnets added up. Inventory and cleanup cut it by 70 percent.
After moving a PHP 7.4 app to PHP 8.3, our logs grew to 9 GB a day of null-to-string deprecation notices and filled a disk. We found the real call sites, fixed them by intent instead of blind casts, and blocked new ones in CI.
Our storefront BFF pods restarted every few hours with OOMKilled and exit code 137, then fell into CrashLoopBackOff during the holiday peak. A max-old-space-size inherited from the VM days was double the container memory limit. Sizing the heap to the limit and bounding a cache fixed it.
Switching our extraction endpoint to strict structured outputs failed with an invalid schema error. In strict mode every property must be listed in required and additionalProperties must be false. Optional fields became nullable fields and the errors stopped.
After moving repositories into a new GitHub organization, our coverage-comment and release-tag steps failed with 403 Resource not accessible by integration. The GITHUB_TOKEN defaulted to read-only. A scoped permissions block fixed it, and fork PRs needed a different design.
Our price streaming API grew from 180 MB to 1.4 GB of heap in a day and was restarted by the orchestrator every night. A listener was attached to a shared EventEmitter on every request and never removed. Heap snapshots proved it; removing listeners on close fixed it.
Half our Argo CD applications showed OutOfSync seconds after every successful sync, and auto-sync kept fighting the HPA over replica counts. The live objects had fields set by controllers and webhooks that Git never mentioned. Targeted ignoreDifferences and server-side diff fixed it.
Event countdowns showed NaN days for iPhone users while Chrome was fine. Our API sent dates like 2024-12-3 9:05:00, which only some engines accept. We switched to strict ISO 8601 with an offset and stopped trusting Date parsing.
Our fraud model scored 0.91 AUC on holdout data and close to random in production. The serving service re-implemented preprocessing in another language: the scaler was refit on each batch, one-hot categories came in a different order and text was not lowercased. Shipping the pipeline with the model fixed it.
Stock updates from supplier feeds started failing with Deadlock found when trying to get lock during peak hours. Concurrent upsert batches touched the same unique keys in different orders under InnoDB next-key locking. Sorting, smaller batches and a retry fixed it.
After a harmless-looking Express refactor, every Stripe webhook failed verification and paid bookings stayed pending. A global JSON parser consumed the body before constructEvent saw it. How we found it, fixed the route and replayed the missed events.
Our CloudWatch line went from 2,100 to 6,900 USD in a month without new services. Debug logging enabled during an incident was never turned off, and every load balancer health check produced a log line. Finding the top log groups and fixing levels cut ingestion by 81 percent.
A new Android Studio with JDK 21 forced a Gradle upgrade, the Gradle upgrade pulled in AGP 8, and AGP 8 rejected two old Flutter plugins with Namespace not specified. How we untangled the Gradle, AGP, Kotlin and JDK chain.
Our embedding API logged a tokenizers fork warning thousands of times a day and tokenisation got slower after we enabled Gunicorn preload. The tokenizer was used in the master process before workers forked. Loading it after fork fixed both.
An internal reporting service read the user ID and role from a JWT with jwt.decode(), which never checks the signature. A route that bypassed our gateway let any edited token through. We switched to verification with a pinned algorithm, issuer, audience and JWKS rotation.
After upgrading to Next.js 15 our dev console filled with sync dynamic API warnings, and one un-awaited auth helper quietly stopped redirecting logged-out users. How we migrated cookies(), headers() and params safely.
Our rebalancing bot computed order sizes as fractions of a portfolio and about one order in five was rejected with Filter failure: LOT_SIZE or PRICE_FILTER. Quantities and prices must sit on each symbol's step and tick grid. Reading exchangeInfo filters and rounding with Decimal fixed it.
Our API container crashed with ECONNREFUSED on about one in five CI runs and on every fresh laptop setup. depends_on only waited for the Postgres container to start, not to accept connections. A real healthcheck, service_healthy and a small retry loop fixed it.
On November 3, 2024, our 01:30 billing run fired at 01:30 CDT and again at 01:30 CST, charging 1,284 subscriptions twice. The in-house scheduler matched local wall-clock time. We moved schedules to UTC and added idempotency keys, a run ledger and a unique constraint.
Our deploy job worked on main and failed on the production environment with Not authorized to perform sts:AssumeRoleWithWebIdentity. Using a GitHub environment changes the OIDC subject claim. We rewrote the trust policy around exact subjects instead of wildcards.
Customers started seeing checkout warnings that belonged to other people's orders. The cause was a list used as a default argument, created once per worker process and appended to on every request. Here is how we traced it and made it impossible to reintroduce.
After moving our PHP app to a new Ubuntu 24.04 server, every page returned 502 Bad Gateway. Fixing socket ownership got us online, then traffic spikes brought the 502s back from an undersized FPM pool. Here is how we fixed both.
Long chats in our internal assistant failed with a context length error even though we kept only the last 20 messages. One pasted log file was bigger than the whole window. We moved to token-based trimming with a reserved output budget.
A routine iOS release bounced from App Store Connect with ITMS-91053 for UserDefaults, file timestamps and system boot time. Our own code needed reasons, and an old statically linked SDK shipped no manifest at all. Here is the full cleanup.
Our onboarding page created two draft workspaces per visit in development. StrictMode was blamed and nearly disabled, but the real bug was a non-idempotent POST inside an effect with no cleanup, and it was also happening in production.