Pipelines, builds, deployments, release failures.
Most deploy outages come from the same handful of gaps: no readiness check, killed in-flight requests, a migration the old code cannot survive, or no fast way back. This is the checklist we run before every release, with the reasoning behind each item.
We enabled BuildKit's GitHub Actions cache and our multi-stage Go build still took 9 minutes on every run. The default cache export mode only stores layers of the final image, not the builder stage where all the time is spent. mode=max and per-branch scopes cut it to 2 minutes.
After Docker Hub tightened limits for unauthenticated pulls in April 2025, our self-hosted CI runners started failing a third of builds with toomanyrequests. All runners shared one NAT IP. We authenticated pulls, added a pull-through cache and moved base images into our own registry.
Our React app called https://undefined/v1/... in production while dev worked fine. Vite replaces import.meta.env at build time, and our Docker image set the variable only at runtime. We fixed the build and then moved to runtime config.
An internal audit found a workflow that used pull_request_target to post preview links and also checked out the pull request's code and ran npm install. Any fork PR could have run code with our deploy token. We split it into an unprivileged build and a privileged comment step.
A new module for DNS validation of ACM certificates failed at plan time with an Invalid for_each argument error on fresh environments. The for_each keys came from attributes only known after apply. Static keys with apply-time values made the plan work in one pass.
In March 2025 the popular tj-actions/changed-files action was compromised and its version tags were repointed to code that printed CI secrets into workflow logs. Three of our repositories used it by tag. This is how we checked exposure, rotated secrets and moved to SHA pinning.
A cancelled GitLab pipeline killed terraform apply mid-run and left the S3 backend locked. Every plan after that failed with Error acquiring the state lock. How we proved no run was active, unlocked safely, checked for drift, and stopped CI from cancelling applies.
After a provider upgrade done on a MacBook, terraform init failed on Linux CI runners with a checksum mismatch against the lock file. The lock file only had hashes for darwin_arm64 packages. Locking all platforms we use fixed it without weakening verification.
Our deploy ran composer update on a 2 GB server until the resolver ran out of memory. Raising the limit let the OOM killer take MySQL instead. Committing the lock file exposed a second bug: it had been resolved on PHP 8.4 for a PHP 8.2 server.
Our self-hosted LLM box produced three tokens per second on a server with an RTX 4090. ollama ps reported 100% CPU: the container had no access to the GPU. Installing the NVIDIA Container Toolkit and requesting the GPU in Compose brought it to 120 tokens per second.
Our self-managed cluster pulled images from ECR fine after setup and failed every morning with no basic auth credentials. The imagePullSecret held an ECR token that expires after 12 hours. We switched to the kubelet credential provider so nodes fetch fresh tokens themselves.
A one-line ALTER TABLE in a deploy hung for eleven minutes and took the whole site down with it. An idle-in-transaction worker held a lock, the ALTER queued behind it, and every query queued behind the ALTER. Here is the diagnosis and the timeouts we now set.
Every deploy of one service failed with another operation is in progress after someone cancelled a pipeline mid-upgrade. The release was stuck in pending-upgrade. We recovered with helm rollback, then added CI concurrency, timeouts and atomic upgrades so it cannot recur.
A hotfix image pushed from an M3 laptop crash-looped on our x86 servers with exec format error. The image was linux/arm64 only. We moved builds to CI with buildx multi-platform manifests and stopped laptop pushes to the registry.
Training jobs that ran fine on a workstation crashed in containers with a bus error in DataLoader workers. Workers pass batches through shared memory, and containers get only 64 MB of /dev/shm by default. A memory-backed emptyDir in Kubernetes and --shm-size in Docker fixed it.
After moving repositories into a new GitHub organization, our coverage-comment and release-tag steps failed with 403 Resource not accessible by integration. The GITHUB_TOKEN defaulted to read-only. A scoped permissions block fixed it, and fork PRs needed a different design.
Half our Argo CD applications showed OutOfSync seconds after every successful sync, and auto-sync kept fighting the HPA over replica counts. The live objects had fields set by controllers and webhooks that Git never mentioned. Targeted ignoreDifferences and server-side diff fixed it.
Our API container crashed with ECONNREFUSED on about one in five CI runs and on every fresh laptop setup. depends_on only waited for the Postgres container to start, not to accept connections. A real healthcheck, service_healthy and a small retry loop fixed it.
Our deploy job worked on main and failed on the production environment with Not authorized to perform sts:AssumeRoleWithWebIdentity. Using a GitHub environment changes the OIDC subject claim. We rewrote the trust policy around exact subjects instead of wildcards.
Every push to our FastAPI service spent seven minutes reinstalling the same Python packages. COPY . . ran before pip install, and a missing .dockerignore let .git changes bust the cache. Reordering layers and a cache mount cut builds to 50 seconds.