Our main pipeline took 34 minutes, and engineers batched changes to avoid waiting, which made reviews worse. We cut it to 14 minutes without new hardware magic: measure the critical path, parallelise, cache the right layers, split tests by timing and build only what changed.
A lockfile-only dependency bump passed CI and then broke production. Our cache key hashed package.json, not package-lock.json, so CI restored old node_modules and skipped npm ci. How we found it and rebuilt dependency caching we can trust.
After Docker Hub tightened limits for unauthenticated pulls in April 2025, our self-hosted CI runners started failing a third of builds with toomanyrequests. All runners shared one NAT IP. We authenticated pulls, added a pull-through cache and moved base images into our own registry.
Our React app called https://undefined/v1/... in production while dev worked fine. Vite replaces import.meta.env at build time, and our Docker image set the variable only at runtime. We fixed the build and then moved to runtime config.
One CI run in eight failed somewhere in our checkout E2E suite, never in the same test twice. The causes were fixed sleeps, URL checks that raced navigation and text selectors that matched two buttons. Web-first assertions, role locators and traces fixed it.
After a provider upgrade done on a MacBook, terraform init failed on Linux CI runners with a checksum mismatch against the lock file. The lock file only had hashes for darwin_arm64 packages. Locking all platforms we use fixed it without weakening verification.
Our deploy ran composer update on a 2 GB server until the resolver ran out of memory. Raising the limit let the OOM killer take MySQL instead. Committing the lock file exposed a second bug: it had been resolved on PHP 8.4 for a PHP 8.2 server.
Every deploy of one service failed with another operation is in progress after someone cancelled a pipeline mid-upgrade. The release was stuck in pending-upgrade. We recovered with helm rollback, then added CI concurrency, timeouts and atomic upgrades so it cannot recur.
After moving repositories into a new GitHub organization, our coverage-comment and release-tag steps failed with 403 Resource not accessible by integration. The GITHUB_TOKEN defaulted to read-only. A scoped permissions block fixed it, and fork PRs needed a different design.
Our API container crashed with ECONNREFUSED on about one in five CI runs and on every fresh laptop setup. depends_on only waited for the Postgres container to start, not to accept connections. A real healthcheck, service_healthy and a small retry loop fixed it.
Every push to our FastAPI service spent seven minutes reinstalling the same Python packages. COPY . . ran before pip install, and a missing .dockerignore let .git changes bust the cache. Reordering layers and a cache mount cut builds to 50 seconds.