Our queue workers took exactly 30 seconds to stop on every deploy and lost the job they were processing. Exit code 137 in every pod. The Dockerfile used shell-form CMD, so /bin/sh was PID 1 and never forwarded SIGTERM to Node. Exec form plus a real shutdown handler fixed it.
During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
Running the test suite inside our app container dropped every table in the development database. Docker Compose injected DB_DATABASE as a real environment variable, and phpunit.xml's env entries do not override existing variables. Here is the full precedence chain and three layers of defence.
We enabled BuildKit's GitHub Actions cache and our multi-stage Go build still took 9 minutes on every run. The default cache export mode only stores layers of the final image, not the builder stage where all the time is spent. mode=max and per-branch scopes cut it to 2 minutes.
A worker image built fine but every container exited at once with "no such file or directory" for a file that clearly existed. A Windows checkout had turned the shebang into /bin/bash with a carriage return. We fixed it with .gitattributes and a CI check.
Our React app called https://undefined/v1/... in production while dev worked fine. Vite replaces import.meta.env at build time, and our Docker image set the variable only at runtime. We fixed the build and then moved to runtime config.
After moving our Symfony app to the official php-fpm image, median response time went from 38 ms to 121 ms with no code change. OPcache was never enabled in the new image, and checking it from the CLI hid the problem. Here is how we found it and tuned it.
Our self-hosted LLM box produced three tokens per second on a server with an RTX 4090. ollama ps reported 100% CPU: the container had no access to the GPU. Installing the NVIDIA Container Toolkit and requesting the GPU in Compose brought it to 120 tokens per second.
After rebuilding our inference image, models silently ran on CPU on half the GPU fleet. The new PyTorch wheel targets CUDA 12, and older nodes still had a driver that only supports CUDA 11. We aligned drivers, added a startup check that fails without a GPU, and pinned images.
Our single-host production server stopped accepting writes at 03:12 with no space left on device. Docker had quietly filled the disk with a 31 GB container log, old images and build cache. Here is how we found each piece and stopped it from coming back.
A hotfix image pushed from an M3 laptop crash-looped on our x86 servers with exec format error. The image was linux/arm64 only. We moved builds to CI with buildx multi-platform manifests and stopped laptop pushes to the registry.
Training jobs that ran fine on a workstation crashed in containers with a bus error in DataLoader workers. Workers pass batches through shared memory, and containers get only 64 MB of /dev/shm by default. A memory-backed emptyDir in Kubernetes and --shm-size in Docker fixed it.
Every push to our FastAPI service spent seven minutes reinstalling the same Python packages. COPY . . ran before pip install, and a missing .dockerignore let .git changes bust the cache. Reordering layers and a cache mount cut builds to 50 seconds.