Docker build reinstalls every dependency on each commit: COPY order and cache busting

Our payments-reconciliation API is a FastAPI service with about 90 Python dependencies, including pandas, pyarrow and cryptography. CI builds its image on every push. Over the summer the image build went from "a bit slow" to seven to nine minutes on every commit, even commits that only changed a log message. Developers started batching changes to avoid waiting, which is the opposite of what we want.

The BuildKit output in CI showed that nothing after the first few steps was ever cached:

#7 [3/6] COPY . /app
#7 DONE 2.4s

#8 [4/6] RUN pip install --no-cache-dir -r requirements.txt
#8 0.912 Collecting fastapi==0.115.0 (from -r requirements.txt (line 1))
#8 1.204   Downloading fastapi-0.115.0-py3-none-any.whl.metadata (27 kB)
...
#8 412.6 Successfully installed anyio-4.6.0 cryptography-43.0.1 fastapi-0.115.0 pandas-2.2.3 ...
#8 DONE 418.3s

Step 4 never showed CACHED, not even on two consecutive pushes with identical requirements.txt.

Reproducing it

Locally it was just as easy to see:

docker build -t recon-api .           # full build
echo "# touch" >> app/main.py
docker build -t recon-api .           # pip install runs again from scratch

What we ruled out

• CI not restoring the cache. We use docker/build-push-action@v6 with cache-from and cache-to set to type=gha. The early steps (FROM, apt-get) did show CACHED, so the cache was being imported.
• A changing base image. The FROM line was pinned to python:3.12.7-slim-bookworm, and the digest in the logs was the same between runs.
• Unpinned dependencies. Every line in requirements.txt was pinned with ==; the file did not change between the commits we compared.

Root cause

The Dockerfile copied the whole repository before installing dependencies:

FROM python:3.12.7-slim-bookworm
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
    && rm -rf /var/lib/apt/lists/*
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

COPY . /app checksums every file in the build context. Any change anywhere in the repository changes that checksum, which invalidates the COPY layer and every layer after it, including the expensive pip install. To make it worse, there was no .dockerignore, so the build context included .git/ (whose contents change on every commit), .pytest_cache/, local .venv/ directories on developer machines and a 180 MB tests/fixtures/ folder. Even a commit that changed nothing in the app changed .git/, so the cache was busted on literally every build.

What is happening under the hood

Docker builds an image as a chain of layers, one per instruction. For each instruction the builder computes a cache key from the parent layer plus the instruction itself. For RUN, the key is the command string. For COPY and ADD, it also includes a checksum of the files being copied. If the key matches a layer in the cache, the builder reuses it; if not, it executes the instruction and every instruction after it must be executed too, because their parent changed.

So the rule is to order instructions from least frequently changing to most frequently changing: base image, system packages, dependency manifests, dependency install, then application code.

The build context is everything sent to the builder, filtered by .dockerignore. Files excluded there are invisible to COPY . ., so they cannot bust the cache.

Environment

Minimal reproduction

mkdir cache-demo && cd cache-demo && git init -q
printf 'requests==2.32.3\n' > requirements.txt
printf 'print("hi")\n' > main.py
cat > Dockerfile <<'EOF'
FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
EOF
docker build -q -t demo . && git add -A && git commit -qm one
docker build --progress=plain -t demo . 2>&1 | grep -E "pip install|CACHED"
# pip install runs again because .git/ changed after the commit

Who is affected

Every Dockerfile that copies the full source tree before installing dependencies, for any language: npm ci after COPY . ., composer install, go mod download, bundle install, mvn dependency:go-offline. It is more visible in CI, where builds start from a fresh runner and depend entirely on the imported cache.

The fix

Three changes. First, copy only the dependency manifest, install, then copy the code:

# syntax=docker/dockerfile:1
FROM python:3.12.7-slim-bookworm
WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
    && rm -rf /var/lib/apt/lists/*

# 1. dependency manifest only: this layer changes only when requirements.txt changes
COPY requirements.txt .

# 2. install with a persistent pip cache mount
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

# 3. application code last
COPY app/ ./app/

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

We dropped --no-cache-dir because the cache mount is not part of the image, so it no longer bloats the final layer; it just gives pip a warm wheel cache when the layer must be rebuilt.

Second, a .dockerignore that sends only what the image needs:

.git
.github
.venv
**/__pycache__
**/*.pyc
.pytest_cache
.mypy_cache
tests/
docs/
*.md
.env
.env.*
Dockerfile
docker-compose*.yml

Third, we copy app/ explicitly instead of ., which makes it obvious what goes into the image.

Alternatives and trade-offs

• Multi-stage build with a wheel stage. Build wheels in a builder stage that has build-essential, then copy only the wheels into a slim runtime stage. Smaller final image (ours went from 1.1 GB to 640 MB), and the same ordering rules apply inside each stage.
• Cache mounts in CI. Cache mounts live in the builder's local storage. They are not exported by cache-to: type=gha, so on fresh hosted runners they start empty. The layer ordering is what saves time in CI; the cache mount helps on developer machines and self-hosted runners.
• --mount=type=bind for the manifest. You can bind-mount requirements.txt into the RUN instead of copying it. It avoids one layer but makes the Dockerfile harder to read for newcomers; we kept COPY.

How to test the fix

Build twice with a code-only change in between and check that the install step is cached:

docker build --progress=plain -t recon-api . > /dev/null 2>&1
echo "# touch" >> app/main.py
docker build --progress=plain -t recon-api . 2>&1 | grep -A1 "pip install"
# expect: "#8 [4/5] RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt"
#         "#8 CACHED"

And check the context size the builder receives:

docker build --progress=plain -t recon-api . 2>&1 | grep "transferring context"
# before: transferring context: 409.81MB
# after:  transferring context: 1.94MB

Confirming in production

Across the first 120 CI runs after the change:

The 9 percent of builds that reinstalled dependencies were exactly the ones that changed requirements.txt, which is what we want.

Regression guard

• A CI step fails the build if the context grows past 20 MB, by parsing the transferring context line.
• hadolint runs on the Dockerfile in pre-commit.
• The pull request template asks whether a change adds files to the image, so .dockerignore gets updated alongside.

Security implications

The fix improves security. Without .dockerignore, our build context included .env files from developer machines and the full .git/ history, and COPY . /app put them in the image. Anyone who could pull the image could read them. We scanned old image tags, found a .env with a sandbox API key (sk_test_xxx style) in three of them, rotated the key and deleted those tags. Cache mounts do not end up in the image, but do not mount secrets as caches; use --mount=type=secret for credentials.

Prevention checklist

• Order Dockerfile instructions from least to most frequently changing.
• Copy dependency manifests and lockfiles first, install, then copy source.
• Keep a .dockerignore that excludes .git, local environments, tests and secrets.
• Watch the build context size and the CACHED ratio in CI.
• Use cache mounts for package manager caches, and secret mounts for credentials.

References

• Docker documentation: Docker build cache — https://docs.docker.com/build/cache/
• Docker documentation: Optimize cache usage in builds — https://docs.docker.com/build/cache/optimize/
• Docker documentation: Build context and .dockerignore files — https://docs.docker.com/build/concepts/context/
• Docker documentation: Dockerfile reference, RUN --mount — https://docs.docker.com/reference/dockerfile/#run---mount
• Docker documentation: Cache management with GitHub Actions — https://docs.docker.com/build/ci/github-actions/cache/