Docker Buildx cache barely helps multi-stage builds in CI: cache mode=min

Our API image is a classic multi-stage build: a builder stage with the Go toolchain that downloads modules and compiles, and a small distroless final stage that copies the binary. Builds in GitHub Actions took around 9 minutes. We added the BuildKit cache backend for GitHub Actions, expecting a big improvement. Builds still took 8 to 9 minutes.

The build log showed the problem:

#12 [builder 4/7] RUN go mod download
#12 CACHED                                          <- only sometimes
#14 [builder 6/7] RUN go build -o /out/api ./cmd/api
#14 DONE 312.4s
#16 [stage-1 2/2] COPY --from=builder /out/api /api
#16 CACHED

The tiny final stage was cached. The expensive builder steps were rebuilt on most runs.

Reproducing it

Running the same workflow twice on the same commit with no changes: the second run should have been fully cached, and it rebuilt the builder stage. The relevant workflow step:

- uses: docker/build-push-action@v6
  with:
    push: true
    tags: ghcr.io/example-org/api:${{ github.sha }}
    cache-from: type=gha
    cache-to: type=gha

What we ruled out

• Cache size limits. The repository cache was far below its limit.
• COPY . . invalidating layers. The Dockerfile copied go.mod and go.sum before the source, the right order for module caching.
• Different runners or architectures. All runs used the same runner type and platform.

Root cause

cache-to defaults to mode=min, which exports cache only for the layers that end up in the final image. In a multi-stage build, the builder stage's layers are not in the final image, so they were never exported. The "CACHED" we sometimes saw for go mod download came from the runner's local state on reruns, not from the remote cache.

What is happening under the hood

BuildKit can import cache from and export cache to external backends: a registry, the GitHub Actions cache service, S3, Azure Blob, or local directories. When exporting, it has two modes:

• mode=min (default): export only layers of the resulting image. Small and fast to upload. Good for single-stage builds.
• mode=max: export all layers of all intermediate stages. Larger and slower to upload, and the only way to reuse expensive work in builder stages across runs.

For the GitHub Actions backend, cache entries are also grouped by scope. Without a scope, all builds in the repository share one, so two different images or branches overwrite each other's cache.

Environment

Minimal reproduction

FROM golang:1.24 AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/api ./cmd/api

FROM gcr.io/distroless/static-debian12
COPY --from=builder /out/api /api
ENTRYPOINT ["/api"]

Build twice with --cache-to type=gha (mode=min) on fresh runners: the builder stage is rebuilt each time.

The fix

1. Export all stages with mode=max and scope the cache per image and branch.

- uses: docker/build-push-action@v6
  with:
    push: true
    tags: ghcr.io/example-org/api:${{ github.sha }}
    cache-from: |
      type=gha,scope=api-${{ github.ref_name }}
      type=gha,scope=api-main
    cache-to: type=gha,scope=api-${{ github.ref_name }},mode=max

Feature branches read from their own scope first and fall back to main's cache, so a new branch starts warm.

2. Cache mounts for the Go caches inside the builder. Layer cache cannot help go build when any source file changes, but the Go build cache can reuse compiled packages:

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -o /out/api ./cmd/api

Cache mounts are not exported to the remote cache by the build action, so on ephemeral runners they mainly help repeated builds in the same job. We combined them with a small step that saves and restores those directories through the Actions cache, using a widely used helper action for BuildKit cache mounts.

Alternatives and trade-offs

• Registry cache (type=registry,ref=ghcr.io/example-org/api:buildcache,mode=max). Not subject to the Actions cache size limit and shared across CI systems. Uploads count against registry storage. We use it for our largest images.
• Self-hosted runners with a persistent BuildKit. Local cache survives between jobs; fastest, with the cost of managing runners.
• mode=max everywhere. Cache exports for big images with many stages can take longer than the time saved. Measure per image.

How to test the fix

Two consecutive runs on fresh runners with no code change, then a run with a one-line change in a Go file:

run                       total   builder stage
before (mode=min)         8m51s   rebuilt
after, no change          1m02s   all CACHED
after, one file changed   2m14s   go mod download CACHED, build partial (go-build cache)

Confirming in production

• Median build time for the API image dropped from 8.8 to 2.1 minutes over two weeks.
• Total CI minutes for the repository fell by about 35 percent.
• The Actions cache usage rose from 1.2 GB to 6.8 GB, still within the repository limit; we watch it because entries beyond the limit are evicted, which would silently slow builds again.

Regression guard

• A job summary step prints how many build steps were CACHED; a drop below 70 percent on unchanged builds posts a warning.
• Cache scopes follow a convention (<image>-<branch>) documented in the shared workflow.
• A scheduled workflow deletes caches for branches merged more than 7 days ago.

Security implications

Cache from feature branches can be read by builds on other branches only if you configure it that way. We let feature branches read from main, never the reverse, so a malicious or broken branch cannot poison the cache used for release builds. Release images are built on main with a cache scope that only main writes to.

Prevention checklist

• Use mode=max for multi-stage builds where builder stages are expensive.
• Scope caches per image and branch, with a fallback to the default branch.
• Order Dockerfile steps for cache reuse and use cache mounts for package and build caches.
• Watch cache size limits and evictions.
• Do not let untrusted branches write to the cache used by release builds.

References

• Docker build cache backends — https://docs.docker.com/build/cache/backends/
• GitHub Actions cache backend for BuildKit — https://docs.docker.com/build/cache/backends/gha/
• Cache management with GitHub Actions — https://docs.docker.com/build/ci/github-actions/cache/
• Optimizing builds with cache mounts — https://docs.docker.com/build/cache/optimize/