CI fails with toomanyrequests: You have reached your pull rate limit on Docker Hub

In early April 2025, our self-hosted CI started failing builds at random, always in the same place:

#3 [internal] load metadata for docker.io/library/python:3.12-slim
#3 ERROR: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to
https://auth.docker.io/token?scope=repository%3Alibrary%2Fpython%3Apull&service=registry.docker.io: 429 Too Many Requests

and in test jobs that start service containers:

Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

About a third of pipelines failed during working hours. Late evenings were fine.

Reproducing it

From one runner, the Docker Hub rate limit headers showed the shared state:

TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl -s --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i ratelimit

ratelimit-limit: 10;w=3600
ratelimit-remaining: 0;w=3600

Ten pulls per hour, and none left.

What we ruled out

• Docker Hub outage. The status page was green, and authenticated pulls from a laptop worked.
• A loop pulling images repeatedly. Pull counts per pipeline were normal: four to eight images.

Root cause

Docker Hub changed its limits on April 1, 2025: unauthenticated users are limited to 10 pulls per hour per IPv4 address (or IPv6 /64), down from a much higher limit. Our 24 runners sit in private subnets behind a single NAT gateway, so Docker Hub saw all of them as one anonymous client. Ten pulls per hour across the whole CI fleet ran out within minutes of the working day starting.

What is happening under the hood

Docker Hub counts pulls per identity. Without credentials, the identity is the source IP. Behind NAT, every machine shares that IP. A "pull" is counted for manifest requests, so even pulling an image that is already cached locally can count if the client checks for a newer manifest, as docker build --pull and imagePullPolicy: Always do.

With a Docker account, limits apply per account instead of per IP and are much higher; paid plans have higher limits still. A registry mirror or pull-through cache inside your network fetches each image once and serves it to every runner.

Environment

Minimal reproduction

From a machine with no Docker Hub login, run 11 pulls of different tags within an hour. The 11th fails with toomanyrequests.

The fix

1. Authenticate every runner. A Docker Hub organisation account with a token stored in the runner configuration. Pulls now count against the account, not the NAT IP:

# /etc/gitlab-runner/config.toml (excerpt)
[[runners]]
  environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"https://index.docker.io/v1/\":{\"auth\":\"<base64 user:token>\"}}}"]

This alone stopped the failures within an hour.

2. A pull-through cache. We use our cloud provider's registry with a pull-through cache rule for Docker Hub, so the first pull of python:3.12-slim fetches it from Docker Hub and later pulls are served from our registry:

# before
FROM python:3.12-slim
# after
FROM 111122223333.dkr.ecr.eu-west-1.amazonaws.com/docker-hub/library/python:3.12-slim

For tools that hardcode docker.io (service containers, Testcontainers), the Docker daemon on runners is configured with a registry mirror pointing at our cache.

3. Pin base images by digest and stop pulling on every build. --pull on every build was the habit that multiplied manifest requests. Builds now use digests, updated by Renovate, and --pull only on the scheduled base image refresh job:

FROM .../docker-hub/library/python:3.12-slim@sha256:<digest>

Alternatives and trade-offs

• Paid Docker subscription only. Raises limits, keeps the external dependency on every build.
• Public mirrors of official images (for example on other public registries). Useful, but you then depend on a different third party.
• Self-hosted registry mirror (registry:2 in proxy mode or Harbor). Full control; one more service to run.

How to test the fix

# on a runner, with auth configured
for i in $(seq 1 30); do docker pull -q redis:7.4-alpine >/dev/null && echo ok; docker rmi -f redis:7.4-alpine >/dev/null; done | sort | uniq -c
#   30 ok

The pull-through cache was verified by watching registry metrics: the first pull of each image hit Docker Hub, the rest were served from cache.

Confirming in production

• Pipeline failures from toomanyrequests dropped from about 30 percent of pipelines to zero.
• Docker Hub pulls from our organisation dropped by 94 percent after the cache, according to the Docker Hub usage report.
• Image pull time in CI fell from 20 to 40 seconds to 3 to 5 seconds for base images, since the cache is in the same region.

Regression guard

• A CI lint rule warns on FROM lines that reference Docker Hub directly.
• An alert on 429 responses from Docker Hub in runner logs.
• Base image digests are updated weekly by Renovate with a test pipeline.

Security implications

Pinning by digest and pulling through our own registry also improves supply chain security: images are scanned when they enter our registry, a tag on Docker Hub being repointed does not change our builds until a reviewed digest update, and we have a record of exactly which base image each build used. The Docker Hub token is read-only and scoped to pulls.

Prevention checklist

• Authenticate all registry pulls, especially behind NAT.
• Put a pull-through cache or mirror between CI and public registries.
• Pin base images by digest and update them deliberately.
• Avoid --pull and imagePullPolicy: Always where not needed.
• Watch registry rate limit headers and 429s.

References

• Docker Hub usage and rate limits — https://docs.docker.com/docker-hub/usage/
• Docker Hub pull rate limit headers — https://docs.docker.com/docker-hub/usage/pulls/
• Amazon ECR pull through cache — https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html
• Docker daemon registry mirrors — https://docs.docker.com/docker-hub/image-library/mirror/