An 18 GB inference image took 9 minutes to scale up: model weights baked into Docker

Our self-hosted LLM service runs on a GPU node pool that autoscales. During a product launch, traffic tripled in ten minutes. The autoscaler added nodes quickly, but new pods took about nine minutes to become ready. Requests queued, timeouts rose to 14 percent, and by the time capacity arrived the spike was half over.

The pod timeline for one new replica:

00:00  node ready
00:05  pod scheduled
00:06  Pulling image "registry.example.com/llm-server:2026.01.12"
07:41  Successfully pulled image (18.2 GB) in 7m35s
07:52  container started
08:58  model loaded, readiness probe passing

Reproducing it

$ docker image inspect registry.example.com/llm-server:2026.01.12 --format '{{.Size}}' | numfmt --to=iec
18G
$ docker history registry.example.com/llm-server:2026.01.12 | head -3
IMAGE     CREATED BY                                      SIZE
<id>      COPY models/ /models/                           15.1GB
<id>      RUN pip install -r requirements.txt             2.4GB

The weights were copied into the image at build time so the service would be "self-contained".

What we ruled out

• Registry throttling. Pull speed was around 40 MB/s, normal for our registry from that region, and the image was simply large.
• Model loading code. Loading from local disk into GPU memory took about a minute, reasonable for 15 GB.

Root cause

Every new node had to download 18 GB before the container could start, and layer extraction for a 15 GB layer adds more time on top. The weights rarely change (we update the model monthly) while code changes weekly, but each code deploy also produced a new image and the weights layer was re-pulled on nodes that had been replaced. Image pull time, not compute, determined how fast we could scale.

What is happening under the hood

Container images are pulled layer by layer and each layer must be fully downloaded and unpacked before the container starts (unless you use lazy-loading snapshotters). Large layers dominate start time. Nodes created by an autoscaler start with an empty image cache, so the first pod on each new node pays the full cost.

Model weights have a different lifecycle from code: they are large, change rarely and can be shared between pods on a node. Packing them into the image ties their distribution to every code release and every new node.

Environment

Minimal reproduction

Build any image with a multi-gigabyte COPY of weights, push it, and time crictl pull on a fresh node.

The fix

1. Remove weights from the image. The image is now 3.1 GB: runtime, libraries, server code. Weights live in S3 under a versioned prefix, referenced by model name and revision in the deployment config.

2. Node-local weight cache. Each GPU node gets a local NVMe-backed cache directory. An init container downloads the weights for the configured revision if they are not already cached, using parallel ranged downloads:

initContainers:
  - name: fetch-weights
    image: registry.example.com/model-fetcher:1.4
    args: ["--src", "s3://example-models/llm-8b/rev-2026-01-05/", "--dst", "/cache/llm-8b/rev-2026-01-05/",
           "--parallel", "16", "--verify-sha256"]
    volumeMounts:
      - {name: model-cache, mountPath: /cache}
volumes:
  - name: model-cache
    hostPath: {path: /mnt/nvme/model-cache, type: DirectoryOrCreate}

The fetcher writes to a temporary directory and renames it when complete and verified, so a partially downloaded model is never used. Pull from S3 with parallel connections reached about 900 MB/s on our instance type, roughly 20 times the registry pull speed for one large layer.

3. Warm new nodes before pods arrive. A DaemonSet on GPU nodes pre-fetches the current production revision as soon as a node starts, so by the time the scheduler places a server pod, the weights are often already local.

4. Readiness reflects the real state. The readiness probe only passes after a warm-up request completes, so traffic never reaches a pod that is still loading.

Alternatives and trade-offs

• Lazy-loading image snapshotters (for example SOCI or stargz) start containers before the whole image is downloaded. They help with large images in general; for weights read in full at startup, the gain is smaller.
• Shared network filesystem (EFS, FSx for Lustre) mounted read-only. No per-node download, but read throughput limits load time and costs money at scale.
• Model streaming loaders that read safetensors directly from object storage into GPU memory. Promising for very large models; we kept a local copy for simpler failure handling.

How to test the fix

We timed scale-up from zero to four new nodes in staging with the old and new setups:

                         old       new (cold node)   new (pre-warmed node)
image pull               7m35s     1m10s             1m10s
weights ready            -         0m45s             0m00s
model load + warm-up     1m06s     1m02s             1m02s
total to ready           ~9m       ~3m               ~2m10s

Image pull is still the longest step for the 3 GB image; we are trimming it further.

Confirming in production

• During the next traffic spike, new pods were ready in 2 to 3 minutes and timeout rates stayed under 1 percent.
• Code-only deploys no longer re-download weights; deploy time dropped from 12 to 4 minutes for the rolling update.

Regression guard

• CI fails the build if the server image exceeds 5 GB.
• Time-to-ready for new pods is a dashboard metric with an alert above 5 minutes.
• Weight revisions are referenced by explicit revision ids, never "latest".

Security implications

Weights downloaded at runtime must be verified: the fetcher checks SHA-256 hashes against a manifest stored with the model in the registry, and the S3 prefix is writable only by the model release pipeline. Using safetensors avoids code execution during loading. The hostPath cache is mounted read-only into the server container.

Prevention checklist

• Keep model weights out of container images.
• Cache weights per node and pre-fetch on node start.
• Download in parallel from object storage and verify hashes.
• Make readiness depend on a warm-up inference.
• Measure time-to-ready for new nodes, not only for restarted pods.

References

• Kubernetes init containers — https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
• Seekable OCI (SOCI) lazy loading — https://github.com/awslabs/soci-snapshotter
• safetensors — https://huggingface.co/docs/safetensors/index
• Karpenter NodePools — https://karpenter.sh/docs/concepts/nodepools/