Ollama in Docker runs on CPU only: GPU not passed to the container

We set up a small on-premises server to run open models for internal tools that cannot send data to external APIs. The hardware was solid: one RTX 4090 with 24 GB of VRAM. The first test with an 8B model was painful: about 3 tokens per second, and the server fans spun up as if it were compiling a kernel.

Ollama itself told us what was wrong:

$ docker exec ollama ollama ps
NAME           ID              SIZE      PROCESSOR    UNTIL
llama3.1:8b    46e0c10c039e    6.7 GB    100% CPU     4 minutes from now

100% CPU. The model was not on the GPU at all.

Reproducing it

Our Compose file was copied from a tutorial for Mac, where there is no GPU passthrough to configure:

services:
  ollama:
    image: ollama/ollama:0.5.11
    ports: ["11434:11434"]
    volumes: ["ollama:/root/.ollama"]

Inside the container:

$ docker exec ollama nvidia-smi
OCI runtime exec failed: exec failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown

On the host, nvidia-smi worked and showed the 4090 idle.

What we ruled out

• Model too large for VRAM. An 8B model at 4-bit quantisation needs well under 24 GB. Partial offload would show a split like 40%/60% CPU/GPU, not 100% CPU.
• Driver problems on the host. nvidia-smi on the host was fine, driver 550.
• An Ollama build without GPU support. The official image includes CUDA support; it just needs the device.

Root cause

Docker containers do not get access to GPUs by default. You need two things: the NVIDIA Container Toolkit on the host, which injects the driver libraries and device nodes into the container, and an explicit request for GPUs when starting the container. We had neither. Ollama detected no GPU and silently fell back to CPU inference.

What is happening under the hood

The GPU driver lives on the host kernel. A container needs the device files (/dev/nvidia*) and matching user-space driver libraries. The NVIDIA Container Toolkit provides a runtime hook that adds both when a container asks for GPUs. With docker run --gpus all or a Compose device reservation, Docker calls that hook.

If the toolkit is not installed, requesting a GPU fails loudly:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

If you do not request a GPU at all, nothing fails. That is the dangerous case: the app starts and runs slowly.

Environment

Minimal reproduction

docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama
docker exec ollama ollama run llama3.1:8b "hello" && docker exec ollama ollama ps
# PROCESSOR: 100% CPU

The fix

1. Install and configure the NVIDIA Container Toolkit on the host.

# after adding NVIDIA's apt repository as described in the toolkit install guide
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# sanity check with a plain CUDA image
docker run --rm --gpus all nvidia/cuda:12.6.3-base-ubuntu24.04 nvidia-smi

2. Request the GPU in Compose.

services:
  ollama:
    image: ollama/ollama:0.5.11
    ports: ["127.0.0.1:11434:11434"]
    volumes: ["ollama:/root/.ollama"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped
volumes:
  ollama: {}

The equivalent docker run flag is --gpus all.

3. Fail fast if the GPU is missing. We added a health check script that runs after startup and fails the container if no GPU is visible, so this can never silently happen again:

#!/bin/sh
nvidia-smi -L >/dev/null 2>&1 || { echo "no GPU visible in container" >&2; exit 1; }

Alternatives and trade-offs

• Run Ollama directly on the host. Uses the GPU without container plumbing. We wanted Docker for consistent upgrades and isolation.
• CDI device specs. Newer Docker versions can use the Container Device Interface to describe GPUs. Equivalent result; the toolkit can generate the spec.
• A different serving engine. For multi-user throughput, engines like vLLM batch requests better. Ollama is simpler for a handful of internal users.

How to test the fix

$ docker exec ollama nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-...)

$ docker exec ollama ollama run llama3.1:8b --verbose "Write a haiku about logs"
...
eval rate:            121.37 tokens/s

$ docker exec ollama ollama ps
NAME           ID              SIZE      PROCESSOR    UNTIL
llama3.1:8b    46e0c10c039e    6.7 GB    100% GPU     4 minutes from now

Confirming in production

• Generation speed went from about 3 to about 120 tokens per second for the 8B model.
• GPU memory usage in nvidia-smi on the host shows the model resident after the first request.
• CPU load during generation dropped from all cores pegged to under 10 percent.

Regression guard

• The GPU health check in the container.
• Prometheus node exporter plus the NVIDIA DCGM exporter; an alert fires if GPU utilisation stays at 0 while the Ollama request rate is above zero.
• Driver and toolkit versions pinned in our Ansible role, upgraded deliberately.

Security implications

Two things to note. The Ollama API has no authentication by default, so the port is bound to 127.0.0.1 and exposed to internal users only through a reverse proxy with SSO. Exposing 11434 on all interfaces is a common mistake that lets anyone use your GPU and pull or delete models. Second, GPU access gives a container access to host device files; only trusted images get the GPU reservation.

Prevention checklist

• Install the NVIDIA Container Toolkit before running GPU workloads in Docker.
• Always request GPUs explicitly in run or Compose.
• Check ollama ps for the PROCESSOR column after deploy.
• Make the container fail if the GPU is missing.
• Never expose the Ollama port publicly without auth.

References

• Ollama Docker image documentation — https://hub.docker.com/r/ollama/ollama
• NVIDIA Container Toolkit install guide — https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
• Docker Compose GPU support — https://docs.docker.com/compose/how-tos/gpu-support/
• Ollama FAQ (GPU, exposing the server) — https://github.com/ollama/ollama/blob/main/docs/faq.md