torch.cuda.is_available() is False after an image rebuild: driver too old for CUDA 12

A routine rebuild of our image embedding service pulled a newer PyTorch. After the rollout, p95 latency doubled and the GPU nodes' utilisation dropped to zero on some of them. No errors in the request logs. The startup log on an affected pod had this warning:

/usr/local/lib/python3.11/site-packages/torch/cuda/__init__.py: UserWarning: CUDA initialization:
The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by
downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with
your version of the CUDA driver.

and our own log line said device=cpu. The service fell back to CPU when CUDA was unavailable, a convenience from early development that hid the problem.

Reproducing it

On an affected node:

$ nvidia-smi | head -4
| NVIDIA-SMI 470.256.02   Driver Version: 470.256.02   CUDA Version: 11.4 |

$ kubectl exec deploy/embedder -- python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
2.5.1+cu121 12.1 False

A small CUDA sample compiled against the 12.1 runtime reported the classic message:

CUDA driver version is insufficient for CUDA runtime version

On newer nodes with driver 550, the same image worked.

What we ruled out

• NVIDIA Container Toolkit problems. nvidia-smi worked inside the container, so the GPU was visible.
• Missing GPU requests. Pods requested nvidia.com/gpu: 1 and were scheduled on GPU nodes.

Root cause

The previous image used PyTorch built for CUDA 11.8 (cu118). The rebuild picked up the default wheel for CUDA 12.1. CUDA 12 runtimes require an NVIDIA driver of the R525 branch or newer; older nodes in our fleet ran an AMI with driver 470, which supports CUDA up to 11.4 (plus 11.x minor version compatibility). The driver cannot be upgraded from inside a container: it belongs to the host kernel. Half of the fleet was on the old AMI because node groups had been created at different times.

What is happening under the hood

A GPU container uses two layers of CUDA:

• The driver, installed on the host, which includes the kernel module and libcuda.so, injected into containers by the NVIDIA Container Toolkit.
• The CUDA runtime and libraries (cudart, cuBLAS, cuDNN), shipped inside the image or inside the PyTorch wheel.

The runtime requires a minimum driver version. Within a major CUDA version, minor version compatibility lets newer 12.x runtimes run on any R525+ driver for most features. Across major versions there is no such compatibility: a CUDA 12 runtime cannot run on a driver that only supports CUDA 11. NVIDIA's forward compatibility packages can help on data center GPUs in specific cases, but the normal fix is a newer driver.

PyTorch reports this as a warning and makes torch.cuda.is_available() return False, which code often treats as "no GPU, use CPU".

Environment

Minimal reproduction

docker run --rm --gpus all python:3.11-slim bash -c \
  "pip -q install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu121 && \
   python -c 'import torch; print(torch.cuda.is_available())'"
# False on a host with driver 470, True on a host with driver >= 525

The fix

1. Upgrade the driver on all GPU nodes by moving every GPU node group to the current EKS-optimised accelerated AMI with driver 550, using a rolling node group update.

2. Fail fast instead of falling back to CPU. The service now refuses to start if it expects a GPU and cannot use one:

import sys, torch

def require_gpu() -> torch.device:
    if not torch.cuda.is_available():
        print(f"FATAL: CUDA unavailable (torch {torch.__version__}, built for CUDA {torch.version.cuda})",
              file=sys.stderr)
        sys.exit(1)
    dev = torch.device("cuda")
    torch.zeros(1, device=dev)               # forces CUDA context creation
    return dev

A pod that cannot use the GPU now crash-loops and is visible immediately, instead of serving slowly.

3. Pin the CUDA flavour explicitly. The Dockerfile installs a specific wheel and records it as an image label:

ARG TORCH_VERSION=2.5.1
ARG CUDA_TAG=cu121
RUN pip install --no-cache-dir torch==${TORCH_VERSION} --index-url https://download.pytorch.org/whl/${CUDA_TAG}
LABEL org.example.cuda-runtime="12.1" org.example.min-driver="525"

4. Schedule by driver capability. GPU nodes carry a label with their driver major version (set by GPU feature discovery from the GPU Operator), and the deployment requires a minimum:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: nvidia.com/cuda.driver.major
              operator: Gt
              values: ["524"]

Alternatives and trade-offs

• Stay on CUDA 11.8 wheels until all nodes are upgraded. A valid short-term fix; newer PyTorch features and some libraries target CUDA 12.
• Forward compatibility package in the image for data center GPUs. Possible in some setups; we preferred a consistent driver fleet.

How to test the fix

A CI job runs the image on a GPU runner with the oldest driver we support and executes the startup check plus one inference. Before rollout, a canary pod on each node group confirmed torch.cuda.is_available() and a matrix multiplication on the GPU.

Confirming in production

• All pods report device=cuda at startup; GPU utilisation is back to normal and p95 latency returned to its previous level.
• No more nodes with driver 470 in the cluster; a weekly inventory confirms driver versions across node groups.

Regression guard

• Startup GPU check in every GPU service.
• An alert on GPU nodes with zero utilisation while pods requesting GPUs run on them.
• Image labels for CUDA runtime and minimum driver, checked against node labels in CI before deployment.

Security implications

Old GPU drivers also miss security fixes; NVIDIA publishes driver security bulletins several times a year. Keeping the GPU fleet on a supported driver branch is a patching requirement, not only a compatibility one. Silent CPU fallback also changed the service's resource profile under load, which could have become an availability problem.

Prevention checklist

• Know the minimum driver for the CUDA runtime in your image.
• Keep GPU node drivers consistent across node groups.
• Fail at startup if a GPU service cannot use the GPU.
• Pin PyTorch and CUDA flavours explicitly in images.
• Label nodes with driver versions and schedule accordingly.

References

• CUDA compatibility (minor version and forward compatibility) — https://docs.nvidia.com/deploy/cuda-compatibility/
• CUDA toolkit release notes, driver requirements — https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/
• PyTorch installation and CUDA wheels — https://pytorch.org/get-started/locally/
• NVIDIA GPU Operator, GPU feature discovery — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/