GPU pods stuck Pending: 0/3 nodes are available: Insufficient nvidia.com/gpu

We added a node pool with H100 GPUs for a fine-tuning project. The nodes joined the cluster and showed Ready. Training pods requesting GPUs stayed Pending:

$ kubectl -n training describe pod finetune-llm-0
...
Events:
  Warning  FailedScheduling  2m  default-scheduler  0/14 nodes are available: 3 Insufficient nvidia.com/gpu,
  8 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {nvidia.com/gpu: present}.
  preemption: 0/14 nodes are available: 14 Preemption is not helpful for scheduling.

Reproducing it

The new nodes' capacity:

$ kubectl get node gpu-h100-a1 -o jsonpath='{.status.allocatable}' | jq
{
  "cpu": "191500m",
  "ephemeral-storage": "...",
  "memory": "2045Gi",
  "pods": "110"
}

No nvidia.com/gpu entry at all. The older A100 nodes showed "nvidia.com/gpu": "8".

$ kubectl -n gpu-operator get pods -o wide | grep gpu-h100
(no device plugin pods on the new nodes)

What we ruled out

• Driver installation. SSH to a node and nvidia-smi listed eight H100s with driver 550.
• The pod's GPU request. Same spec worked on the A100 pool.
• Quota. The namespace had no GPU quota limits.

Root cause

Kubernetes does not discover GPUs by itself. The NVIDIA device plugin, a DaemonSet, runs on each GPU node and registers nvidia.com/gpu with the kubelet. The new pool had a taint nvidia.com/gpu=present:NoSchedule (the cloud provider's default for GPU pools, different from the taint key/value our older pool used). The GPU Operator's DaemonSets were configured with a toleration for the old taint only, so none of its components, including the device plugin, were scheduled on the new nodes. The nodes never advertised GPUs, so the scheduler saw zero.

What is happening under the hood

Extended resources like nvidia.com/gpu appear in a node's capacity only when a device plugin registers them through the kubelet's device plugin API. The NVIDIA GPU Operator manages the device plugin along with the container toolkit, GPU feature discovery, DCGM exporter and optionally the driver itself, all as DaemonSets that must run on every GPU node.

Taints keep general workloads off expensive GPU nodes. Every DaemonSet that must run there, and every workload that should, needs a matching toleration. A mismatch on the device plugin means no GPU resource at all, with no error on the node.

The scheduler's message lists reasons per node group: "Insufficient nvidia.com/gpu" here meant "these nodes have zero allocatable GPUs", not "the GPUs are busy".

Environment

Minimal reproduction

Taint a GPU node with a key the device plugin DaemonSet does not tolerate, delete the device plugin pod on it, and watch nvidia.com/gpu disappear from the node's allocatable resources after the kubelet restarts or resyncs.

The fix

1. Tolerations for GPU taints in the operator's DaemonSets. In the GPU Operator Helm values:

daemonsets:
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
    - key: gpu
      operator: Exists
      effect: NoSchedule

operator: Exists tolerates any value for the key. After the upgrade, device plugin pods started on the new nodes and allocatable updated within a minute:

$ kubectl get node gpu-h100-a1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
8

2. One taint convention. We standardised on nvidia.com/gpu=present:NoSchedule for all GPU pools and migrated the old pool, so new pools and DaemonSets follow one rule.

3. Matching tolerations and selectors in training jobs.

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
nodeSelector:
  nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
resources:
  limits:
    nvidia.com/gpu: 8

The product label comes from GPU feature discovery, which also only works when its DaemonSet runs on the node.

Alternatives and trade-offs

• Device plugin DaemonSet without the operator. Fewer moving parts if drivers come from the node image; you manage each component's tolerations yourself.
• No taints on GPU nodes. Avoids toleration mistakes, and lets ordinary pods fill expensive nodes. We keep taints.

How to test the fix

A post-provisioning check for every new GPU node pool:

kubectl get nodes -l nvidia.com/gpu.present=true \
  -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu
kubectl run gpu-smoke --rm -it --restart=Never --image=nvidia/cuda:12.6.3-base-ubuntu24.04 \
  --overrides='{"spec":{"tolerations":[{"key":"nvidia.com/gpu","operator":"Exists"}],"containers":[{"name":"c","image":"nvidia/cuda:12.6.3-base-ubuntu24.04","command":["nvidia-smi","-L"],"resources":{"limits":{"nvidia.com/gpu":"1"}}}]}}'

Confirming in production

• Fine-tuning pods scheduled within a minute of the operator upgrade.
• All GPU nodes show non-zero allocatable GPUs; GPU feature discovery labels are present on every GPU node.

Regression guard

• An alert for nodes labelled as GPU nodes with zero allocatable nvidia.com/gpu for more than 10 minutes.
• An alert on pods pending for more than 15 minutes with GPU requests.
• The GPU node pool Terraform module sets the standard taint, and the platform test suite runs the smoke pod on each pool.

Security implications

Taints on GPU nodes are partly a security boundary for cost and isolation: they keep arbitrary workloads off nodes with privileged device access. Tolerating nvidia.com/gpu with Exists is fine for the operator's components; for workloads, we grant the toleration only through the training job templates, and an admission policy rejects it elsewhere.

Prevention checklist

• Check node allocatable for nvidia.com/gpu when a GPU node joins.
• Make sure GPU operator DaemonSets tolerate every GPU taint in use.
• Standardise GPU taints across pools and providers.
• Run a GPU smoke pod on every new pool.
• Alert on GPU nodes advertising zero GPUs.

References

• Kubernetes, scheduling GPUs — https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
• Kubernetes device plugins — https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
• NVIDIA GPU Operator getting started — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html
• NVIDIA k8s-device-plugin — https://github.com/NVIDIA/k8s-device-plugin