Idle GPU nodes billed all weekend: why the cluster autoscaler would not scale down

The ML platform's GPU node pool runs training jobs and data scientists' notebooks. Cost reports showed it running eight g5.2xlarge nodes all weekend, every weekend, while GPU utilisation from DCGM metrics was under 5 percent from Friday evening to Monday morning. The pool had autoscaling enabled with a minimum of zero.

The cluster autoscaler's status and logs explained what it thought:

I0110 21:14:03 Fast evaluation: node ip-10-0-41-17.ec2.internal for removal
I0110 21:14:03 Fast evaluation: node ip-10-0-41-17.ec2.internal cannot be removed: pod with local storage present: jupyter-lan-ng
I0110 21:14:03 Fast evaluation: node ip-10-0-44-82.ec2.internal cannot be removed: kube-system/gpu-feature-discovery-extra-7c9f is not replicated
I0110 21:14:03 Fast evaluation: node ip-10-0-47-05.ec2.internal cannot be removed: non-daemonset, non-mirrored, non-pdb-assigned kube-system pod present: nvidia-dcgm-debug-6b4d

Reproducing it

Listing non-DaemonSet pods per GPU node on a Saturday:

kubectl get pods -A -o wide --field-selector spec.nodeName=ip-10-0-41-17.ec2.internal \
  | grep -v -E 'aws-node|kube-proxy|nvidia-device-plugin|dcgm-exporter'

Each node had one or two pods keeping it alive: idle Jupyter notebook servers left running by users over the weekend, a debug pod someone had started by hand, and a stray kube-system Deployment from a GPU tooling experiment.

What we ruled out

• Minimum node count. The node group minimum was 0.
• Autoscaler disabled. It scaled up correctly for training jobs.
• Training jobs holding GPUs. No training pods were running.

Root cause

The cluster autoscaler only removes a node if it can move every pod on it somewhere else. By default it refuses to evict:

• Pods with local storage (emptyDir or hostPath), because the data would be lost. Our notebook pods used an emptyDir scratch volume.
• Pods not managed by a controller ("not replicated"), because nothing would recreate them. The debug pod was a bare pod.
• kube-system pods without a PodDisruptionBudget, to protect system components.

Each GPU node had at least one such pod. None were doing useful work, but the autoscaler cannot know that, so eight GPU nodes stayed up for 60 hours every weekend.

What is happening under the hood

Scale-down runs every 10 seconds by default. A node is a candidate when its requested resources are below a utilisation threshold (50 percent by default) for a period (scale-down-unneeded-time, 10 minutes by default). Then the autoscaler simulates evicting each pod. Any pod that cannot be moved blocks the node. The reasons are logged and also published in the cluster-autoscaler-status ConfigMap.

The annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "true" on a pod tells the autoscaler it may evict that pod despite local storage or other conditions. "false" does the opposite and pins the node.

GPU nodes make this expensive: one forgotten pod on a GPU node costs far more than on a general-purpose node.

Environment

Minimal reproduction

apiVersion: v1
kind: Pod
metadata: {name: debug, namespace: ml}
spec:
  nodeSelector: {nvidia.com/gpu.present: "true"}
  containers: [{name: c, image: busybox:1.37, command: ["sleep", "infinity"]}]

A bare pod like this on an otherwise empty GPU node keeps the node forever.

The fix

1. Cull idle notebooks. JupyterHub's idle culler stops servers with no activity for 2 hours. User files live on a persistent volume, so nothing important is lost:

# JupyterHub Helm values
cull:
  enabled: true
  timeout: 7200
  every: 600

2. Mark scratch-only pods as safe to evict. The notebook pod's emptyDir is only scratch space, so the pod template gets:

metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

3. No bare pods on GPU nodes. An admission policy rejects pods without an owner reference on nodes with GPUs, except in a ml-debug namespace whose pods get an activeDeadlineSeconds of 4 hours.

4. Clean up kube-system. The stray Deployment was deleted. Real system components that run on GPU nodes are DaemonSets, which the autoscaler ignores for scale-down purposes.

5. Taint GPU nodes so only GPU workloads land there. General pods were sometimes scheduled onto GPU nodes during busy periods and then pinned them:

taints:
  - key: nvidia.com/gpu
    value: "true"
    effect: NoSchedule

Alternatives and trade-offs

• Karpenter with consolidation removes underused nodes more aggressively and respects a karpenter.sh/do-not-disrupt annotation; the same blockers apply in spirit.
• Scheduled scale-down of the pool on weekends. Blunt; breaks long training jobs that run over the weekend.
• Lower scale-down-unneeded-time for the GPU pool. Faster savings, at the cost of more churn when jobs arrive in bursts.

How to test the fix

On a Friday evening we watched the autoscaler status ConfigMap and node count:

kubectl -n kube-system get configmap cluster-autoscaler-status -o jsonpath='{.data.status}' | grep -A3 ScaleDown
kubectl get nodes -l nvidia.com/gpu.present=true --watch

Within 2.5 hours of the last notebook going idle, the pool was at zero nodes.

Confirming in production

• GPU node hours dropped by 41 percent in the first month; GPU spend for the platform by 38 percent.
• Weekday morning scale-up for notebooks takes about 4 minutes, which the data science team accepted.

Regression guard

• A weekly report of GPU node hours with utilisation below 10 percent.
• An alert when GPU nodes exist with zero GPU allocation for more than 2 hours.
• Admission policies for owner references and tolerations on GPU nodes.

Security implications

Bare debug pods with broad images and long lifetimes on shared nodes are also a security concern: they often run as root, mount host paths for debugging, and are forgotten. Restricting them to a debug namespace with time limits reduces that exposure.

Prevention checklist

• Read the autoscaler's scale-down reasons; they name the blocking pod.
• Cull idle interactive workloads like notebooks.
• Mark pods with scratch-only local storage as safe to evict.
• Forbid bare pods and stray kube-system workloads on expensive nodes.
• Taint GPU nodes so only GPU workloads run there.

References

• Cluster Autoscaler FAQ, what types of pods can prevent scale-down — https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node
• Cluster Autoscaler on AWS — https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws
• JupyterHub idle culler — https://z2jh.jupyter.org/en/stable/resources/reference.html#cull
• Kubernetes taints and tolerations — https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/