GPU inference never scaled under load: HPA on CPU for a GPU-bound LLM server

An internal assistant built on a self-hosted model got a surge of use after a company-wide announcement. Users reported that answers took 30 to 60 seconds to start. The vLLM deployment had 2 replicas and an HPA allowing up to 8. It stayed at 2.

$ kubectl -n llm get hpa vllm
NAME   REFERENCE         TARGETS          MINPODS   MAXPODS   REPLICAS
vllm   Deployment/vllm   cpu: 14%/70%     2         8         2

vLLM's own metrics, scraped by Prometheus:

vllm:num_requests_running   {pod="vllm-0"}  48
vllm:num_requests_waiting   {pod="vllm-0"}  131
vllm:gpu_cache_usage_perc   {pod="vllm-0"}  0.98

The GPU's KV cache was full, 131 requests were waiting, and CPU was nearly idle.

Reproducing it

A load test with 200 concurrent streaming requests against a staging deployment reproduced it: time to first token rose from 0.4 seconds to over 30, while CPU utilisation stayed below 20 percent and the HPA never acted.

What we ruled out

• Cluster capacity. The GPU node pool could add nodes; it was never asked to because no new pods were created.
• HPA misconfiguration such as missing CPU requests. The HPA computed CPU correctly; CPU was simply low.

Root cause

The HPA was copied from a CPU-bound web service template. An LLM server's work happens on the GPU; the CPU handles tokenisation and HTTP, which stays light even when the GPU is saturated. When the KV cache is full, vLLM queues new requests, and latency grows without any change in CPU. CPU utilisation was the wrong signal for this workload.

What is happening under the hood

For LLM inference servers, useful scaling signals are:

• Waiting requests (vllm:num_requests_waiting): requests queued because the engine cannot schedule them. Directly tied to user-visible latency.
• KV cache usage (vllm:gpu_cache_usage_perc; some newer releases rename it to vllm:kv_cache_usage_perc, so check your version's /metrics output): how full the cache is; near 100 percent means the server is at capacity.
• Time to first token percentiles: what users feel, but noisy and lagging.

GPU utilisation from DCGM is a weaker signal: a GPU can show high utilisation at moderate load, and it does not say whether requests are waiting.

Kubernetes HPA can scale on custom or external metrics through an adapter (Prometheus Adapter) or through KEDA, which creates and manages the HPA for you from scalers such as Prometheus queries.

Scaling a GPU server takes minutes (node provisioning, image pull, model load), so the signal needs to react early and scale-down must be conservative.

Environment

Minimal reproduction

Any GPU-bound server with a CPU-based HPA: generate load that saturates the GPU and observe that CPU stays low and replicas stay constant.

The fix

1. Scale on waiting requests with KEDA.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm
  namespace: llm
spec:
  scaleTargetRef:
    name: vllm
  minReplicaCount: 2
  maxReplicaCount: 8
  pollingInterval: 15
  cooldownPeriod: 600
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies: [{type: Pods, value: 2, periodSeconds: 60}]
        scaleDown:
          stabilizationWindowSeconds: 900
          policies: [{type: Pods, value: 1, periodSeconds: 300}]
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{namespace="llm"})
        threshold: "10"
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: avg(vllm:gpu_cache_usage_perc{namespace="llm"}) * 100
        threshold: "85"

With the prometheus scaler, the threshold is a target value per replica, so 10 waiting requests per replica triggers scale-up. The KV cache trigger scales before queues form. Scale-down is slow and one pod at a time, because adding capacity back costs minutes.

2. Make new replicas useful fast. Model weights are cached on node-local disk (see our post on weights baked into images), and the readiness probe passes only after a warm-up generation, so traffic does not hit a pod still loading.

3. A small buffer. minReplicaCount: 2 during working hours plus one GPU node kept warm in the node pool through a low-priority placeholder pod, which is evicted when real pods need the node. That reduced effective scale-up time from about 5 minutes to under 2.

Alternatives and trade-offs

• Prometheus Adapter with a plain HPA on custom metrics. Equivalent result; KEDA was simpler for us.
• Request-based autoscaling in a serverless platform (Knative, managed model endpoints). Good fit for spiky traffic, with cold start costs.
• Admission control: return a fast "busy" error when the queue is long, so clients can retry or degrade, instead of waiting a minute. We added this for very long queues.

How to test the fix

The same 200-concurrency load test: replicas went from 2 to 6 within 4 minutes, waiting requests peaked at 40 and fell to near zero, and time to first token p95 settled at 1.1 seconds after scale-up.

Confirming in production

• During the next company event, replicas scaled from 2 to 7 and back over the afternoon; p95 time to first token stayed under 2 seconds.
• GPU hours rose only during busy periods; overnight the deployment sits at 2 replicas.

Regression guard

• Alerts on vllm:num_requests_waiting above 50 for 5 minutes and on time to first token p95 above 5 seconds.
• Our service templates for GPU workloads ship with KEDA queue-based scaling; CPU-based HPAs are rejected by a policy for pods requesting GPUs.

Security implications

Queue-based scaling makes it easier for anyone with access to drive up GPU spend by sending many long requests. Per-user rate limits and maximum input and output lengths at the gateway cap that, and maxReplicaCount is a hard upper bound on cost.

Prevention checklist

• Do not autoscale GPU-bound servers on CPU.
• Scale on waiting requests and KV cache usage.
• Scale up quickly, scale down slowly.
• Make new replicas ready fast with cached weights and warm-up probes.
• Cap replicas and protect the queue with rate limits.

References

• vLLM production metrics — https://docs.vllm.ai/en/latest/serving/metrics.html
• KEDA Prometheus scaler — https://keda.sh/docs/latest/scalers/prometheus/
• Kubernetes HPA scaling behavior — https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior
• GKE, autoscaling LLM inference workloads on GPUs — https://cloud.google.com/kubernetes-engine/docs/best-practices/machine-learning/inference/autoscaling