Kubernetes HPA shows <unknown>/70% and never scales: missing request for cpu

A marketing email went out at 10:00, traffic to checkout tripled, and p99 latency went from 300 ms to 6 seconds. CPU on the three checkout pods was at their limits. The HorizontalPodAutoscaler, configured to scale up to 30 replicas at 70 percent CPU, did nothing.

$ kubectl -n shop get hpa checkout
NAME       REFERENCE             TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
checkout   Deployment/checkout   cpu: <unknown>/70%   3         30        3          214d

The events explained it:

$ kubectl -n shop describe hpa checkout
...
Conditions:
  Type            Status  Reason                   Message
  AbleToScale     True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive   False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container "log-shipper" of Pod "checkout-7d9f8c6b4-2xkqp"
Events:
  Warning  FailedGetResourceMetric  2m (x6181 over 21d)  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu in container "log-shipper" of Pod "checkout-7d9f8c6b4-2xkqp"

6,181 warnings over 21 days. Nobody was watching HPA conditions.

Reproducing it

21 days earlier, the platform team had added a log-shipper sidecar to several workloads through a mutating policy. The sidecar had a memory request and limit but no CPU request. Any HPA using CPU utilisation for those workloads broke at that moment.

What we ruled out

• metrics-server down. kubectl top pods -n shop returned values for every container.
• Wrong scale target. AbleToScale was true.
• Max replicas reached. The HPA was at 3, its minimum.

Root cause

For a Resource metric with target.type: Utilization, the HPA computes utilisation as the pod's total CPU usage divided by the sum of CPU requests of the containers. If any container in the pod has no CPU request, the ratio is undefined, and the HPA refuses to compute it. It reports <unknown>, sets ScalingActive=False, and keeps the current replica count.

What is happening under the hood

The HPA controller loops every 15 seconds by default. For CPU utilisation targets it:

1. Fetches per-pod CPU usage from the resource metrics API (served by metrics-server).
2. Divides by the pod's total CPU request.
3. Averages across pods and computes desiredReplicas = ceil(currentReplicas * current / target).

Step 2 needs a request on every container, including sidecars and, depending on the metric configuration, not counting init containers. A missing request on any container makes utilisation undefined for the pod.

Kubernetes also supports ContainerResource metrics, which scale on the utilisation of one named container, ignoring sidecars. It became stable in Kubernetes 1.30.

Environment

Minimal reproduction

containers:
  - name: app
    resources:
      requests: {cpu: 500m, memory: 512Mi}
  - name: log-shipper
    resources:
      requests: {memory: 64Mi}          # no cpu request
      limits:   {memory: 128Mi}
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  metrics:
    - type: Resource
      resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}

The fix

Immediate mitigation during the incident: we scaled the Deployment by hand to 15 replicas, which brought latency back within a few minutes. HPA ownership of replicas means it can override a manual scale later, but with ScalingActive=False it was not touching anything.

1. CPU requests on every container, including injected ones. The sidecar policy now sets requests explicitly:

resources:
  requests: {cpu: 20m, memory: 64Mi}
  limits:   {memory: 128Mi}

We sized 20m from a week of container_cpu_usage_seconds_total for the sidecar: p95 was 12m.

2. Scale on the app container only. Sidecar CPU is noise for scaling decisions, so we switched the checkout HPA to a container metric:

metrics:
  - type: ContainerResource
    containerResource:
      name: cpu
      container: app
      target:
        type: Utilization
        averageUtilization: 70

3. A namespace default as a safety net. A LimitRange in workload namespaces sets default CPU requests for containers that do not declare one, so a future sidecar cannot break HPAs silently:

apiVersion: v1
kind: LimitRange
metadata: {name: defaults, namespace: shop}
spec:
  limits:
    - type: Container
      defaultRequest: {cpu: 50m, memory: 64Mi}

Alternatives and trade-offs

• Scale on request rate or queue depth through custom or external metrics. Often a better signal than CPU for web services; it needs Prometheus Adapter or KEDA. We added an RPS metric as a second signal later.
• AverageValue targets (absolute millicores) do not need requests, but they do not adjust when you change pod sizes.

How to test the fix

kubectl -n shop get hpa checkout
# cpu: 41%/70%
kubectl -n shop describe hpa checkout | grep ScalingActive
# ScalingActive  True  ValidMetricFound  ...

A load test in staging with k6 at 3x normal traffic scaled checkout from 3 to 11 replicas in about 2 minutes and kept p99 under 500 ms.

Confirming in production

• All 37 HPAs in the cluster now report numeric targets; we found four more with <unknown> caused by the same sidecar.
• During the next campaign, checkout scaled to 14 replicas and latency stayed flat.

Regression guard

• An alert on kube_horizontalpodautoscaler_status_condition{condition="ScalingActive",status="false"} for more than 10 minutes (from kube-state-metrics).
• A Kyverno validate policy that rejects pods in workload namespaces with any container lacking a CPU request.
• Any change to injected sidecars goes through a staging soak that checks HPA conditions.

Security implications

None directly. Indirectly, autoscaling is part of availability: an HPA that cannot scale turns a traffic spike, legitimate or abusive, into an outage. Monitoring HPA health belongs with the rest of the denial-of-service protections.

Prevention checklist

• Set CPU and memory requests on every container, including sidecars and injected ones.
• Alert when an HPA's ScalingActive condition is false.
• Prefer ContainerResource metrics when pods have sidecars.
• Use a LimitRange default as a safety net.
• Load test autoscaling, not just the service.

References

• Kubernetes Horizontal Pod Autoscaling — https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
• HPA container resource metrics — https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics
• Kubernetes LimitRange — https://kubernetes.io/docs/concepts/policy/limit-range/
• kube-state-metrics HPA metrics — https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/horizontalpodautoscaler-metrics.md