Node drain hangs: Cannot evict pod as it would violate the pod's disruption budget

We started a routine node group upgrade for our EKS cluster on a Tuesday morning. The managed upgrade drains nodes one by one, and it stopped on the third node. Two hours later it was still there. The node was cordoned, most pods had moved, and the drain log repeated:

evicting pod shop/session-store-0
error when evicting pods/"session-store-0" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Reproducing it

$ kubectl -n shop get pdb
NAME            MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
session-store   2               N/A               0                     188d

$ kubectl -n shop get statefulset session-store
NAME            READY   AGE
session-store   2/2     188d

Two replicas, minAvailable: 2. Allowed disruptions: zero, always.

What we ruled out

• A pod not ready. Both replicas were ready. Even a perfectly healthy workload would block with this PDB.
• An eviction API problem. Other pods on the node had been evicted fine.
• Upgrade misconfiguration. The node group's max unavailable setting was 1, as intended.

Root cause

The PDB required two available pods for a workload that only had two. Any voluntary eviction would drop it to one, so the eviction API refused every request, and the drain retried forever. The PDB was originally created when the StatefulSet had three replicas; someone reduced replicas to two to save cost and nobody updated the PDB.

What is happening under the hood

kubectl drain and managed node upgrades use the Eviction API, not plain deletes. For each eviction, the API server checks PDBs matching the pod. The PDB status field disruptionsAllowed is computed from the number of healthy pods and the budget:

• With minAvailable: N, disruptions allowed = healthy pods minus N.
• With maxUnavailable: M, disruptions allowed = M minus currently unhealthy pods.

If disruptionsAllowed is 0, evictions return HTTP 429 with the message above, and drain retries. PDBs do not apply to involuntary disruptions like node crashes, and direct pod deletes bypass them.

Cloud upgrade tools behave differently when blocked: some wait, some fail the upgrade after a timeout, and some force-terminate pods after a period. For EKS managed node groups, an upgrade can fail with a PodEvictionFailure if pods cannot be drained in time, unless you choose to force the upgrade.

Environment

Minimal reproduction

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: demo}
spec:
  minAvailable: 2
  selector: {matchLabels: {app: demo}}

With a 2-replica Deployment labelled app: demo, kubectl drain on a node hosting one of its pods never completes.

The fix

Unblock the upgrade safely. We scaled session-store to 3 replicas. Once the third pod was ready, disruptionsAllowed became 1 and the drain continued within seconds. Scaling up, rather than deleting the PDB, kept the protection in place during the rest of the upgrade.

Fix the budget. We changed the PDB to express the real requirement, "at most one replica down at a time", which works for any replica count:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: session-store
  namespace: shop
spec:
  maxUnavailable: 1
  selector:
    matchLabels: {app: session-store}
  unhealthyPodEvictionPolicy: AlwaysAllow

unhealthyPodEvictionPolicy: AlwaysAllow lets drains evict pods that are already not ready, so a crash-looping pod cannot block maintenance forever.

Keep replicas at three for quorum. The session store replicates between members and is only safe to lose one member when there are three. Going to two had been a cost decision that made the service fragile; we reverted it.

Check PDBs before every upgrade. A pre-upgrade script lists any PDB with zero allowed disruptions:

kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name) allowed=0 healthy=\(.status.currentHealthy) desired=\(.status.desiredHealthy)"'

Alternatives and trade-offs

• Delete the PDB during upgrades. Fast, and removes the guardrail exactly when nodes are being drained, which is when it matters.
• Force the node group upgrade. Pods are terminated regardless of PDBs. Acceptable for stateless workloads with enough replicas, risky for stateful ones.
• Percentages (maxUnavailable: 25%). Scale with replica counts; with small counts, rounding can make the budget zero, so check the effective value.

How to test the fix

On a staging cluster with the same workloads:

kubectl drain ip-10-0-12-34.eu-west-1.compute.internal --ignore-daemonsets --delete-emptydir-data --timeout=10m

The drain completed in 3 minutes, evicting one session-store pod and waiting for its replacement to be ready before the next node.

Confirming in production

• The upgrade finished the remaining nodes without stalls.
• The pre-upgrade check found four other PDBs with zero allowed disruptions, all caused by replica counts reduced after PDBs were written. All were fixed the same week.

Regression guard

• A Kyverno policy rejects PDBs using minAvailable equal to or greater than the target workload's replica count, and recommends maxUnavailable.
• An alert on any PDB with disruptionsAllowed == 0 for more than an hour outside of deploys.
• The PDB check runs as the first step in our upgrade runbook.

Security implications

None directly. Upgrades are how security patches reach nodes, though, so anything that blocks them extends exposure to known vulnerabilities. A PDB that blocks drains forever is effectively a patching blocker and should be treated with the same urgency.

Prevention checklist

• Prefer maxUnavailable over minAvailable for most workloads.
• Never let a PDB require all replicas to be available.
• Review PDBs whenever replica counts change.
• Set unhealthyPodEvictionPolicy: AlwaysAllow unless you have a reason not to.
• Check disruptionsAllowed across the cluster before upgrades.

References

• Kubernetes, specifying a disruption budget — https://kubernetes.io/docs/tasks/run-application/configure-pdb/
• Kubernetes disruptions concepts — https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
• API-initiated eviction — https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/
• Amazon EKS managed node group update behavior — https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html