Kubernetes namespace stuck in Terminating for days: finalizers and a dead APIService

Our preview environments live in one namespace per pull request. When a PR closes, a cleanup job deletes the namespace. The cluster had 63 namespaces in Terminating, the oldest eight days old, and new preview deploys started failing because a quota on the number of namespaces had been reached.

$ kubectl get ns | grep Terminating | head -3
pr-4812   Terminating   8d
pr-4815   Terminating   8d
pr-4820   Terminating   7d

Reproducing it

The namespace object explains why it is stuck, in its status conditions:

$ kubectl get ns pr-4812 -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
NamespaceDeletionDiscoveryFailure=True: Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1
NamespaceDeletionGroupVersionParsingFailure=False: All legacy kube types successfully parsed
NamespaceDeletionContentFailure=False: All content successfully deleted, may be waiting on finalization
NamespaceContentRemaining=True: Some resources are remaining: databaseclaims.db.example.com has 1 resource instances
NamespaceFinalizersRemaining=True: Some content in the namespace has finalizers remaining: databaseclaim.db.example.com/finalizer in 1 resource instances

Two separate problems.

What we ruled out

• Pods stuck terminating. No pods remained in these namespaces.
• PersistentVolumeClaims with protection finalizers. All PVCs had been deleted.
• RBAC of the cleanup job. The delete call succeeded; the namespace entered Terminating as expected.

Root cause

First, metrics.k8s.io/v1beta1 was registered as an APIService backed by metrics-server, which had been crash-looping since a node pool change. The namespace controller must list every resource type in the namespace before it can finish deletion; when discovery for any API group fails, it cannot be sure the namespace is empty, so it waits.

Second, preview environments created a DatabaseClaim custom resource handled by an internal operator. That operator had been uninstalled from the cluster a week earlier during a migration. Its finalizer on each claim was never removed, because the controller that should process it no longer existed.

What is happening under the hood

Deleting a namespace sets a deletion timestamp and the kubernetes finalizer on the namespace. The namespace controller then:

1. Discovers all namespaced resource types through the API, including aggregated APIs (APIServices) and CRDs.
2. Deletes every object of every type in the namespace.
3. Waits for those objects to actually disappear, which requires their finalizers to be removed by whatever controller owns them.
4. Removes its own finalizer from the namespace, which then disappears.

An unavailable APIService breaks step 1 for every namespace in the cluster. A finalizer without a running controller blocks step 3 for every object that has it. Finalizers exist so a controller can clean up external resources (here, a real database) before the object is removed. Removing the finalizer by hand skips that cleanup.

Environment

Minimal reproduction

# 1. a CR with a finalizer and no controller
kubectl apply -f crd.yaml
kubectl -n demo apply -f - <<'EOF'
apiVersion: db.example.com/v1
kind: DatabaseClaim
metadata: {name: test, finalizers: [databaseclaim.db.example.com/finalizer]}
spec: {size: small}
EOF
kubectl delete ns demo      # hangs in Terminating forever

The fix

1. Fix discovery. metrics-server was failing to scrape kubelets on the new node pool because of a TLS setting. We fixed its arguments and confirmed the APIService was available:

kubectl get apiservice v1beta1.metrics.k8s.io
# NAME                     SERVICE                      AVAILABLE   AGE
# v1beta1.metrics.k8s.io   kube-system/metrics-server   True        412d

If an aggregated API is gone for good (the backing service was removed), delete the APIService object itself, not the namespaces.

2. Clean up what the finalizer was protecting, then release it. The DatabaseClaim finalizer existed to drop a per-preview database on the shared Postgres server. With the operator gone, nobody did that. We listed the claims, dropped the matching databases ourselves with the same naming scheme the operator used, and only then removed the finalizers:

kubectl get databaseclaims -A -o json \
  | jq -r '.items[] | select(.metadata.deletionTimestamp) | "\(.metadata.namespace) \(.metadata.name) \(.status.databaseName)"' \
  > orphaned-claims.txt

# after dropping each database listed in orphaned-claims.txt:
while read ns name db; do
  kubectl -n "$ns" patch databaseclaim "$name" --type=merge -p '{"metadata":{"finalizers":[]}}'
done < orphaned-claims.txt

All 63 namespaces finished deleting within a minute of the last patch.

3. Reinstall or retire the operator properly. The migration plan had removed the operator before removing its custom resources. The correct order is: stop creating CRs, delete existing CRs while the operator still runs, wait for them to disappear, then uninstall the operator and finally the CRD.

Alternatives and trade-offs

• Remove the namespace's own finalizer through the finalize subresource. Popular online; it makes the namespace vanish while objects with finalizers may remain in etcd as orphans, and external resources leak. Last resort, after you know what you are skipping.
• Keep the operator running in a minimal mode until all CRs are gone. Cleaner, if the operator is still installable.

How to test the fix

kubectl create ns pr-test && kubectl -n pr-test apply -f preview/
kubectl delete ns pr-test --wait=true --timeout=120s && echo deleted

The full preview lifecycle (create, deploy, delete) now runs nightly as a synthetic test.

Confirming in production

• Zero namespaces in Terminating for more than 10 minutes since the fix.
• The preview cleanup job's success rate went from 0 percent over the bad week back to 100 percent.
• The shared Postgres server lost 58 orphaned preview databases after our manual cleanup, freeing 41 GB.

Regression guard

• Alert on any APIService with Available=False for more than 5 minutes.
• Alert on namespaces in Terminating for more than 30 minutes, with the namespace conditions included in the alert text.
• An operator uninstall checklist in our platform runbook: CRs first, operator second, CRD last.

Security implications

Removing finalizers blindly can leave external resources behind: databases, cloud load balancers, DNS records, IAM roles. Orphaned resources cost money and, worse, may stay reachable with credentials nobody tracks anymore. Our orphaned preview databases contained copies of seeded test data only, but the same pattern with production-like data would be a data exposure risk. Always find out what a finalizer protects before removing it.

Prevention checklist

• Read namespace conditions first; they name the blocker.
• Keep aggregated APIs healthy or remove dead APIService registrations.
• Uninstall operators only after their custom resources are gone.
• Treat finalizer removal as skipping cleanup, and do the cleanup yourself.
• Alert on long-running Terminating namespaces.

References

• Kubernetes finalizers — https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/
• Namespace deletion and status conditions — https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
• Kubernetes API aggregation layer — https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/
• metrics-server — https://github.com/kubernetes-sigs/metrics-server