502 and 503 spikes on every Kubernetes rollout: readiness probes, preStop and graceful shutdown

Our pricing API, a Go service behind ingress-nginx, is called by checkout on every cart change. Each time we deployed it, the error-rate graph showed a sharp spike for 30 to 60 seconds: around 1.5 percent of requests failing with 502 or 503, enough to trigger checkout's circuit breaker and show customers "prices temporarily unavailable". We had accepted it as the cost of deploying, until we started deploying ten times a day.

The ingress-nginx controller logs during a rollout had two kinds of errors:

2025/02/11 14:03:17 [error] 1184#1184: *88213 connect() failed (111: Connection refused) while connecting to upstream,
  client: 10.0.12.44, server: pricing.internal.example.com, request: "POST /v2/quote HTTP/1.1",
  upstream: "http://10.0.41.17:8080/v2/quote", host: "pricing.internal.example.com"
2025/02/11 14:03:18 [error] 1184#1184: *88240 upstream prematurely closed connection while reading response header from upstream,
  client: 10.0.12.51, server: pricing.internal.example.com, request: "POST /v2/quote HTTP/1.1",
  upstream: "http://10.0.41.17:8080/v2/quote", host: "pricing.internal.example.com"

And in the access log, a burst of 503s right as new pods appeared.

Reproducing it

We ran a steady load of 300 requests per second against staging and triggered a rollout:

k6 run --vus 60 --duration 3m load/quote.js &
kubectl -n pricing rollout restart deployment/pricing-api

Every run produced between 200 and 900 failed requests. It was fully reproducible, which was good news.

What we ruled out

• Application errors in the new version. kubectl rollout restart redeploys the same image, and it still failed.
• Resource starvation. Nodes had plenty of headroom, and failures stopped as soon as the rollout finished.
• ingress-nginx configuration reloads. Since the controller moved endpoint updates to a Lua balancer, endpoint changes do not reload nginx; the controller logs confirmed no reload during the rollout.

Root cause

The deployment had no readiness probe, and the Go server had no shutdown handling. Two separate races followed from that:

1. Startup: without a readiness probe, a pod is considered ready as soon as its container starts. Our service spends 4 to 6 seconds loading price rules from Redis before it opens its HTTP listener. During that window the ingress sent requests to a pod that refused connections.
2. Shutdown: when Kubernetes terminates an old pod, it sends SIGTERM and, at the same time, starts removing the pod from the Service's endpoints. Our Go binary exited immediately on SIGTERM, killing in-flight requests, while ingress-nginx still had the pod's IP in its upstream list for another second or two.

What is happening under the hood

Pod termination is not sequenced the way most of us assume. When a pod is deleted:

1. The API server marks the pod as terminating.
2. In parallel:
   • the kubelet runs the preStop hook, if any, then sends SIGTERM to the container;
   • the EndpointSlice controller removes the pod from ready endpoints, and every consumer of those endpoints (kube-proxy on every node, ingress controllers, service meshes) notices the change on its own schedule.
3. After terminationGracePeriodSeconds (default 30), anything still running gets SIGKILL.

There is no guarantee that step 2's routing updates finish before the process exits. If the app exits the instant it gets SIGTERM, there is a window where traffic is still routed to a pod that is gone. A short preStop sleep delays SIGTERM long enough for routing to catch up; graceful shutdown in the app then finishes requests that were already accepted.

On the startup side, a pod only enters the ready endpoints when its readiness probe succeeds. Without a probe, "container started" counts as ready. A rolling update also uses readiness to decide when to move on to the next pod, so missing probes make the whole rollout go faster than the app can keep up.

Environment

Minimal reproduction

The server, before:

func main() {
    rules := loadRules() // 4-6 s
    http.HandleFunc("/v2/quote", quoteHandler(rules))
    log.Fatal(http.ListenAndServe(":8080", nil)) // exits immediately on SIGTERM
}

Any deployment without probes running this code, under load, shows errors on rollout.

Who is affected

Any Kubernetes workload behind a Service or Ingress that lacks a readiness probe, exits immediately on SIGTERM, or uses a language runtime that ignores SIGTERM until SIGKILL (for example a shell script as PID 1 that does not forward signals). It is more visible with ingress controllers and external load balancers that route directly to pod IPs, and with long-lived keep-alive connections.

The fix

1. A server that reports readiness honestly and shuts down gracefully.

var ready atomic.Bool

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/healthz/live", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
    mux.HandleFunc("/healthz/ready", func(w http.ResponseWriter, _ *http.Request) {
        if !ready.Load() {
            http.Error(w, "not ready", http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })

    srv := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
    go func() {
        if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            log.Fatalf("listen: %v", err)
        }
    }()

    rules := loadRules()
    mux.HandleFunc("/v2/quote", quoteHandler(rules))
    ready.Store(true)

    // Graceful shutdown on SIGTERM.
    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
    defer stop()
    <-ctx.Done()

    ready.Store(false)
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf("shutdown: %v", err)
    }
}

http.Server.Shutdown closes listeners, closes idle keep-alive connections and waits for active requests to complete, up to the context deadline.

2. Probes, a preStop sleep and an adequate grace period in the deployment.

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: pricing-api
          image: registry.example.com/pricing-api:2025.02.14-1
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              sleep:
                seconds: 10
          startupProbe:
            httpGet: { path: /healthz/ready, port: 8080 }
            periodSeconds: 1
            failureThreshold: 30
          readinessProbe:
            httpGet: { path: /healthz/ready, port: 8080 }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz/live, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3

The timing budget: 10 seconds of preStop sleep for routing to converge, then up to 20 seconds of graceful shutdown, inside a 45-second grace period, leaving margin before SIGKILL. The grace period covers the preStop hook too, so it must be longer than the sleep plus the shutdown timeout.

The sleep action for preStop is built in since Kubernetes 1.30 (beta, enabled by default). On older clusters, use exec: { command: ["sleep", "10"] }, which requires a sleep binary in the image; our distroless image has none, which is why the built-in action mattered to us.

maxUnavailable: 0 keeps full capacity during the rollout, so the surge pods must become ready before any old pod is removed.

Alternatives and trade-offs

• Sleep in the application on SIGTERM instead of preStop. Works without cluster support but mixes infrastructure timing into app code. We prefer the manifest.
• Pod readiness gates with a cloud load balancer. When a cloud load balancer routes directly to pod IPs, its controller can hold pods unready until the target is healthy in the load balancer. Use it in addition to the above, not instead.
• Retries at the ingress. nginx retries idempotent requests on connection errors by default, but our quote endpoint is a POST and is not retried. Masking errors with retries also hides the real problem.

How to test the fix

k6 run --vus 60 --duration 5m load/quote.js &
for i in 1 2 3 4 5; do
  kubectl -n pricing rollout restart deployment/pricing-api
  kubectl -n pricing rollout status deployment/pricing-api --timeout=5m
done
wait
# k6 summary: http_req_failed..........: 0.00%  0 out of 89412

Confirming in production

• Error rate during deploys went from a 1.5 percent spike lasting 30 to 60 seconds to no visible change on the graph. Over the next 120 production rollouts, ingress-nginx logged 3 failed requests in total, all client timeouts unrelated to the rollout.
• The checkout circuit breaker for pricing has not opened during a deploy since.
• Rollouts take longer, about 2 min 40 s instead of 50 s, because each pod waits for readiness and drains for 10 seconds. That is the trade we wanted.

Regression guard

• A policy in our admission controller rejects Deployments behind an Ingress without a readiness probe or with terminationGracePeriodSeconds lower than the preStop sleep plus 15 seconds.
• The rollout test above runs in staging on every release candidate and fails the pipeline on any failed request.
• Every Go service uses the same shared server package with graceful shutdown built in.

Security implications

None beyond availability. A readiness endpoint should not expose internals; ours returns only a status code. Keep health endpoints off the public ingress, or restrict them, so they cannot be used to probe for deploy timing.

Prevention checklist

• Every serving pod needs a readiness probe that reflects real readiness.
• Handle SIGTERM with graceful shutdown, and make sure PID 1 receives the signal.
• Add a short preStop sleep so routing updates before the process stops.
• Set terminationGracePeriodSeconds longer than sleep plus shutdown time.
• Run a load test across repeated rollouts before trusting a deployment.

References

• Kubernetes documentation: Pod lifecycle, termination of pods — https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
• Kubernetes documentation: Container lifecycle hooks — https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
• Kubernetes documentation: Configure liveness, readiness and startup probes — https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
• Kubernetes documentation: Deployments, rolling update strategy — https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment
• Go documentation: net/http Server.Shutdown — https://pkg.go.dev/net/http#Server.Shutdown
• ingress-nginx documentation — https://kubernetes.github.io/ingress-nginx/