Retry storm without jitter: how a 20-second database blip became a 41-minute outage

On a Thursday morning our managed PostgreSQL primary for the ordering platform failed over to its standby during a scheduled maintenance window. The provider documents failovers of 15 to 30 seconds, and the logs showed connections refused for about 20 seconds. The ordering API, however, returned errors for 41 minutes. Mobile users saw a spinner and then "Something went wrong", and the API gateway logged a wall of upstream failures:

[2025-05-22T08:03:41.552Z] "POST /v2/orders HTTP/1.1" 503 URX 0 91 30012 - "-" "shop-ios/7.14.0" upstream=orders-svc
[2025-05-22T08:03:41.559Z] "GET /v2/cart HTTP/1.1" 503 URX 0 91 30009 - "-" "shop-android/7.14.1" upstream=orders-svc

URX is Envoy's response flag for "upstream retry limit exceeded". In the order service the logs were dominated by a single error from the database driver:

failed to acquire connection: context deadline exceeded (pool: 50/50 in use, 1,947 waiting)

The database itself was back and accepting connections at 08:01:12. Twenty-six minutes later it was still at 100% CPU, with pg_stat_activity showing the same handful of cart and order queries repeated thousands of times.

Reproducing it

We rebuilt the topology in a staging environment with production-shaped load (about 2,400 requests per second at the gateway) and injected a 20-second outage by blocking the database port with a network policy. The pattern reproduced on the first run: request volume at the database jumped from roughly 3,100 queries per second to more than 60,000 attempted queries per second in the first minute after connectivity came back, and the service did not recover until we cut client traffic.

What we ruled out

• The failover itself. The provider's event log and our connection metrics agreed: 20 seconds of refused connections, then a healthy new primary with the same capacity as the old one.
• A slow query introduced by a deploy. The last deploy was two days earlier, and the query plans for the top statements were unchanged when we compared EXPLAIN output against a snapshot.
• Connection limits on the database. max_connections was 800 and we peaked at 610. The problem was not that we could not connect; it was that every connection was busy doing duplicate work.

Root cause

Three layers of our stack retried failures independently, each immediately and without coordination:

When the database blipped, one user action could become 3 x 3 x 3 = 27 attempts against the database. Three attempts per layer, three layers deep: 27x the load. Worse, the retries were synchronised: every client that saw an error at the same moment retried at the same moment, so the recovering database received traffic in waves rather than a smooth stream. Each wave filled the connection pool, queries queued behind it, request timeouts expired, and the expired requests were retried again. The system had a stable bad state, which is why it stayed down long after the original cause was gone.

What is happening under the hood

Retries are a bet that a failure is transient. That bet is good when failures are rare and independent, for example one bad host behind a load balancer. It becomes a bad bet when the failure is shared and caused by load, because every retry adds load to the thing that is already struggling.

Two effects compound:

• Amplification. Retries at each layer multiply. With n attempts per layer and k layers, the worst case is n^k calls to the bottom dependency. Three attempts per layer and three layers give 27. Raising each layer to four attempts (one initial plus three retries) gives 64.
• Synchronisation. Without randomness, clients that failed together retry together. Even with exponential backoff, if every client waits exactly 100 ms, then 200 ms, then 400 ms, the load arrives in spikes at those moments. Adding jitter spreads them out.

Timeouts make it worse. Our gateway timeout was 30 seconds, but the mobile client gave up after 10 seconds and retried. The gateway kept processing the abandoned request, so for a while we were doing work for requests nobody was waiting for.

Environment

Minimal reproduction

The order service helper looked harmless:

// Before: immediate retries, no jitter, no budget.
func withRetry(ctx context.Context, fn func(context.Context) error) error {
	var err error
	for attempt := 0; attempt < 3; attempt++ {
		if err = fn(ctx); err == nil {
			return nil
		}
	}
	return err
}

And the gateway route had been copied from an example:

retry_policy:
  retry_on: "5xx,connect-failure"
  num_retries: 2

Neither is wrong in isolation. Together with a mobile client doing the same thing, they form the 27x multiplier.

Who is affected

Any system with more than one layer of automatic retries: SDKs that retry by default, service meshes, gateways, job runners and database drivers. It hits hardest after a shared dependency fails briefly, such as a failover, a DNS hiccup or a deploy that restarts many pods at once.

The fix

We made four changes. The first one alone would have prevented most of the outage.

1. One layer owns retries. We chose the gateway, because it sees the whole request and can enforce budgets centrally. The order service no longer retries database calls inside a request; it returns a clear error and lets the caller decide. The mobile client now retries only idempotent GET requests and requests that carry an idempotency key, at most once, and only when the server says it is safe.

2. Exponential backoff with full jitter wherever retries remain. For background jobs that still retry in the service, the delay is a random value between zero and an exponentially growing cap:

func backoff(attempt int, base, max time.Duration) time.Duration {
	ceiling := base << attempt // base * 2^attempt
	if ceiling <= 0 || ceiling > max {
		ceiling = max
	}
	return time.Duration(rand.Int64N(int64(ceiling)))
}

func withRetry(ctx context.Context, b *RetryBudget, fn func(context.Context) error) error {
	err := fn(ctx)
	for attempt := 0; err != nil && attempt < 2; attempt++ {
		if !isTransient(err) || !b.Allow() {
			return err
		}
		select {
		case <-time.After(backoff(attempt, 50*time.Millisecond, 2*time.Second)):
		case <-ctx.Done():
			return ctx.Err()
		}
		err = fn(ctx)
	}
	return err
}

3. Retry budgets instead of fixed counts. A fixed count says "each request may retry twice". A budget says "retries may add at most 10% on top of normal traffic". When everything is failing, the budget runs out quickly and the system stops amplifying. Envoy supports this directly on the cluster's circuit breakers:

circuit_breakers:
  thresholds:
    - priority: DEFAULT
      retry_budget:
        budget_percent:
          value: 10.0
        min_retry_concurrency: 5

The route keeps num_retries: 2, but the budget now caps how many of those retries can be in flight at once relative to active requests.

4. Timeouts that shrink down the stack. The client waits 10 seconds, the gateway 8 seconds, the service passes the remaining deadline to the database with context.WithTimeout. A layer below should never keep working after the layer above has given up.

We also added a circuit breaker around the database pool in the service: after 50% of calls fail in a 10-second window, it fails fast for 5 seconds and then lets a few probe requests through. That keeps the pool from filling with doomed work while the database recovers.

Alternatives and trade-offs

• Retry only at the edge (client). Simple, but you lose central control, and you cannot ship a fix to old app versions quickly.
• Retry only at the lowest layer. Cheap per retry, but that layer often lacks the context to know whether the whole operation is still worth doing.
• Server-driven retry hints. Returning Retry-After on 503 and 429 lets clients back off by the amount the server needs. We added it, and the mobile client honours it, but it only helps clients that read it.
• Load shedding. Rejecting excess work early with a cheap 503 is often better than accepting it and timing out. We added an admission limit at the gateway as a later action item.

How to test the fix

We re-ran the staging fault injection with the same 20-second database blackout and the same load:

kubectl apply -f chaos/block-postgres-20s.yaml
k6 run --vus 800 --duration 10m load/orders-mixed.js

We also added a unit test for the backoff function to make sure delays are bounded and actually random:

func TestBackoffIsBoundedAndJittered(t *testing.T) {
	seen := map[time.Duration]bool{}
	for i := 0; i < 1000; i++ {
		d := backoff(3, 50*time.Millisecond, 2*time.Second)
		if d < 0 || d >= 400*time.Millisecond {
			t.Fatalf("delay %v out of range", d)
		}
		seen[d] = true
	}
	if len(seen) < 100 {
		t.Fatalf("expected jittered delays, got %d distinct values", len(seen))
	}
}

Confirming in production

In staging the database peaked at 3,600 queries per second after the blackout instead of over 60,000, and error rates were back to baseline 9 seconds after connectivity returned. Two weeks later the provider performed another maintenance failover in production. This time:

• Gateway 5xx rate peaked at 7.8% for 24 seconds and returned to baseline without intervention.
• Envoy's upstream_rq_retry_overflow counter for the orders cluster increased during the event, showing the budget capping retries as designed.
• Database CPU rose from 38% to 52% and fell back within two minutes. No queue built up in the connection pool.

Regression guard

• A CI check scans our Envoy route configs and fails if a cluster has retry_policy without a retry_budget.
• A lint rule in the Go service flags loops that call a database function more than once without going through the shared withRetry helper.
• The 20-second database blackout runs as a monthly game day in staging, with a pass condition of recovery within 60 seconds.

Security implications

Retry amplification is also a denial-of-service amplifier: a cheap request from an attacker that triggers errors downstream could cost us up to 27 database calls before this change. Budgets and single-layer retries reduce that multiplier for malicious traffic as well. We made sure retries only apply to idempotent operations or requests with idempotency keys, which also prevents duplicate orders and duplicate charges when a retry follows a request that actually succeeded.

Prevention checklist

• Decide which layer owns retries for each call path and disable them elsewhere.
• Always combine exponential backoff with jitter; never retry in a tight loop.
• Cap retries with a budget relative to traffic, not only a per-request count.
• Make timeouts decrease down the stack and propagate deadlines.
• Only retry idempotent operations, or use idempotency keys.

References

• AWS Builders' Library: Timeouts, retries, and backoff with jitter — https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
• AWS Architecture Blog: Exponential Backoff and Jitter — https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• Google SRE Book: Addressing Cascading Failures — https://sre.google/sre-book/addressing-cascading-failures/
• Envoy documentation: Retry policy and retry budgets — https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/circuit_breaker.proto
• RFC 9110, Retry-After — https://www.rfc-editor.org/rfc/rfc9110#name-retry-after