#backoff

Performance & Scaling Bug Report

Retry storm without jitter: how a 20-second database blip became a 41-minute outage

Daniel Tran · Jun 5, 2025 · 10 min read

A routine database failover lasted 20 seconds, but our API stayed down for 41 minutes. Three layers each made up to three attempts with no delay, multiplying load on the database by 27. Here is the math, the fix and how we proved it.

Performance & Scaling Bug Report

LLM API 429 errors at 40 requests per minute: token limits, headers and backoff

Daniel Tran · May 8, 2025 · 9 min read

Our document tagger hit HTTP 429 constantly while sending only 40 requests a minute, far below the request limit. The token-per-minute limit was the real ceiling, inflated by an oversized max_tokens and minute-aligned bursts. How we read the headers and built a client-side limiter.