LLM API 429 errors at 40 requests per minute: token limits, headers and backoff

We run a background service that tags incoming documents (contracts, invoices, support attachments) with categories and a short summary using an LLM API. After a customer onboarding doubled the daily document count, about a third of calls started failing with HTTP 429. The odd part: the service sent around 40 requests per minute, and the organisation's request limit for that model was 5,000 RPM. The error body explained more than the dashboard did:

{
  "error": {
    "message": "Rate limit reached for gpt-4o-mini in organization org-xxxxxxxx on tokens per min (TPM): Limit 200000, Used 192380, Requested 11650. Please try again in 1.209s. Visit https://platform.openai.com/account/rate-limits to learn more.",
    "type": "tokens",
    "param": null,
    "code": "rate_limit_exceeded"
  }
}

We were not limited on requests. We were limited on tokens, and each request was being counted as roughly 11,650 tokens even though the usage data said a typical call consumed about 3,800 input and 250 output tokens.

Reproducing it

The failures clustered at the start of each minute. Our scheduler pulled a batch of 40 documents every 60 seconds and fanned them out with asyncio.gather, so all 40 requests left within about 300 ms. Running one batch by hand against a staging project with the same limits reproduced 429s on 11 of 40 requests. Sending the same 40 requests spaced evenly across the minute produced none.

What we ruled out

• A shared key with another service. The organisation had several projects, but per-project usage in the dashboard showed this service consuming most of the TPM for that model.
• A provider incident. The status page was green and the error explicitly named the TPM limit, not a capacity problem.
• The SDK hiding retries. It was doing the opposite: the official Python SDK retries 429s twice by default with its own backoff, so each failure we logged had already cost three attempts against the same limit.

Root cause

Three things stacked:

1. max_tokens was set to 8,192 "to be safe". The provider's rate-limit guide explains that max_tokens is part of how a request is counted against TPM before the model has produced anything, because the limiter cannot know the real output length up front. In our logs the "Requested" figure in every 429 sat close to the request's input tokens plus 8,192. A call that really used about 4,000 tokens was being admitted as if it might use nearly three times that.
2. Bursts. All 40 requests landed in the same second. Rate limits are measured per minute but are not a simple "reset at :00" counter, so a burst can exhaust capacity even when the minute's average is fine.
3. Retries without jitter. The SDK's built-in retries and our own retry decorator both fired, and the 40 clients retried at nearly the same moments, so the second wave also hit the limit.

What is happening under the hood

LLM providers limit several dimensions at once. The exact set differs per provider, so it is worth reading the docs for the one you use:

Each response tells you where you stand. On the OpenAI API, these headers come back on every call:

x-ratelimit-limit-requests: 5000
x-ratelimit-limit-tokens: 200000
x-ratelimit-remaining-requests: 4961
x-ratelimit-remaining-tokens: 8104
x-ratelimit-reset-requests: 12ms
x-ratelimit-reset-tokens: 57.5s

Anthropic's API returns the equivalent as anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining, anthropic-ratelimit-output-tokens-remaining and related -limit and -reset headers, plus retry-after on 429 responses. Its documentation also notes that limits use a token bucket that refills continuously and that short bursts can trigger errors even when the per-minute total is under the limit. Whatever the provider, the lesson is the same: the headers, not your request count, are the source of truth.

Environment

Minimal reproduction

# Excerpt: the original batch loop.
async def tag_batch(docs):
    return await asyncio.gather(*(tag(doc) for doc in docs))  # 40 at once

@retry(stop=stop_after_attempt(3), wait=wait_fixed(1))  # second retry layer
async def tag(doc):
    return await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=build_messages(doc),
        max_tokens=8192,
    )

Who is affected

Any workload with long prompts (RAG, document processing, code review), large max_tokens values, bursty schedulers or several retry layers. It often appears after a growth step: the same code was fine at half the volume.

The fix

1. Right-size max_tokens. Our tagging output is a small JSON object. The 99.9th percentile of real output was 410 tokens, so we set max_tokens=800. That single change cut the tokens counted per request by more than half.

2. One retry layer, honouring the server. We set max_retries=0 on the SDK client and removed the decorator, then wrote one retry loop that respects retry-after when present and otherwise uses exponential backoff with full jitter:

import asyncio, random
from openai import AsyncOpenAI, RateLimitError

client = AsyncOpenAI(max_retries=0, timeout=60)

async def call_with_backoff(make_call, attempts=5, base=0.5, cap=20.0):
    for attempt in range(attempts):
        try:
            return await make_call()
        except RateLimitError as e:
            if attempt == attempts - 1:
                raise
            retry_after = e.response.headers.get("retry-after")
            delay = float(retry_after) if retry_after else random.uniform(0, min(cap, base * 2 ** attempt))
            await asyncio.sleep(delay + random.uniform(0, 0.25))

3. A client-side limiter that budgets tokens, not just requests. Before sending, we estimate the request's cost as input tokens (counted locally with the provider's tokenizer) plus max_tokens, and take that many tokens from a local bucket that refills at 85% of our TPM limit. Requests that do not fit wait in a queue instead of failing:

class TokenBucket:
    def __init__(self, per_minute: int):
        self.capacity = per_minute
        self.tokens = per_minute
        self.rate = per_minute / 60.0
        self.updated = asyncio.get_running_loop().time()
        self.lock = asyncio.Lock()

    async def take(self, amount: int):
        async with self.lock:
            while True:
                now = asyncio.get_running_loop().time()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= amount:
                    self.tokens -= amount
                    return
                await asyncio.sleep((amount - self.tokens) / self.rate)

tpm = TokenBucket(per_minute=170_000)  # 85% of 200k, shared by this replica's workers

With two replicas we split the budget between them through a Redis-backed version of the same bucket, so the replicas do not each assume they own the whole limit.

4. Smooth the scheduler. Workers pull from a queue continuously with a concurrency cap of 8, instead of a cron-style batch at the top of each minute.

Alternatives and trade-offs

• Ask for a higher limit or move up a usage tier. Reasonable once the client is well behaved, but it only moves the ceiling.
• Batch APIs. For work that can wait hours, providers offer asynchronous batch endpoints with separate limits and lower prices. Our tagging has a 15-minute freshness target, so we use it only for nightly re-tagging.
• Smaller inputs. We trimmed boilerplate (signature blocks, repeated headers) from documents before sending, which saved about 18% of input tokens.
• Fallback models or providers. Useful for availability, but outputs must be validated per model, and it adds operational surface.

How to test the fix

We replayed one hour of production documents against the staging project with the same limits:

python -m tagger.replay --input samples/2025-05-02-hour.jsonl --rate-from-file

The replay logs the headers from every response, so we could plot x-ratelimit-remaining-tokens over time and confirm it never dropped to zero. A unit test covers the backoff path by mocking a 429 with and without retry-after and asserting the delays used.

Confirming in production

• 429 responses fell from about 31% of calls to 0.2%, and every remaining one succeeded on its first retry.
• Average tokens counted per request, read from the difference in x-ratelimit-remaining-tokens, dropped from about 11,600 to about 4,600.
• Median end-to-end tagging latency rose slightly, from 3.1 to 3.6 seconds, because some requests now wait briefly in the queue, but p99 fell from 74 seconds to 9 seconds because nothing loops through failed retries anymore.

Regression guard

• A metric tracks the minimum remaining-tokens value seen per minute; an alert fires if it stays under 5% for 10 minutes, which tells us we are close to the ceiling before users see errors.
• A unit test fails if any call site sets max_tokens above the documented maximum output for its feature.
• Code review checklist item: no new retry wrappers around the LLM client; use call_with_backoff.

Security implications

Rate limits are also a cost and abuse control. The client-side limiter now enforces a per-tenant share of the budget, so one customer uploading 10,000 documents cannot starve everyone else or run up the bill. We log rate-limit headers but never request or response bodies at this layer, since documents contain customer data. API keys stay in the secret manager and are scoped to one project, so a leaked key cannot consume another service's limits.

Prevention checklist

• Know which limit you are hitting: read the 429 message and the rate-limit headers.
• Set max_tokens from measured output sizes, not "just in case" values.
• Budget tokens on the client and smooth bursts with a queue and concurrency cap.
• Keep exactly one retry layer, honour retry-after, add jitter.
• Monitor remaining-capacity headers, not only error counts.

References

• OpenAI API documentation: Rate limits — https://platform.openai.com/docs/guides/rate-limits
• Anthropic documentation: Rate limits — https://docs.anthropic.com/en/api/rate-limits
• AWS Architecture Blog: Exponential Backoff and Jitter — https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• RFC 6585, Section 4: 429 Too Many Requests — https://www.rfc-editor.org/rfc/rfc6585#section-4