Designing an LLM gateway: timeouts, retries and provider fallbacks

When our third team shipped an LLM feature, we found three different retry policies, two hardcoded API keys in config maps, no shared view of spend, and one service that retried a 60-second streaming call five times on any error. During a provider incident, that service multiplied its load by six and made its own outage worse. We consolidated LLM access behind a small internal gateway.

This post is the design we ended up with: what the gateway does, the rules for timeouts and retries, how fallback works, and what it deliberately does not do. It applies whether you build the gateway yourself or configure an off-the-shelf one.

What the gateway is for

Keep the scope small. Ours does six things:

1. Holds provider credentials. Services authenticate to the gateway with their own identity; provider keys never leave it.
2. Applies timeouts and retries consistently per call type.
3. Routes and falls back between models and providers by policy.
4. Enforces budgets and rate limits per service and per feature.
5. Emits telemetry: latency, tokens, cost, errors, per service and model.
6. Normalises errors into a small set of categories services can act on.

It does not own prompts or business logic, and it does not do semantic caching by default. Those belong to the features.

Timeouts by call type

A single global timeout is always wrong for someone. We classify calls:

First-token timeout matters most for streaming. A provider that accepted the connection but has not sent a single token after 10 seconds is usually overloaded; waiting another 80 seconds helps nobody.

Retry rules

Retries are where gateways cause outages, so the rules are strict:

• Retry only retryable errors: connection errors, 429, 500, 502, 503, 504, and provider-specific "overloaded" errors. Never retry 400-class validation errors or content policy refusals.
• Honour Retry-After when present; otherwise exponential backoff with full jitter.
• One layer of retries. Services call the gateway with retries disabled in their SDKs. Nested retries multiply: three layers of three retries is 27 attempts.
• A retry budget. Retries may add at most 10 percent to traffic per provider over a rolling minute. When the budget is spent, fail fast.
• Never retry a stream after tokens were sent to the client. The user already saw partial output. Report the error instead, and let the client decide.

import random, time

RETRYABLE = {408, 429, 500, 502, 503, 504}

def call_with_retry(send, max_attempts=3, base=0.5, cap=8.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return send()
        except ProviderError as e:
            if e.status not in RETRYABLE or attempt == max_attempts or not retry_budget.take():
                raise
            delay = e.retry_after or random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(delay)

Fallback: model first, then provider

Fallback is a policy per feature, not a global switch, because models are not interchangeable. Each route lists ordered targets that a feature's owners have actually evaluated:

routes:
  support-answer:
    targets:
      - provider: primary
        model: large-2026-01
      - provider: primary
        model: large-2025-08        # previous snapshot, already evaluated
      - provider: secondary
        model: secondary-large      # passed the support eval within tolerance
    fallback_on: [timeout, overloaded, unavailable]
    max_fallbacks: 1
  invoice-extract:
    targets:
      - provider: primary
        model: small-2026-01
    fallback_on: []                 # strict schema; no untested fallback

Rules we learned the hard way:

• Only fall back to targets that passed the feature's eval. A fallback that produces subtly worse output is a silent incident.
• Prompts may need per-model variants. Store them with the route, not in the service.
• Use circuit breakers. If the primary is failing, skip it for a short window instead of timing out every request first. Half-open after 30 seconds and let a few requests probe.
• Fall back once. A request that has failed on two targets should fail, not tour every provider.

Streaming through the gateway

Streaming needs the gateway to pass bytes through as they arrive. Disable response buffering in every proxy layer (we covered the nginx side in an earlier post on streaming arriving all at once). Count tokens from the final usage chunk when the provider supplies it, and estimate from the streamed text when it does not. Fallback for streams is only possible before the first token is forwarded.

Budgets and limits

Every request carries a service identity and a feature tag. The gateway tracks tokens and cost per feature per hour and per day, and enforces limits set by the feature owner. When a limit is hit, the error category is budget_exceeded, which services handle differently from provider errors (usually by degrading the feature rather than retrying).

Observability

Per request, the gateway logs metadata, not prompts: service, feature, route, target chosen, attempt count, fallback used, status category, latency, time to first token, input and output tokens, estimated cost, and a trace id. With that, the questions that matter are one query away: which feature drives spend, which provider is slow this week, how often fallbacks fire.

Common mistakes

Key takeaways

• Centralise credentials, timeouts, retries, budgets and telemetry in one place.
• Classify calls and set timeouts per class, including time to first token.
• Retry only retryable errors, once per stack, within a budget.
• Make fallback a per-feature policy using evaluated targets and circuit breakers.
• Log metadata so cost and reliability questions are easy to answer.

References

• AWS Architecture Blog, exponential backoff and jitter — https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• Google SRE book, handling overload — https://sre.google/sre-book/handling-overload/
• OpenAI error codes — https://platform.openai.com/docs/guides/error-codes
• Anthropic API errors — https://docs.anthropic.com/en/api/errors