LLM token streaming arrives all at once in production: nginx proxy buffering and SSE

We shipped a support assistant inside our customer dashboard. In development it felt fast: the first words appeared in under a second and the answer typed itself out. In production, users stared at a blinking cursor for 10 to 20 seconds and then the complete answer appeared in one go. Nothing errored. The browser's network panel showed the request to /api/assistant/stream as a single response with status 200 and content-type: text/event-stream, and the EventStream tab filled with all events at the same instant.

Our own metrics made the gap obvious:

assistant.ttft_ms (server, first token from provider)   p50=  620   p95= 1,180
assistant.ttfb_ms (browser, first byte received)        p50=13,900  p95=21,400

The server was getting tokens from the model provider quickly. The browser was not receiving them until the end.

Reproducing it

curl -N disables curl's own output buffering, so it shows exactly when bytes arrive. Hitting the app container directly, bypassing nginx, events arrived one by one:

curl -N -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"question":"How do I rotate an API key?"}' \
  http://10.0.3.14:8000/api/assistant/stream

The same request through the public hostname sat silent for 12 seconds and then printed everything at once. That narrowed it to the layers between the internet and the app: our nginx reverse proxy.

What we ruled out

• The provider not streaming. We logged a timestamp for every chunk received from the provider SDK. Chunks arrived every 20 to 60 ms from about 600 ms onward.
• The application buffering. Direct curl to the container streamed correctly, and our handler yields each chunk as soon as it arrives.
• The browser or frontend code. Our frontend reads the stream with fetch and a ReadableStream reader; we confirmed with curl that the problem existed before any JavaScript ran.

Root cause

The production nginx configuration had proxy_buffering at its default, on, for the whole API location. With buffering on, nginx reads the upstream response into its buffers and sends it to the client when buffers fill or the response ends. A chat answer of a few kilobytes never fills the buffers, so the browser received it only at the end. On top of that, the same server block had gzip on with text/event-stream in gzip_types, added months earlier by someone copying a "compress all text types" snippet. Compression is a second place where small writes get held back to build a better compressed block. Either one alone was enough to ruin streaming.

What is happening under the hood

Server-Sent Events are a long-lived HTTP response with Content-Type: text/event-stream, where the server writes data: ... lines followed by a blank line for each event. The protocol relies on every intermediary passing bytes through promptly. Several common defaults work against that:

nginx has a per-response escape hatch: if the upstream sends the response header X-Accel-Buffering: no, nginx disables buffering for that response only, unless proxy_ignore_headers is configured to ignore it.

Environment

Minimal reproduction

The app side was fine:

# Excerpt: Ask (request model) and llm_stream (provider wrapper) live elsewhere.
import json

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

async def events(question: str):
    async for chunk in llm_stream(question):  # wraps the provider's streaming API
        yield f"data: {json.dumps({'delta': chunk})}\n\n"
    yield "data: [DONE]\n\n"

@app.post("/api/assistant/stream")
async def stream(body: Ask):
    return StreamingResponse(events(body.question), media_type="text/event-stream")

The proxy side was the problem:

gzip on;
gzip_types text/plain text/css application/json application/javascript text/event-stream;

location /api/ {
    proxy_pass http://app_upstream;
    proxy_set_header Host $host;
}

Who is affected

Anyone streaming LLM output, logs, progress updates or any SSE through nginx, a Kubernetes ingress built on nginx, a CDN or a load balancer that buffers by default. It also affects chunked JSON or NDJSON streaming over plain fetch, not just EventSource. The pattern is always the same: works locally, works hitting the app directly, fails through the edge.

The fix

We fixed it in two places so it does not depend on one layer being configured right.

1. The application declares that the response must not be buffered or cached. These headers travel with the response wherever it is proxied:

SSE_HEADERS = {
    "Cache-Control": "no-cache",
    "X-Accel-Buffering": "no",
}

@app.post("/api/assistant/stream")
async def stream(body: Ask):
    return StreamingResponse(
        events(body.question),
        media_type="text/event-stream",
        headers=SSE_HEADERS,
    )

2. nginx gets an explicit location for streaming routes. Buffering off, no compression, HTTP/1.1 to the upstream and a read timeout long enough for slow generations:

- gzip_types text/plain text/css application/json application/javascript text/event-stream;
+ gzip_types text/plain text/css application/json application/javascript;

+ location /api/assistant/stream {
+     proxy_pass http://app_upstream;
+     proxy_http_version 1.1;
+     proxy_set_header Connection "";
+     proxy_set_header Host $host;
+     proxy_buffering off;
+     proxy_cache off;
+     gzip off;
+     proxy_read_timeout 300s;
+ }

We did not turn buffering off for all of /api/. Buffering exists for good reasons: it frees the upstream worker quickly when clients are slow, which matters for normal JSON endpoints.

3. Heartbeats for long pauses. Some tool-using answers pause for several seconds while the model waits on a tool. We send an SSE comment line every 15 seconds while idle, which keeps every hop's idle timer from closing the connection and is ignored by clients:

yield ": keep-alive\n\n"

Alternatives and trade-offs

• Only X-Accel-Buffering: no. Enough for nginx, and convenient when you cannot edit the proxy config, but it does nothing about gzip and nothing for non-nginx layers.
• WebSockets. No buffering issues once upgraded, but more moving parts: upgrade headers through every proxy, reconnection logic, and no automatic reconnect like EventSource.
• Polling for chunks. Robust behind anything, but adds latency and request overhead. We use it only as a fallback for one corporate customer whose outbound proxy buffers everything.

How to test the fix

curl -N with a timestamp per line makes buffering visible at a glance:

curl -sN https://app.example.com/api/assistant/stream \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"question":"How do I rotate an API key?"}' \
  | while IFS= read -r line; do printf '%s %s\n' "$(date +%T.%3N)" "$line"; done

Before the fix every line had the same timestamp. After it, lines were spread over several seconds. We also checked response headers to confirm no content-encoding: gzip on the stream:

curl -sN -D - -o /dev/null https://app.example.com/api/assistant/stream \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"question":"hi"}' \
  | grep -iE 'content-type|content-encoding|x-accel'

Confirming in production

• Browser-side time to first byte for the assistant dropped from a p50 of 13.9 seconds to 0.71 seconds, matching the server-side first-token time plus network.
• The share of assistant sessions abandoned before the answer appeared fell from 18% to 4% over the following week.
• nginx worker memory and upstream connection counts did not change measurably; only streaming routes are unbuffered.

Regression guard

• A synthetic check runs every five minutes against production, asks a fixed short question and fails if the gap between the first and last SSE event received is under 200 ms, which only happens when something buffers the whole response.
• Our nginx config tests include a rule that text/event-stream must not appear in any gzip_types directive.
• The streaming route's integration test asserts the X-Accel-Buffering: no and Cache-Control: no-cache headers.

Security implications

Turning buffering off means a slow client holds an upstream connection and an app worker for the full duration of the stream. That is a resource exhaustion risk, so the streaming route has its own rate limit per user, a cap on concurrent streams per user and the 300-second read timeout as a hard ceiling. We also made sure Cache-Control: no-cache is present: an intermediary caching an SSE response would be a data exposure problem, since answers can contain account-specific information.

Prevention checklist

• Give streaming endpoints their own proxy location with buffering and compression off.
• Send X-Accel-Buffering: no and Cache-Control: no-cache from the app for every stream.
• Exclude text/event-stream from all compression layers.
• Measure time to first byte at the client, not just time to first token at the server.
• Add heartbeats and explicit timeouts for long generations.

References

• nginx documentation: ngx_http_proxy_module (proxy_buffering, X-Accel-Buffering, proxy_read_timeout) — https://nginx.org/en/docs/http/ngx_http_proxy_module.html
• nginx documentation: ngx_http_gzip_module — https://nginx.org/en/docs/http/ngx_http_gzip_module.html
• HTML Living Standard: Server-sent events — https://html.spec.whatwg.org/multipage/server-sent-events.html
• MDN: Using server-sent events — https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
• FastAPI documentation: Custom Response, StreamingResponse — https://fastapi.tiangolo.com/advanced/custom-response/