FastAPI freezes all requests: a blocking call inside async def stalls the event loop

Our quote service is a FastAPI app that prices shipments by calling three carrier APIs. On a busy Monday one carrier's API slowed from around 300 ms to between 4 and 9 seconds. We expected the quote endpoint to get slower. We did not expect every endpoint on the service to stop responding, including /healthz, which only returns a constant.

Kubernetes started failing liveness probes and restarting pods, which made things worse:

Warning  Unhealthy  pod/quotes-7c9f8d6b5-x2lqp  Liveness probe failed: Get "http://10.42.3.17:8000/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal   Killing    pod/quotes-7c9f8d6b5-x2lqp  Container quotes failed liveness probe, will be restarted

Our latency dashboard showed /healthz p99 jumping from 2 ms to 8.7 seconds at the same moment the carrier slowed down.

Reproducing it

We pointed a staging instance at a fake carrier that sleeps five seconds before answering, then ran two commands at once:

curl -s -o /dev/null -w '%{time_total}\n' 'http://localhost:8000/v1/quotes?from=HAN&to=SGN&kg=3' &
sleep 0.2
curl -s -o /dev/null -w '%{time_total}\n' http://localhost:8000/healthz

5.021
4.833

The health check, which does nothing, waited for the slow quote to finish. With a single Uvicorn worker, every request in flight queued behind the one waiting on the carrier.

What we ruled out

• CPU saturation. Pods were at 15% CPU during the incident. A blocked event loop does not burn CPU; it just waits.
• Connection pool exhaustion in the database driver. /healthz does not touch the database, and asyncpg pool metrics showed idle connections throughout.
• Uvicorn worker count. We run two workers per pod. More workers would raise the threshold, not remove it: two slow quotes are enough to freeze both.

Root cause

The carrier client for that one provider was written with requests and called directly from an async def endpoint. Its retry helper also used time.sleep() for backoff. Both are synchronous calls that block the thread they run on. In an async def FastAPI endpoint, that thread is the one running the asyncio event loop, so while it waited on the socket, no other coroutine in the process could run: no other requests, no health checks, not even the timeouts we had configured with asyncio.wait_for. Running the service with asyncio debug mode made it obvious:

WARNING:asyncio:Executing <Task pending name='Task-38' coro=<RequestResponseCycle.run_asgi() running at /usr/local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py:...> cb=[set.discard()]> took 5.017 seconds

What is happening under the hood

asyncio is cooperative multitasking on a single thread. A coroutine runs until it reaches an await on something that is not ready, such as a socket read through an async library. Only then can the event loop switch to another task. A call like requests.get() or time.sleep() never yields: it holds the thread inside a blocking system call. From the event loop's point of view, one callback simply took five seconds to return.

FastAPI makes this easy to get wrong because it supports both styles. A path operation declared with plain def is run in a worker thread pool, so blocking inside it is fine. A path operation declared with async def runs directly on the event loop, so it must only await non-blocking code. Our endpoint had been converted from def to async def to call the async database pool, and the sync carrier call came along for the ride.

This is also why asyncio.wait_for(..., timeout=2) did not save us: the timeout is implemented by the same event loop, which cannot fire it while the thread is blocked.

Environment

Minimal reproduction

# app.py
import time
import requests
from fastapi import FastAPI

app = FastAPI()

@app.get("/healthz")
async def healthz():
    return {"ok": True}

@app.get("/slow")
async def slow():
    time.sleep(5)                                   # blocks the event loop
    requests.get("https://example.com/", timeout=10)  # so does this
    return {"done": True}

uvicorn app:app --workers 1

Call /slow, then /healthz from another terminal: the health check waits for /slow.

Who is affected

Any asyncio-based server: FastAPI, Starlette, aiohttp, Sanic, Quart, Django async views. Common blockers are requests, urllib, time.sleep, synchronous database drivers such as psycopg2 and pymysql, boto3, file I/O on slow volumes, heavy JSON or image processing, and SDKs that are synchronous under the hood.

The fix

We moved the carrier client to httpx.AsyncClient, which awaits network I/O, and replaced the sleeping retry with asyncio.sleep. The client is created once at startup and shared, so connections are pooled:

-import requests
-import time
+import asyncio
+import httpx

 class CarrierBClient:
-    def quote(self, payload: dict) -> dict:
+    def __init__(self, client: httpx.AsyncClient):
+        self._client = client
+
+    async def quote(self, payload: dict) -> dict:
         for attempt in range(3):
-            r = requests.post(URL, json=payload, timeout=10)
+            r = await self._client.post(URL, json=payload, timeout=httpx.Timeout(3.0, connect=1.0))
             if r.status_code < 500:
                 r.raise_for_status()
                 return r.json()
-            time.sleep(0.5 * 2 ** attempt)
+            await asyncio.sleep(0.5 * 2 ** attempt)
         r.raise_for_status()

from contextlib import asynccontextmanager

@asynccontextmanager
async def lifespan(app: FastAPI):
    async with httpx.AsyncClient(limits=httpx.Limits(max_connections=100)) as client:
        app.state.carrier_b = CarrierBClient(client)
        yield

app = FastAPI(lifespan=lifespan)

Because the call now awaits properly, the timeout works too. We lowered it from 10 seconds to 3 and return a quote without that carrier when it times out, which is a product decision the team had wanted anyway.

Alternatives and trade-offs

1. asyncio.to_thread() for code you cannot rewrite, such as a vendor SDK that only exists in a sync version. It runs the call in the default thread pool executor and frees the event loop: result = await asyncio.to_thread(sdk.get_rates, payload). Threads are a limited resource, so bound the concurrency with a semaphore if the call can be slow.
2. Declare the endpoint with plain def. FastAPI then runs it in a thread pool. This works when the endpoint has no async dependencies. The pool has a limited number of threads (AnyIO's default is 40), so many slow calls can still queue up, just without freezing the whole process.
3. Starlette's run_in_threadpool. Equivalent to option 1 but uses the same AnyIO pool that FastAPI uses for def endpoints.

How to test the fix

We added a test that runs a slow quote and a health check concurrently against the ASGI app and asserts the health check is not delayed. The fake carrier is an httpx.MockTransport whose handler awaits asyncio.sleep:

import asyncio, time
import httpx
import pytest
from app.main import app

@pytest.mark.anyio
async def test_slow_carrier_does_not_block_healthz(slow_carrier):
    transport = httpx.ASGITransport(app=app)
    async with httpx.AsyncClient(transport=transport, base_url="http://test") as c:
        quote = asyncio.create_task(c.get("/v1/quotes", params={"from": "HAN", "to": "SGN", "kg": 3}))
        await asyncio.sleep(0.05)
        start = time.perf_counter()
        r = await c.get("/healthz")
        assert r.status_code == 200
        assert time.perf_counter() - start < 0.2
        await quote

We also run the whole suite with asyncio debug mode enabled, so that a fixture (described below) can fail any test that triggers a slow-callback warning:

PYTHONASYNCIODEBUG=1 pytest tests/

Confirming in production

We added an event-loop lag gauge: a background task that sleeps 100 ms in a loop and records how late it wakes up. Before the fix, lag spiked to several seconds whenever carrier B was slow. After the deploy, the gauge stayed under 15 ms at p99 for a full week, including a day when carrier B was again answering in 6 to 8 seconds. On that day /healthz p99 stayed at 3 ms and there were zero liveness restarts, against 41 restarts on the original incident day.

async def loop_lag_monitor(interval: float = 0.1):
    loop = asyncio.get_running_loop()
    while True:
        start = loop.time()
        await asyncio.sleep(interval)
        LOOP_LAG.observe(loop.time() - start - interval)

Regression guard

• The concurrency test above runs in CI.
• A test fixture enables loop.set_debug(True) with loop.slow_callback_duration = 0.25 and fails the test if the asyncio logger emits an "Executing ... took" warning.
• Ruff's ASYNC rules are enabled; for example ASYNC210 flags blocking HTTP calls inside async functions and ASYNC251 flags time.sleep in async functions.
• An alert fires when event-loop lag p99 exceeds 250 ms for five minutes.

Security implications

The bug turned a slow third party into a full outage, which is a denial-of-service vector: anyone who can trigger the slow path repeatedly, or a slow or malicious upstream, can freeze the service. Proper async I/O, strict timeouts on every outbound call, and a cap on concurrent calls per upstream close that path. We also separated the liveness probe from upstream health so that a slow dependency can no longer trigger restart storms.

Prevention checklist

• Never call blocking libraries (requests, time.sleep, sync DB drivers) inside async def.
• Use def endpoints or asyncio.to_thread for code that must stay synchronous.
• Set explicit timeouts on every outbound HTTP call.
• Export event-loop lag as a metric and alert on it.
• Enable Ruff ASYNC rules and run tests with asyncio debug mode.

References

• Python docs, Developing with asyncio (debug mode, blocking code) — https://docs.python.org/3/library/asyncio-dev.html
• Python docs, asyncio.to_thread — https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread
• FastAPI documentation, Concurrency and async / await — https://fastapi.tiangolo.com/async/
• HTTPX documentation, Async Support — https://www.python-httpx.org/async/
• Ruff rules, flake8-async (ASYNC) — https://docs.astral.sh/ruff/rules/#flake8-async-async