huggingface/tokenizers: the current process just got forked warning in Gunicorn

Our internal embedding API wraps a sentence-transformer model behind FastAPI, served by Gunicorn with Uvicorn workers. To cut memory and startup time, someone enabled --preload, so the model loads once in the master and is shared with workers through copy-on-write. Memory did drop. Two other things happened: the logs filled with this block, once per worker, on every restart and deploy,

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

and batch tokenisation for large requests got noticeably slower: p95 for 256-document batches went from 140 ms to 390 ms.

Reproducing it

gunicorn app:app -k uvicorn.workers.UvicornWorker -w 4 --preload

With --preload, warnings and slowdown. Without it, neither. The warning appeared only if the master had actually run the tokenizer, which it did: our startup code ran a warm-up model.encode(["warmup"]) so the first request would not be slow.

What we ruled out

• CPU throttling. The pod had spare CPU, and the slowdown was in tokenisation specifically, not in the model forward pass.
• A different tokenizers version. Same lockfile before and after.
• Thread oversubscription. We checked torch.get_num_threads(); it had not changed.

Root cause

The fast tokenizers library is written in Rust and uses a thread pool to tokenise batches in parallel. Threads do not survive fork(): the child process gets a copy of memory, including any locks held by threads in the parent, but only the forking thread. If the parent had already used the thread pool, a child could deadlock on a lock that nobody will ever release. The library detects this case and disables parallelism in the child, printing the warning. Our warm-up call used the pool in the master, so every forked worker ran tokenisation on a single thread.

What is happening under the hood

Gunicorn's --preload imports the application in the master process, then forks workers. Everything created during import is inherited. That is good for read-only memory like model weights, and risky for anything holding threads, sockets or locks: database connection pools, gRPC channels, CUDA contexts and Rust or C++ thread pools.

The tokenizers library tracks whether its parallelism has been used. On fork, if it has, it sets parallelism off in the child to be safe. Setting TOKENIZERS_PARALLELISM explicitly silences the warning, but true does not make the inherited pool safe; it only tells the library you take responsibility.

The same message is common with PyTorch DataLoader(num_workers>0), which forks worker processes after the tokenizer was used in the main process.

Environment

Minimal reproduction

# app.py, imported by the gunicorn master with --preload
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("intfloat/e5-base-v2")
model.encode(["warmup"])           # uses the tokenizer thread pool before fork

from fastapi import FastAPI
app = FastAPI()

@app.post("/embed")
def embed(texts: list[str]):
    return model.encode(texts, batch_size=64).tolist()

The fix

We kept --preload for the weights and moved anything that uses the tokenizer's thread pool to after the fork.

# app.py
from fastapi import FastAPI
from contextlib import asynccontextmanager
from sentence_transformers import SentenceTransformer

_model = SentenceTransformer("intfloat/e5-base-v2")   # load weights only, no encode()

@asynccontextmanager
async def lifespan(app: FastAPI):
    _model.encode(["warmup"])                           # runs inside each worker
    yield

app = FastAPI(lifespan=lifespan)

FastAPI's lifespan runs in each worker after it starts, so the warm-up now happens post-fork. Loading the model files does not use the parallel tokenisation path; only encoding does.

We also set the variable explicitly in the deployment, so a future change that reintroduces pre-fork usage fails loudly in review rather than silently slowing down:

env:
  - name: TOKENIZERS_PARALLELISM
    value: "true"

and a startup check in the lifespan handler that logs the effective parallelism setting per worker.

Alternatives and trade-offs

• Drop --preload. Each worker loads its own copy. Simple and safe; costs memory (about 450 MB per worker for us).
• One process, more threads. Run a single worker and rely on the tokenizer and torch thread pools. Fine for CPU-bound inference, but one crash takes down all capacity.
• TOKENIZERS_PARALLELISM=false. Silences the warning and makes behaviour explicit. Correct for DataLoader workers, where you already parallelise across processes; it does not bring back the batch speed for an API.
• Use spawn instead of fork for multiprocessing code you control. Spawned processes start clean, at the cost of slower startup.

How to test the fix

gunicorn app:app -k uvicorn.workers.UvicornWorker -w 4 --preload 2>&1 | grep -c "just got forked"
# 0

A small benchmark posted 256 documents 200 times; p95 returned to 135 ms. We also confirmed memory sharing still worked: total RSS across workers was within 5 percent of the broken preload setup.

Confirming in production

• The fork warning count in logs dropped to zero after deploy.
• p95 for large batches returned to its pre-preload level, and p50 was unchanged.
• Pod memory stayed at the lower preload level.

Regression guard

• A CI smoke test starts Gunicorn with production flags and fails if the fork warning appears in stderr.
• A code review note in the app module: no inference calls at import time.

Security implications

None directly. The general lesson does have a security edge: objects created before fork are shared, including random number generator state. Anything that generates tokens or nonces in workers must be seeded or created after fork. We checked; our app uses secrets, which reads from the OS and is safe.

Prevention checklist

• With --preload, load data, do not run work, before fork.
• Warm up models inside worker startup hooks.
• Set TOKENIZERS_PARALLELISM explicitly in every deployment.
• In DataLoader workers, set it to false and parallelise across processes.
• Benchmark after changing process models, not only memory.

References

• Hugging Face tokenizers — https://github.com/huggingface/tokenizers
• Gunicorn settings, preload_app — https://docs.gunicorn.org/en/stable/settings.html#preload-app
• FastAPI lifespan events — https://fastapi.tiangolo.com/advanced/events/
• Python multiprocessing start methods — https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods