Performance & Scaling
Bug Report
huggingface/tokenizers: the current process just got forked warning in Gunicorn
Our embedding API logged a tokenizers fork warning thousands of times a day and tokenisation got slower after we enabled Gunicorn preload. The tokenizer was used in the master process before workers forked. Loading it after fork fixed both.