AI & Machine Learning
Bug Report
Embedding backfill crashes: input exceeds the model's maximum context length
Our nightly embedding backfill died on the same few documents every night. A handful of chunks were far over the embedding model's token limit because the chunker split by characters and some files were minified JSON. Token-aware chunking and batching fixed it.
