Embedding backfill crashes: input exceeds the model's maximum context length

We re-embed changed documents for our internal search every night. For three nights in a row, the job failed at around 62 percent and left the index partially updated. The worker log showed:

openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length
is 8192 tokens, however you requested 23514 tokens (23514 in your prompt; 0 for the completion).
Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None,
'code': None}}

The job sent batches of 100 chunks. One bad chunk failed the whole batch, the retry failed the same way, and after three retries the worker gave up on the entire run.

Reproducing it

We logged the token count of each chunk in the failing batch:

chunk_id=wiki/infra/terraform-outputs.json#0   chars=7980  tokens=23514
chunk_id=wiki/infra/terraform-outputs.json#1   chars=7990  tokens=22871
chunk_id=wiki/runbooks/db-failover.md#3        chars=7995  tokens=1840

Our chunker capped chunks at 8,000 characters. For English prose that is about 1,800 tokens. For a minified JSON file full of ARNs, hashes and UUIDs, it was nearly 24,000.

What we ruled out

• A wrong model name or dimension. The rest of the batches embedded fine.
• Batch size limits. The total batch was under the per-request limits once the oversized chunks were removed.
• Encoding issues. The text was valid UTF-8.

Root cause

The chunker assumed roughly 4 characters per token. That ratio holds for English prose and fails badly for code, JSON, base64, logs with identifiers, and non-Latin scripts. A new team started publishing Terraform output dumps to the wiki, and those files produced chunks three times over the limit. Because one invalid input fails the whole request, and the job had no per-item error handling, one file stopped the whole backfill.

What is happening under the hood

Embedding models have a maximum input length in tokens per input (8,192 for the OpenAI embedding models we use). An embeddings request can include many inputs, with additional limits on the number of inputs and total tokens per request. If any single input is too long, the request fails; the API does not skip it.

Token density varies widely:

Environment

Minimal reproduction

def chunk(text: str, max_chars: int = 8000) -> list[str]:
    return [text[i:i + max_chars] for i in range(0, len(text), max_chars)]

chunks = chunk(open("terraform-outputs.json").read())
client.embeddings.create(model="text-embedding-3-small", input=chunks)   # 400

The fix

1. Chunk by tokens with the model's tokenizer. We keep structure-aware splitting (headings, paragraphs, code blocks) and enforce a token cap at the end.

import tiktoken

ENC = tiktoken.get_encoding("cl100k_base")
MAX_TOKENS = 800          # retrieval quality, well under the 8192 limit
HARD_LIMIT = 8000

def split_tokens(text: str, max_tokens: int = MAX_TOKENS, overlap: int = 80) -> list[str]:
    ids = ENC.encode(text)
    out, start = [], 0
    while start < len(ids):
        end = min(start + max_tokens, len(ids))
        out.append(ENC.decode(ids[start:end]))
        if end == len(ids):
            break
        start = end - overlap
    return out

2. Batch by tokens, not by count.

def batches(items: list[tuple[str, str]], max_batch_tokens: int = 250_000, max_items: int = 512):
    batch, tokens = [], 0
    for chunk_id, text in items:
        n = len(ENC.encode(text))
        if n > HARD_LIMIT:
            yield_dead_letter(chunk_id, n)          # should never happen after split
            continue
        if batch and (tokens + n > max_batch_tokens or len(batch) >= max_items):
            yield batch
            batch, tokens = [], 0
        batch.append((chunk_id, text))
        tokens += n
    if batch:
        yield batch

3. Per-item failure handling. If a batch still fails with a 400, we split it in half and retry each half, down to single items. A single item that fails goes to a dead-letter table with its id and token count, and the job continues.

4. Skip content that should not be embedded. Minified JSON and machine output have little search value. Files over 70 percent non-alphanumeric characters, or with a known generated-file header, are now indexed by title and path only.

Alternatives and trade-offs

• Truncate long inputs. Quick, and it silently drops content. Acceptable for titles, not for documents.
• Average embeddings of sub-chunks. Gives one vector per document but blurs meaning; re-normalise if you do it.
• Character limits tuned per file type. Better than one global ratio, still a guess. Counting tokens is cheap enough.

How to test the fix

def test_dense_json_is_split_under_limit():
    text = json.dumps({f"arn:aws:iam::123456789012:role/r{i}": uuid4().hex for i in range(20000)})
    parts = split_tokens(text)
    assert all(len(ENC.encode(p)) <= MAX_TOKENS for p in parts)

def test_bad_item_does_not_fail_batch(fake_client):
    fake_client.reject_ids({"x"})
    results = embed_all([("a", "ok"), ("x", "bad"), ("b", "ok")])
    assert set(results.ok) == {"a", "b"} and results.dead_letter == ["x"]

Confirming in production

• The next nightly run completed in 41 minutes with zero failed batches and 7 dead-lettered items, all generated files.
• Index freshness, measured as the share of documents embedded within 24 hours of change, went from 62 percent to 99.8 percent.
• Embedding spend dropped 9 percent, because dense generated files are no longer embedded in full.

Regression guard

• Metric and alert for dead-letter count above 50 per run.
• A guard in the chunker that raises if any chunk exceeds the hard limit, so a future change to splitting rules fails tests rather than production.
• Job success is a dashboard tile next to search quality metrics.

Security implications

The failing files were Terraform outputs containing internal ARNs and, in one case, a database endpoint with a password in the connection string. They should not have been in the wiki at all. We reported them to the owning team, rotated the credential, and added a secret scanner to the ingestion step so secrets are dropped before embedding. Embeddings can leak information about their source text, so the vector store is protected like the documents themselves.

Prevention checklist

• Chunk and batch by tokens using the model's tokenizer.
• Expect dense content: code, JSON, ids, non-Latin scripts.
• Isolate failures per item; never let one input stop a backfill.
• Decide which content is worth embedding.
• Scan ingested content for secrets.

References

• OpenAI embeddings guide — https://platform.openai.com/docs/guides/embeddings
• OpenAI embeddings API reference — https://platform.openai.com/docs/api-reference/embeddings
• tiktoken — https://github.com/openai/tiktoken