LLM maximum context length exceeded: trimming chat history by tokens, not messages

Our internal engineering assistant answers questions about runbooks and lets people paste logs for a first look. After a few weeks in use, the error dashboard showed a steady trickle of failures from long conversations:

openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context
length is 128000 tokens. However, your messages resulted in 131874 tokens (129912 in the messages,
1962 in the functions). Please reduce the length of the messages or functions.",
'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

The failing user saw "Assistant unavailable", and every message after that failed too, because the history only grew.

Reproducing it

We already trimmed history to the last 20 messages. That sounded generous for a 128k window. Reproduction took one step: paste a 9 MB application log into the chat, then ask a follow-up. The paste alone tokenised to about 2.4 million tokens, and our UI happily accepted it. A smaller variant was harder to spot: ten pastes of 15k tokens each also overflowed, while staying well under 20 messages.

What we ruled out

• A wrong model name. The error reported the right window size for the model we used.
• Tool schemas. They added about 2k tokens. Not great, but not the cause.
• A bug in the SDK. The SDK sends what you give it; the API counts the tokens.

Root cause

We trimmed by message count, but the model limit is in tokens, and message sizes vary by four orders of magnitude. We also ignored three other parts of the budget: the system prompt (3.1k tokens of instructions), tool definitions, and the tokens reserved for the response. Once a conversation contained one huge message, no amount of message-count trimming could bring it under the limit.

What is happening under the hood

The context window is shared by everything in the request: system prompt, every message, tool definitions, and the output the model is going to generate. If you set max_tokens (or the newer max_completion_tokens), that amount must also fit. Providers count with the model's own tokenizer, so a character limit is only a rough proxy. English prose is roughly four characters per token, but code, JSON, logs with UUIDs and non-Latin scripts can be much denser.

Each chat message also carries a small overhead for role and formatting tokens, a few tokens per message. It rarely matters, but it is the reason simple sums come out slightly low.

Environment

Minimal reproduction

MAX_MESSAGES = 20

def build_messages(system: str, history: list[dict], user_msg: str) -> list[dict]:
    recent = history[-MAX_MESSAGES:]           # count-based trimming
    return [{"role": "system", "content": system}, *recent,
            {"role": "user", "content": user_msg}]

One message of 200k tokens anywhere in the last 20 makes every subsequent request fail.

Who is affected

Chat products that let users paste or upload large text, agent loops that put whole tool results (web pages, SQL results) into history, and any code that trims by message count or characters.

The fix

We changed three things.

1. Count tokens with the right encoding and keep a budget.

import json
import tiktoken

ENC = tiktoken.get_encoding("o200k_base")
CONTEXT_WINDOW = 128_000
RESERVED_OUTPUT = 4_000
SAFETY_MARGIN = 1_000
PER_MESSAGE_OVERHEAD = 4

def count(msg: dict) -> int:
    content = msg.get("content") or ""
    if not isinstance(content, str):
        content = json.dumps(content)
    return len(ENC.encode(content)) + PER_MESSAGE_OVERHEAD

def fit_history(system: dict, history: list[dict], user: dict, tools_tokens: int) -> list[dict]:
    budget = CONTEXT_WINDOW - RESERVED_OUTPUT - SAFETY_MARGIN - tools_tokens
    budget -= count(system) + count(user)
    if budget <= 0:
        raise ValueError("system prompt and user message alone exceed the context budget")

    kept: list[dict] = []
    for msg in reversed(history):          # newest first
        cost = count(msg)
        if cost > budget:
            break
        kept.append(msg)
        budget -= cost
    return [system, *reversed(kept), user]

When trimming, we drop whole turns. If an assistant message with tool_calls is kept, its tool messages must be kept too, otherwise the API rejects the history. Our real code groups messages into turns before trimming.

2. Cap single inputs at the edge. The chat UI and the API now reject pastes over 60k tokens with a clear message and offer to upload the file instead. Uploaded logs go through a separate path that extracts the error lines and the surrounding context, rather than the full file.

3. Summarise what falls off. When older turns are dropped, a cheap model call summarises them into a short "conversation so far" note that is stored and placed after the system prompt. This keeps long sessions coherent without unbounded growth.

Alternatives and trade-offs

• Truncate the big message instead of dropping it. Keep the first and last N tokens of a log. Useful for logs, misleading for code.
• Retrieve instead of stuffing. Store long pastes as documents and retrieve relevant chunks per question. More moving parts, much better for large inputs.
• Use a bigger window model. Moves the limit, does not remove it, and long prompts are slower and more expensive on every turn.

How to test the fix

def test_huge_message_is_dropped_not_sent():
    big = {"role": "user", "content": "x " * 300_000}
    msgs = fit_history(SYSTEM, [big, *SMALL_TURNS], {"role": "user", "content": "hi"}, 2_000)
    assert big not in msgs
    assert sum(count(m) for m in msgs) <= CONTEXT_WINDOW - RESERVED_OUTPUT

def test_tool_turn_kept_whole(history_with_tool_calls):
    msgs = fit_history(SYSTEM, history_with_tool_calls, USER, 2_000)
    assert_every_tool_call_has_response(msgs)

Confirming in production

• context_length_exceeded errors went from about 40 a day to zero after deploy.
• We log prompt_tokens from the API usage field next to our own estimate. The difference stayed within 1 percent, which confirmed the encoding choice.
• Average prompt size per turn dropped by 35 percent because huge pastes no longer rode along in every request, which also cut cost.

Regression guard

• A metric for estimated prompt tokens per request with an alert at 90 percent of the window.
• Unit tests for the trimming function, including tool-call turns.
• The paste limit is enforced in the API, not only in the UI.

Security implications

Large pastes are where secrets end up: environment dumps, config files, tokens in logs. Before this fix every paste was resent on every turn and stored in our conversation logs. We now run a secret scanner on pastes before storage and redact matches. Limiting input size also removes an easy way to burn the team's LLM budget with one request.

Prevention checklist

• Budget in tokens: system + tools + history + reserved output + margin.
• Count with the model's tokenizer, and compare to the API's reported usage.
• Limit single input size at the API boundary.
• Trim whole turns, never split tool calls from their results.
• Summarise or retrieve instead of resending everything.

References

• OpenAI tokenizer library tiktoken — https://github.com/openai/tiktoken
• OpenAI text generation and token limits — https://platform.openai.com/docs/guides/text-generation
• Counting tokens cookbook — https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken