LLM returns invalid JSON 2% of the time: code fences, truncation and structured outputs

Our accounts-payable product extracts invoice fields (supplier, invoice number, dates, line items, totals, currency) from emails and PDFs that customers forward to us. An LLM call turns the extracted text into a JSON object, and a worker writes it to the database. After a model upgrade and a prompt rewrite, the worker's error rate settled at around 2.1% of documents, all with the same exception:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

with a long tail of others:

json.decoder.JSONDecodeError: Unterminated string starting at: line 41 column 22 (char 1873)
json.decoder.JSONDecodeError: Extra data: line 24 column 1 (char 811)

Failed documents went to a manual review queue, which at our volume meant roughly 700 invoices a day for the operations team to key in by hand.

Reproducing it

The failures were not deterministic: re-running a failed document succeeded about 60% of the time. To study them we stored the raw model output for every failure for a week (in our own encrypted bucket, with a seven-day lifecycle rule) and classified 1,000 of them:

The truncated ones were reproducible every time: any invoice with enough line items produced a cut-off object.

What we ruled out

• Encoding problems in our pipeline. Raw bytes from the API matched what we parsed; nothing was being mangled between the SDK and json.loads.
• Temperature. We already used 0. Lowering it does not guarantee format compliance, and the fence and preamble failures still appeared.
• The prompt alone. We had "Respond ONLY with valid JSON" in capital letters in two places. Instructions shift probabilities; they do not enforce a grammar.

Root cause

We were relying on instructions plus JSON mode, which only guarantees syntactically valid JSON when it is enabled, and our new prompt template had silently dropped the response_format parameter during the rewrite. Without it, nothing constrained the output format at all. Separately, we never checked why generation stopped. A response with finish_reason: "length" is by definition incomplete, and we were feeding those straight to the parser.

What is happening under the hood

A language model produces tokens one at a time. Unless the provider constrains decoding to a grammar, the model can emit anything, and fenced code blocks are an extremely common pattern in its training data for "here is some JSON". Providers now offer three levels of control:

Even with constraints, two failure paths remain. If generation hits the output token limit, the object is cut off wherever it was; the response reports this as finish_reason: "length" in Chat Completions (other APIs have an equivalent stop reason, for example max_tokens on Anthropic's Messages API). And a model may decline to answer, which the OpenAI API reports in a separate refusal field rather than as schema-shaped content.

Environment

Minimal reproduction

# Excerpt: the failing call path.
resp = client.chat.completions.create(
    model="gpt-4.1-mini",
    temperature=0,
    max_tokens=2048,
    messages=[
        {"role": "system", "content": "Extract invoice fields. Respond ONLY with valid JSON."},
        {"role": "user", "content": invoice_text},
    ],
)
data = json.loads(resp.choices[0].message.content)  # fails ~2% of the time

Who is affected

Anyone parsing free-form model output as JSON, especially for long or variable-length objects, after prompt or model changes, or across providers that format output differently. Agents parsing tool arguments by hand have the same problem.

The fix

1. Define the schema once, in code. We already had a Pydantic model for the database layer, so we derived the extraction schema from a dedicated model rather than describing fields in prose:

from pydantic import BaseModel, ConfigDict

class LineItem(BaseModel):
    model_config = ConfigDict(extra="forbid")
    description: str
    quantity: float
    unit_price: str  # decimal as string to avoid float rounding
    amount: str

class Invoice(BaseModel):
    model_config = ConfigDict(extra="forbid")
    supplier_name: str
    invoice_number: str
    issue_date: str  # ISO 8601, validated below
    due_date: str | None
    currency: str
    total: str
    line_items: list[LineItem]

2. Ask for schema-constrained output. Structured Outputs requires every property to be listed as required and additionalProperties to be false; optional values are expressed as a union with null, which is what str | None produces. We generate the JSON Schema from the model and pass it with strict: true:

schema = Invoice.model_json_schema()
resp = client.chat.completions.create(
    model="gpt-4.1-mini",
    temperature=0,
    max_tokens=6000,
    messages=messages,
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "invoice", "schema": schema, "strict": True},
    },
)

None of the fields have defaults, so Pydantic lists all of them as required, and extra="forbid" emits additionalProperties: false. A unit test checks both properties of the generated schema, because adding a default to a field later would silently break strict mode.

3. Check why generation stopped before parsing.

choice = resp.choices[0]
if choice.message.refusal:
    raise ExtractionRefused(choice.message.refusal)
if choice.finish_reason == "length":
    raise ExtractionTruncated(resp.usage.completion_tokens)
invoice = Invoice.model_validate_json(choice.message.content)

Truncated invoices are retried once with a higher output limit. If they still truncate (a handful of 300-line invoices a week), we split extraction into header fields and line items in pages of 50.

4. Validate semantics, then repair once. A schema guarantees shape, not truth. After parsing we check that dates parse as ISO 8601, the currency is a valid ISO 4217 code and line item amounts sum to the total within one minor unit. On a validation failure we make exactly one repair call that includes the validation errors, and if that fails the document goes to the review queue with the errors attached.

Alternatives and trade-offs

• Strip code fences and extract the first {...} with a regex. It rescues the fence and preamble cases cheaply and we keep it as a last-resort fallback for older code paths, but it cannot fix truncation or wrong fields, and it hides regressions.
• Forced tool calling. Defining a tool whose input schema is your object and requiring the model to call it works on providers without a dedicated structured output mode. You still need to check the stop reason and validate the arguments.
• JSON mode only. Better than nothing and supported on more models, but it does not enforce required fields or types.

How to test the fix

We built a regression set of 400 documents: the 1,000 classified failures deduplicated, plus long invoices and multi-currency edge cases. A test runs the extractor against recorded responses in CI and against the live API nightly:

pytest tests/extraction -m "not live"
pytest tests/extraction -m live --maxfail=20

The nightly live run reports parse failures, truncations, validation failures and field-level accuracy against hand-labelled answers.

Confirming in production

• Parse failures dropped from 2.1% to 0.00% over two weeks; the parser has not raised JSONDecodeError since the deploy.
• Truncation, now detected explicitly, occurs on 0.06% of documents, and 90% of those succeed on the retry with a higher limit.
• Semantic validation catches about 0.4% of documents (mostly totals that do not add up on scanned invoices), and the repair call fixes about half of those. The manual review queue went from about 700 to about 70 documents a day.
• Output tokens per call fell by about 6%, since the model no longer writes preambles.

Regression guard

• A test asserts every extraction call passes response_format with strict: True; it exists because the original regression was a template rewrite dropping one parameter.
• Dashboards track parse failures, truncations, refusals and validation failures separately. Each has its own alert, because they have different fixes.
• Model or prompt changes must pass the 400-document regression set before rollout.

Security implications

The documents are untrusted input. An invoice can contain text that tries to change the model's behaviour, so constrained output also narrows what a manipulated response can look like. We never eval or execute anything from model output, field values are treated as untrusted strings with length limits, and amounts are parsed as decimals with bounds. Storing failed raw outputs for analysis exposed invoice data in a new place, so that bucket is encrypted, access-logged and set to delete objects after seven days.

Prevention checklist

• Use schema-constrained output where the provider supports it; otherwise forced tool calls plus validation.
• Always check the stop reason and refusal fields before parsing.
• Size output limits from measured worst cases and split large objects.
• Validate business rules after schema validation, with at most one repair attempt.
• Keep a regression set of real failures and run it on every prompt or model change.

References

• OpenAI API documentation: Structured Outputs — https://platform.openai.com/docs/guides/structured-outputs
• Anthropic documentation: Tool use with Claude — https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/overview
• Pydantic documentation: JSON Schema — https://docs.pydantic.dev/latest/concepts/json_schema/
• JSON Schema: Understanding JSON Schema — https://json-schema.org/understanding-json-schema/