LLM Batch API job expired and our pipeline wrote results to the wrong rows

We generate and refresh product descriptions for a catalogue of about 60,000 items. The job runs nightly through the Batch API, which costs half the price of synchronous calls and is fine for work that can wait. One morning the merchandising team reported nonsense in the shop: a hiking backpack described as a coffee grinder, a pair of socks with the description of a tent.

The job had logged success:

batch_6f2c... status=expired  request_counts={total: 8214, completed: 5741, failed: 0}
import: wrote 5741 descriptions

status=expired, and we still wrote 5,741 descriptions, many to the wrong products.

Reproducing it

The importer code:

inputs = read_jsonl("batch_input.jsonl")                 # in submission order
outputs = read_jsonl(download(batch.output_file_id))
for item, result in zip(inputs, outputs):                # assumes same order and length
    save_description(item["product_id"], result["response"]["body"]["choices"][0]["message"]["content"])

With an expired batch, the output file had 5,741 lines, and they were not in input order. zip paired them with the first 5,741 inputs.

What we ruled out

• A model or prompt problem. Each generated description was coherent and matched some product, just not the one it was saved to.
• Our product id mapping. Product ids in the input file were correct.

Root cause

Two assumptions were wrong. First, we treated the output file as a list aligned with the input. The Batch API does not guarantee output order; every result carries the custom_id of its request, and that is the only correct join key. Second, the job only checked for failed. A batch that hits its completion window becomes expired; finished requests are in the output file, and unfinished ones are reported in the error file. Most nights the batch finished early, so order and completeness happened to look aligned often enough that nobody noticed. That night, provider load was high, and the batch expired at 70 percent done.

What is happening under the hood

A batch goes through statuses: validating, in_progress, finalizing, then a terminal status: completed, failed, expired or cancelled. Each request in the input file has a custom_id you choose, and each line in the output or error file includes it. Results come back in whatever order they finished.

When the completion window ends before all requests are done, the batch is expired. Completed requests are still billed and their results are available. Requests that did not run appear in the error file with an expiry error. The job must decide whether to resubmit them.

Environment

Minimal reproduction

# Simulate: outputs shuffled and incomplete
outputs = random.sample(all_outputs, k=int(len(all_outputs) * 0.7))
for item, result in zip(inputs, outputs):
    ...   # mismatched pairs

The fix

1. Stable, meaningful custom_id. It encodes the product id and the content version so results can be matched without any other state:

def build_request(product) -> dict:
    return {
        "custom_id": f"desc:{product.id}:{product.content_hash[:12]}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {"model": MODEL, "messages": prompt_for(product), "max_tokens": 400},
    }

2. Join by custom_id and handle every status.

TERMINAL = {"completed", "expired", "failed", "cancelled"}

def import_batch(batch) -> ImportReport:
    assert batch.status in TERMINAL
    report = ImportReport(status=batch.status)

    if batch.status == "failed":
        report.error = batch.errors
        return report

    if batch.output_file_id:
        for line in iter_jsonl(download(batch.output_file_id)):
            _, product_id, content_hash = line["custom_id"].split(":")
            resp = line["response"]
            if resp["status_code"] != 200:
                report.retry.append(line["custom_id"]); continue
            product = products.get(product_id)
            if product is None or product.content_hash[:12] != content_hash:
                report.stale.append(line["custom_id"]); continue   # product changed meanwhile
            save_description(product_id, resp["body"]["choices"][0]["message"]["content"])
            report.saved += 1

    if batch.error_file_id:
        for line in iter_jsonl(download(batch.error_file_id)):
            report.retry.append(line["custom_id"])
    return report

3. Resubmit what did not finish. Unfinished and retryable requests go into a new batch the next run, with the same custom_id scheme. Because writes are keyed by product id and content hash, re-processing is idempotent.

4. Validate before writing. A cheap check that the description mentions the product's category or name catches any future mismatch before it reaches the shop.

Alternatives and trade-offs

• Smaller batches. Several smaller batches finish more predictably than one large one, and a single expiry affects less work.
• Fall back to synchronous calls for the most important items after an expiry, at full price.
• Do not save until the whole batch completes. Simpler, but throws away paid-for results on expiry.

How to test the fix

Unit tests with recorded output and error files, shuffled and truncated:

def test_import_matches_by_custom_id(fake_files):
    out, err = fake_files(shuffled=True, completed_ratio=0.7)
    report = import_batch(FakeBatch(status="expired", output=out, error=err))
    for pid, text in saved_descriptions():
        assert expected_text_for(pid) == text
    assert len(report.retry) == int(TOTAL * 0.3)

Confirming in production

• We restored the 5,741 affected descriptions from the previous night's snapshot within two hours and re-ran those products.
• Since the fix, two batches have expired partially. Both imports saved only correctly matched results and queued the rest; the next night caught up.
• The validation check has blocked zero writes, which is what we expect when matching is correct.

Regression guard

• Import refuses any batch without a recognised terminal status handler.
• Metrics per run: saved, retry, stale, by status; alert if retry exceeds 20 percent two nights in a row.
• Product descriptions keep their previous version for one-click rollback.

Security implications

None in terms of access control. It was a data integrity incident in customer-facing content, with possible consumer protection implications if a wrong description had stated wrong specifications. Generated content now passes the validation check and a small daily human sample before publishing.

Prevention checklist

• Join batch results by custom_id, never by position.
• Handle expired, failed and cancelled, not just completed.
• Make writes idempotent and keyed by content version.
• Resubmit unfinished requests explicitly.
• Validate generated content against its target before saving.

References

• OpenAI Batch API guide — https://platform.openai.com/docs/guides/batch
• Batch object reference — https://platform.openai.com/docs/api-reference/batch/object