OpenAI 400 error: tool_calls must be followed by tool messages for each tool_call_id

A week after we moved our customer support assistant to a newer chat model, error rates on the conversation service went from almost nothing to about 8 percent of conversations. Every failure was the same 400 from the Chat Completions API, on the request that followed a tool call:

openai.BadRequestError: Error code: 400 - {'error': {'message': "An assistant message with
'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following
tool_call_ids did not have response messages: call_7Qy2pXcV0aLm3RfT9bWkN1sE",
'type': 'invalid_request_error', 'param': 'messages.[6].role', 'code': None}}

The conversation then stopped responding, and the user saw our generic "something went wrong" message.

Reproducing it

The failing conversations had one thing in common: the user asked two things at once. "Where is order 4471 and can you also check my refund for 4390?" reproduced it on the first try. A single question never did.

We logged the raw assistant message before our loop processed it:

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {"id": "call_Ab3kP0qLm8XzY1vT", "type": "function",
     "function": {"name": "get_order_status", "arguments": "{\"order_id\":\"4471\"}"}},
    {"id": "call_7Qy2pXcV0aLm3RfT9bWkN1sE", "type": "function",
     "function": {"name": "get_refund_status", "arguments": "{\"order_id\":\"4390\"}"}}
  ]
}

Two tool calls in one assistant turn. Our code executed only tool_calls[0].

What we ruled out

• A malformed tool schema. The schemas validated, and the error points at the message list, not at tools.
• Message ordering by our database. We store messages with a sequence number; replaying the stored list gave the same order the API received.
• A model regression. The older model also supports parallel tool calls; it simply chose to use them less often on our prompts. The new model used them whenever the user asked about more than one order.

Root cause

The agent loop was written when we only ever saw one tool call per turn. It read message.tool_calls[0], ran that function, appended one role: tool message and called the API again. When the assistant message contained two calls, the second tool_call_id had no matching tool message, and the API rejects the whole request in that state.

What is happening under the hood

With function calling, the model can return more than one tool call in a single assistant message. This is called parallel function calling and it is on by default for models that support it. The conversation contract is strict: after an assistant message that contains tool_calls, the next messages must be role: "tool" messages, one per tool_call_id, before any new user or assistant message. The API checks that every id has a response and fails fast if any is missing.

The order of the tool messages among themselves is not important for validation, but keeping them in the same order as the calls makes logs easier to read and keeps behaviour predictable.

Environment

Minimal reproduction

# broken: only the first tool call is answered
msg = resp.choices[0].message
messages.append(msg)

if msg.tool_calls:
    call = msg.tool_calls[0]
    result = TOOLS[call.function.name](**json.loads(call.function.arguments))
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": json.dumps(result),
    })
    resp = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOL_SCHEMAS)

Any prompt that asks the model to do two independent lookups will trigger the 400 on the second request.

Who is affected

Any hand-written agent loop that assumes one tool call per turn. Frameworks that implement the loop for you usually handle this already. Code that stores the assistant message but filters tool_calls when saving can also hit it later when the history is replayed.

The fix

Iterate over every tool call, append a tool message for each one, and only then call the model again. We also made tool failures return a structured error instead of raising, so a failing tool still produces a tool message.

import json

def run_tool(call) -> str:
    fn = TOOLS.get(call.function.name)
    if fn is None:
        return json.dumps({"error": f"unknown tool {call.function.name}"})
    try:
        args = json.loads(call.function.arguments or "{}")
        return json.dumps(fn(**args))
    except Exception as exc:  # report the failure to the model, keep the contract
        return json.dumps({"error": type(exc).__name__, "detail": str(exc)[:500]})


def agent_turn(messages: list[dict], max_steps: int = 8):
    for _ in range(max_steps):
        resp = client.chat.completions.create(
            model=MODEL, messages=messages, tools=TOOL_SCHEMAS,
        )
        msg = resp.choices[0].message
        messages.append(msg.model_dump(exclude_none=True))

        if not msg.tool_calls:
            return msg.content

        for call in msg.tool_calls:          # answer every call, in order
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": run_tool(call),
            })
    raise RuntimeError("agent exceeded max_steps")

The independent lookups are good candidates for concurrency. We later ran them with a thread pool and then appended results in the original order:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(run_tool, msg.tool_calls))
for call, content in zip(msg.tool_calls, results):
    messages.append({"role": "tool", "tool_call_id": call.id, "content": content})

Alternatives and trade-offs

• Disable parallel calls with parallel_tool_calls=False. This works and is right for tools with side effects that must run in sequence, such as "create refund" followed by "send email". It also means more round trips and higher latency for read-only lookups.
• Use a framework's agent runner. Fine if you already use one, but we wanted to keep control over retries, timeouts and logging.

We kept parallel calls on for read-only tools and set parallel_tool_calls=False on the request when the tool set includes write operations.

How to test the fix

We added a unit test with a fake client that returns two tool calls, and asserted the message list sent on the next request:

def test_answers_every_parallel_tool_call(fake_client):
    fake_client.queue_tool_calls([
        ("call_a", "get_order_status", {"order_id": "1"}),
        ("call_b", "get_refund_status", {"order_id": "2"}),
    ])
    fake_client.queue_text("Order 1 shipped, refund 2 pending.")

    agent_turn([{"role": "user", "content": "status of 1 and refund of 2?"}])

    sent = fake_client.requests[1]["messages"]
    tool_ids = [m["tool_call_id"] for m in sent if m["role"] == "tool"]
    assert tool_ids == ["call_a", "call_b"]

Then we replayed 200 failed conversations from logs against staging. All completed.

Confirming in production

• The invalid_request_error count on the conversation service dropped to zero within an hour of rollout and stayed there for the following two weeks.
• Median latency for multi-question conversations fell by about 40 percent once the lookups ran concurrently.
• We added a log field tool_calls_per_turn; about 11 percent of turns now contain two or more calls, which explains the 8 percent failure rate we saw.

Regression guard

• The unit test above runs in CI.
• An alert on any 400 from the LLM provider with invalid_request_error, because these always mean a bug in our code, never a transient issue.
• A history validator runs before every request and checks that every assistant tool_calls id has a matching tool message. It raises in tests and logs loudly in production.

Security implications

Tool results are untrusted input to the model. Returning raw exception text could leak internal details such as SQL or file paths into the conversation, which the model might repeat to the user. We truncate error detail and map known exceptions to short safe messages. Running tools concurrently also means a single user message can trigger several backend calls at once, so per-user rate limits must count tool executions, not just chat requests.

Prevention checklist

• Treat tool_calls as a list everywhere: execution, storage and replay.
• Always return a tool message, even when the tool fails.
• Decide explicitly whether parallel calls are allowed for each tool set.
• Validate the message history shape before sending it.
• Test with prompts that ask for several things at once.

References

• OpenAI function calling guide — https://platform.openai.com/docs/guides/function-calling
• Chat Completions API reference — https://platform.openai.com/docs/api-reference/chat/create