Streaming LLM tool calls: Unexpected end of JSON input from partial arguments

Our web assistant streams answers to the browser through a Node.js backend. When we enabled streaming for requests that can also call tools, two kinds of failures appeared in Sentry within an hour:

SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at handleChunk (/app/dist/llm/stream.js:58:27)

and a quieter, worse one: a "create calendar event" tool ran with the arguments that belonged to a "lookup contact" call in the same turn, creating an event titled with a contact's email address.

Reproducing it

A prompt that triggers two tool calls at once reproduced both: "Find Lan's email and book 30 minutes with her tomorrow at 10." Logging raw stream chunks showed what we were receiving:

{"delta":{"tool_calls":[{"index":0,"id":"call_P1","type":"function","function":{"name":"lookup_contact","arguments":""}}]}}
{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"na"}}]}}
{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"me\": \"Lan\"}"}}]}}
{"delta":{"tool_calls":[{"index":1,"id":"call_P2","type":"function","function":{"name":"create_event","arguments":""}}]}}
{"delta":{"tool_calls":[{"index":1,"function":{"arguments":"{\"title\": \"Sync\", "}}]}}

Arguments arrive as string fragments, split at arbitrary points, and later chunks for a call carry only its index, not its id or name.

What we ruled out

• A broken model output. Concatenating fragments per index gave valid JSON every time.
• Network truncation. Chunks were complete SSE events; the split was inside the JSON payload by design.
• The SDK. Its stream helper can assemble calls for you; we were not using it because we forward text deltas to the browser ourselves.

Root cause

Our handler tried to JSON.parse the arguments of each chunk as it arrived, and when that failed it buffered fragments into a single string for "the current tool call", tracked by order of arrival. With one call per turn, the buffer usually worked by accident. With two parallel calls, fragments from both were appended to whichever call we thought was current, so arguments were mixed or incomplete.

What is happening under the hood

In streaming mode, each chunk's delta contains only what is new. For tool calls:

• The first chunk for a call carries index, id, type and function.name, usually with empty arguments.
• Following chunks carry the same index and a fragment of function.arguments.
• The arguments are a JSON document split at token boundaries, not at JSON boundaries, so individual fragments are almost never valid JSON on their own.
• When the turn is done, the choice ends with finish_reason: "tool_calls".

The index is the stable key for the duration of the stream. The id is what you must echo back in the tool result message.

Environment

Minimal reproduction

let current = { name: "", args: "" };

for await (const chunk of stream) {
  const tc = chunk.choices[0]?.delta?.tool_calls?.[0];   // only first element
  if (!tc) continue;
  if (tc.function?.name) current = { name: tc.function.name, args: "" };
  current.args += tc.function?.arguments ?? "";
  try {
    run(current.name, JSON.parse(current.args));          // parsed too early
  } catch { /* wait for more */ }
}

The fix

Accumulate every tool call by index, stream text deltas to the browser as before, and parse arguments only once the stream finishes with finish_reason === "tool_calls".

type PendingCall = { id: string; name: string; args: string };

export async function consume(stream: AsyncIterable<ChatCompletionChunk>, onText: (t: string) => void) {
  const calls = new Map<number, PendingCall>();
  let finish: string | null = null;

  for await (const chunk of stream) {
    const choice = chunk.choices[0];
    if (!choice) continue;

    if (choice.delta.content) onText(choice.delta.content);

    for (const tc of choice.delta.tool_calls ?? []) {
      const call = calls.get(tc.index) ?? { id: "", name: "", args: "" };
      if (tc.id) call.id = tc.id;
      if (tc.function?.name) call.name = tc.function.name;
      if (tc.function?.arguments) call.args += tc.function.arguments;
      calls.set(tc.index, call);
    }
    if (choice.finish_reason) finish = choice.finish_reason;
  }

  if (finish !== "tool_calls") return { toolCalls: [] as ParsedCall[], finish };

  const toolCalls = [...calls.entries()]
    .sort(([a], [b]) => a - b)
    .map(([, c]) => ({ id: c.id, name: c.name, args: parseArgs(c) }));
  return { toolCalls, finish };
}

function parseArgs(c: PendingCall): unknown {
  try {
    return JSON.parse(c.args || "{}");
  } catch (err) {
    throw new ToolArgumentError(c.name, c.id, c.args.length);
  }
}

Arguments are then validated against the tool's schema (we use zod) before anything runs. If validation fails, we send a tool result with the validation error back to the model instead of executing.

When finish_reason is "length", the arguments may be cut off; we treat that as an error and do not execute partial calls.

Alternatives and trade-offs

• Use the SDK's stream helper that assembles the final message for you. Less code; we needed raw deltas for our own SSE protocol, but the helper is a good choice if you do not.
• Incremental JSON parsers let you show arguments while they stream, for example a live preview of an email being drafted. Never execute based on partial parses.
• Disable parallel tool calls for tools with side effects, so calls happen one per turn.

How to test the fix

We recorded the raw chunk sequence from the failing prompt and replayed it in a unit test, plus a fuzz test that re-splits the same arguments at random points and interleaves two calls:

test("assembles interleaved parallel calls", async () => {
  const chunks = splitRandomly(recordedTwoCallStream, { seed: 42 });
  const { toolCalls } = await consume(fromArray(chunks), () => {});
  expect(toolCalls.map(c => c.name)).toEqual(["lookup_contact", "create_event"]);
  expect(toolCalls[1].args).toMatchObject({ title: "Sync" });
});

Confirming in production

• JSON parse errors from the stream handler dropped to zero after deploy.
• We log a hash of each call's arguments at parse time and at execution time; they matched for 100 percent of calls over two weeks.
• No more mismatched calendar events. The few bad events created during the bug window were found by matching titles that looked like email addresses and were cleaned up with the affected users.

Regression guard

• The replay and fuzz tests run in CI.
• Schema validation of arguments before execution is mandatory in the tool registry.
• An alert on ToolArgumentError counts.

Security implications

Mixing arguments between calls is an integrity problem: an action ran with data the user never intended for it. Two controls now sit between the model and side effects: schema validation, and a confirmation step for write tools such as creating events or sending messages. Tool arguments are model output and must be treated as untrusted input.

Prevention checklist

• Key streaming tool calls by index, keep id for the response.
• Parse arguments only after the call is complete.
• Never execute on finish_reason: "length".
• Validate arguments with a schema before running a tool.
• Test with interleaved parallel calls and random fragment splits.

References

• OpenAI streaming responses — https://platform.openai.com/docs/api-reference/chat-streaming
• OpenAI function calling guide — https://platform.openai.com/docs/guides/function-calling
• MDN, Server-sent events — https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events