LLM agent stuck in a tool-calling loop overnight: 400 searches and a surprise bill

Our market research agent takes a company name, searches the web and our internal CRM, and writes a one-page brief. It runs as a background job. One Monday the LLM provider's usage dashboard showed a spike on Sunday night larger than the previous week combined. One job had been running for seven hours.

The job log was the same lines repeated:

02:14:07 step=388 tool=search_crm args={"company": "Northwind Traders"} result_len=0
02:14:11 step=389 tool=search_crm args={"company": "Northwind Traders"} result_len=0
02:14:16 step=390 tool=search_crm args={"company": "Northwind Traders"} result_len=0

412 calls to search_crm with identical arguments, each one sending the full, growing conversation back to the model.

Reproducing it

We pointed a local run at the same company and made the CRM API return a 503. The agent called search_crm, received an empty string, and the model said something like "The search returned no results; let me try again." Then it tried again, forever.

What we ruled out

• A model regression. Older traces with the same model showed the agent giving up after two empty searches when the CRM returned a clear "no match" message.
• A retry wrapper in our HTTP client. It retried twice at the HTTP layer and then raised, which our tool code caught.
• A prompt change. The system prompt had not changed in a month.

Root cause

Three things combined. First, the tool caught every exception and returned "". To the model, an empty result looks like "try again, maybe with the same query". Second, the agent loop was while True with no maximum number of steps; it stopped only when the model produced a final answer. Third, nothing tracked tokens or money per run, so a single run had no ceiling. The CRM had a partial outage from 19:00 to 03:00, and every research job started in that window looped. Most hit the job runner's 2-hour timeout; one had been started from a different queue without a timeout.

What is happening under the hood

An agent loop sends the conversation to the model, executes the tools it asks for, appends the results and repeats. Every iteration resends the full history, so cost per step grows linearly with the number of steps and total cost grows roughly quadratically. A loop of 400 steps with a few thousand tokens added per step ends with requests close to the context limit, each one billed in full.

Models choose the next action based on what they see. An empty tool result carries no information, so retrying is a reasonable guess. A clear error such as "CRM unavailable (503), do not retry; continue without CRM data" gives the model a way out.

Environment

Minimal reproduction

def search_crm(company: str) -> str:
    try:
        return json.dumps(crm.search(company))
    except Exception:
        return ""                                  # silent failure

def run_agent(task: str) -> str:
    messages = [SYSTEM, {"role": "user", "content": task}]
    while True:                                    # no step limit, no budget
        reply = llm(messages, tools=TOOLS)
        if not reply.tool_calls:
            return reply.content
        messages += execute(reply.tool_calls)

The fix

1. Tools return explicit, actionable errors.

class ToolError(Exception):
    def __init__(self, message: str, retryable: bool):
        super().__init__(message)
        self.retryable = retryable

def search_crm(company: str) -> str:
    try:
        rows = crm.search(company)
    except CRMUnavailable as exc:
        return json.dumps({"error": "crm_unavailable", "retryable": False,
                           "hint": "Continue without CRM data and say so in the brief."})
    if not rows:
        return json.dumps({"results": [], "note": f"No CRM records for {company!r}."})
    return json.dumps({"results": rows[:20]})

2. Hard limits in the loop: steps, tokens, money and wall time.

from dataclasses import dataclass, field
import time

@dataclass
class Budget:
    max_steps: int = 25
    max_tokens: int = 400_000
    max_seconds: int = 600
    steps: int = 0
    tokens: int = 0
    started: float = field(default_factory=time.monotonic)

    def charge(self, usage) -> None:
        self.steps += 1
        self.tokens += usage.input_tokens + usage.output_tokens
        if self.steps > self.max_steps:
            raise BudgetExceeded("steps")
        if self.tokens > self.max_tokens:
            raise BudgetExceeded("tokens")
        if time.monotonic() - self.started > self.max_seconds:
            raise BudgetExceeded("time")

3. Loop detection. If the same tool is called with the same arguments three times in a run, we stop executing it and return a tool result telling the model the call is blocked for this run:

key = (call.name, canonical_json(call.arguments))
seen[key] = seen.get(key, 0) + 1
if seen[key] >= 3:
    result = json.dumps({"error": "repeated_call_blocked",
                         "hint": "This exact call already ran twice. Use what you have."})

4. Graceful finish on budget. When a budget is exceeded, we make one last call without tools, asking the model to write the brief with what it has and to list what is missing. The user gets a partial result instead of nothing.

Alternatives and trade-offs

• Only a step limit. Simple and catches loops, but a few steps with huge tool outputs can still be expensive. Token budgets cover that.
• Provider-side spend limits. Useful as a last line of defence for the whole account; too coarse to protect one feature from another.
• Circuit breaker on the CRM client. We added one as well, so that during an outage the tool fails fast for every run instead of each run discovering it.

How to test the fix

def test_agent_stops_when_tool_keeps_failing(monkeypatch):
    monkeypatch.setattr(crm, "search", raise_unavailable)
    result = run_agent("Brief on Northwind Traders", budget=Budget(max_steps=10))
    assert result.status in {"completed_partial", "completed"}
    assert result.steps <= 10
    assert result.tool_counts["search_crm"] <= 2

We also replayed the Sunday incident against staging with the CRM mocked as down. Runs finished in 6 to 9 steps with briefs that said CRM data was unavailable.

Confirming in production

• Steps per run: p50 7, p99 16, max 25 (the limit) over the following month.
• Cost per run is now logged and charted; p99 is about 4 percent of the Sunday run.
• During a later CRM maintenance window, runs completed with a "CRM unavailable" note, and no loops occurred.

Regression guard

• Alerts on any run that hits a budget limit, and on spend per hour per feature above three times its trailing average.
• Every job queue has a time limit; the worker refuses to start agent jobs without one.
• The failing-tool test runs in CI for every tool.

Security implications

An agent without limits is a denial-of-wallet risk: any input that makes the agent loop, including content planted on a web page it reads, can run up cost. Budgets cap that per run, and per-user quotas cap it per account. We also stopped returning raw exception text to the model, since internal hostnames and stack traces were ending up in briefs.

Prevention checklist

• Never return empty strings from tools on failure; return structured, actionable errors.
• Give every agent run a step, token and time budget.
• Detect repeated identical tool calls.
• Finish gracefully with partial results when a budget is hit.
• Track and alert on cost per run and per feature.

References

• OpenAI function calling guide — https://platform.openai.com/docs/guides/function-calling
• Anthropic tool use documentation — https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview
• Celery task time limits — https://docs.celeryq.dev/en/stable/userguide/workers.html#time-limits