A timeout and a retry opened two positions: idempotent order placement with client order ids

A reconciliation alert fired: the bot's position in ETHUSDT was twice what its strategy state said. The order log showed one intended order and two fills:

14:31:02.118 INFO  place_order ETHUSDT BUY MARKET qty=2.000
14:31:07.120 WARN  place_order timeout after 5000 ms, retrying
14:31:07.391 INFO  place_order ETHUSDT BUY MARKET qty=2.000 -> orderId=4820019123 FILLED

The exchange's trade history showed two orders, 4820018877 at 14:31:06.9 and 4820019123 at 14:31:07.4.

Reproducing it

On the testnet, we added an artificial 6-second delay in a proxy between the bot and the exchange for the first request only. The first request reached the exchange and executed; our client gave up after 5 seconds and sent the retry, which also executed.

What we ruled out

• A double call from strategy logic. The strategy emitted one order intent.
• Exchange duplication. The two orders had different request timestamps matching our two attempts.

Root cause

Our HTTP client treated a timeout as "the order failed" and retried. A timeout only means we did not get a response in time. The request may have been lost, rejected, or fully executed. For non-idempotent operations like placing orders, a blind retry after an unknown outcome can do the operation twice. Our orders had no client-assigned id, so the exchange had no way to recognise the retry as the same order.

What is happening under the hood

Order placement is a write. Distributed systems give three possible outcomes for a write request: success, failure, or unknown (timeout, connection reset, 5xx after the request was processed). Only the first two are safe to act on directly.

Most exchanges let you attach a client order id to each order (newClientOrderId on Binance, clientOrderId on many others). It serves two purposes:

• Lookup: you can query the order by your own id, without knowing the exchange's order id.
• Duplicate protection on some venues: an exchange may reject a new order with a client id already used by an open order. Behaviour varies by exchange and order state, so do not rely on it alone.

Environment

Minimal reproduction

Place a market order through a proxy that delays the response beyond the client timeout, with retries enabled. Two orders execute.

The fix

1. Assign a client order id before the first attempt, derived from the strategy's intent so that any retry of the same intent carries the same id:

import hashlib

def client_order_id(strategy: str, intent_id: str) -> str:
    # Binance allows up to 36 chars from a limited character set
    return f"{strategy[:8]}-{hashlib.sha1(intent_id.encode()).hexdigest()[:24]}"

2. Never auto-retry order placement on unknown outcomes. Retries for writes were removed from the generic HTTP client. Order placement has its own logic:

async def place_order_once(intent) -> OrderState:
    cid = client_order_id(intent.strategy, intent.id)
    try:
        resp = await api.new_order(symbol=intent.symbol, side=intent.side, type=intent.type,
                                   quantity=intent.qty, newClientOrderId=cid, timeout=5)
        return OrderState.from_response(resp)
    except (TimeoutError, ConnectionError, ServerError):
        return OrderState.unknown(cid)

async def resolve_unknown(cid: str, symbol: str, attempts: int = 10) -> OrderState:
    for i in range(attempts):
        try:
            remote = await api.get_order(symbol=symbol, origClientOrderId=cid)
            return OrderState.from_response(remote)          # it exists: do not resend
        except OrderNotFound:
            if i >= 2:                                       # give the exchange time to register it
                return OrderState.not_placed(cid)
        await asyncio.sleep(0.5 * (i + 1))
    raise StillUnknown(cid)

Only when the order is confirmed not to exist does the strategy decide whether to place it again, with the same client order id.

3. Strategy waits on unknown. While an order is in UNKNOWN, the strategy does not issue new orders for that symbol. If StillUnknown is raised, the bot pauses the symbol and alerts a human.

4. Reconcile fills by client order id prefix, so any order the bot placed, even one it lost track of, is attributed correctly.

Alternatives and trade-offs

• WebSocket order placement APIs where offered. Lower latency, same unknown-outcome problem on disconnects.
• Limit orders instead of market orders for non-urgent entries. A duplicate limit order is easier to detect and cancel before it fills.

How to test the fix

The delayed-proxy test on testnet now produces exactly one order: the bot reports UNKNOWN, queries by client order id, finds the filled order, and records it. A second test drops the request entirely; the query returns not found and the bot places the order once.

Confirming in production

• No duplicate positions since the change. Unknown outcomes occur a few times per week, almost all resolved within a second by query.
• Reconciliation differences attributable to order placement dropped to zero.

Regression guard

• The HTTP client refuses to auto-retry non-GET requests; order code must use the order manager.
• Every order sent without a client order id fails a check in the order manager.
• Chaos tests with delayed and dropped responses run against the testnet weekly.

Security implications

Duplicate orders directly create unintended exposure, a financial risk. Treating unknown outcomes explicitly and pausing when uncertain are core risk controls. Separately, client order ids should not encode sensitive information, since they appear in exchange records and exports.

Prevention checklist

• Never blindly retry order placement after a timeout.
• Assign a client order id per intent, before the first attempt.
• Resolve unknown outcomes by querying with the client order id.
• Block new orders on a symbol while an order is unknown.
• Test with delayed and dropped responses.

References

• Binance Spot API, new order (newClientOrderId) — https://developers.binance.com/docs/binance-spot-api-docs/rest-api/trading-endpoints
• Binance Spot API, query order by origClientOrderId — https://developers.binance.com/docs/binance-spot-api-docs/rest-api/account-endpoints
• Stripe, designing robust and predictable APIs with idempotency — https://stripe.com/blog/idempotency