Order state machine crashes with unknown order: the fill arrived before the REST response

The order manager in our execution service logged a few errors a day, always on aggressive limit orders or market orders:

ERROR ws execution handler: KeyError: 'unknown order 4917722013'
Traceback (most recent call last):
  File "oms/handlers.py", line 88, in on_execution_report
    order = self.orders_by_exchange_id[ev["i"]]
KeyError: 4917722013

The fill was then lost from the bot's state until the next reconciliation, a few minutes later.

Reproducing it

Timestamps from one occurrence:

12:03:41.102 send   POST /api/v3/order clientOrderId=mm-7f2c... (market buy)
12:03:41.131 ws     executionReport x=NEW   i=4917722013 c=mm-7f2c...
12:03:41.132 ws     executionReport x=TRADE i=4917722013 c=mm-7f2c... X=FILLED
12:03:41.158 rest   200 {"orderId": 4917722013, "status": "FILLED", ...}

The websocket delivered the acknowledgment and the fill 27 ms before the REST response arrived. Our code only inserted the order into orders_by_exchange_id after the REST call returned.

What we ruled out

• Exchange out-of-order delivery. Events were in a sensible order; they simply travelled on a faster path than the HTTP response.
• Lost events. Every event was received; the handler failed on it.

Root cause

Order placement involved two channels: the REST request-response and the user data websocket. Their relative timing is not guaranteed. For orders that fill immediately, execution reports can arrive before the REST response. The order manager assumed the REST response always came first, and it indexed orders by the exchange's order id, which it did not know until that response arrived.

What is happening under the hood

When you place an order, the exchange processes it, generates execution events, and returns the HTTP response. The HTTP response and websocket messages travel through different systems and connections, so either can arrive first. Any code that depends on "response first, then events" has a race that shows up under load or for orders that execute instantly.

The client order id, which you choose before sending, is the only identifier available on both paths from the start. Execution reports carry it (c on Binance), as do REST responses.

Environment

Minimal reproduction

In a test, deliver a recorded executionReport for a new order to the handler before resolving the mocked REST call. The handler raises KeyError.

The fix

1. Register the order before sending, keyed by client order id.

async def place(self, intent) -> Order:
    cid = new_client_order_id(intent)
    order = Order(client_id=cid, symbol=intent.symbol, side=intent.side,
                  qty=intent.qty, state=OrderState.PENDING_NEW)
    self.orders[cid] = order                         # visible to the event handler now
    try:
        resp = await self.gateway.new_order(intent, client_order_id=cid)
        order.apply_rest_response(resp)              # idempotent
    except (TimeoutError, ConnectionError):
        order.state = OrderState.UNKNOWN             # resolved by query later
    return order

2. Event handler uses the client order id and tolerates any order of events.

def on_execution_report(self, ev):
    order = self.orders.get(ev["c"])
    if order is None:
        self.unmatched.append((time.monotonic(), ev))   # placed by another session or unknown
        return
    order.exchange_id = ev["i"]
    order.apply_execution(ev)                          # idempotent by trade id; monotonic states

apply_execution and apply_rest_response both update state monotonically: a FILLED order never goes back to NEW because a late REST response says NEW. Fills are applied once per trade id.

3. Unmatched events are kept, not dropped. Events whose client id is not known (for example after a restart, or for orders placed manually) are buffered briefly and then handed to reconciliation, which creates or updates orders from exchange records.

4. Cancel requests use the same rule. Cancels reference the client order id, so they work even before the exchange id is known.

Alternatives and trade-offs

• Wait for the websocket only, ignoring REST responses except for errors. Simpler state, but REST errors and timeouts still need handling.
• Serialise everything through one event queue, including REST responses as events. A clean design that makes ordering explicit; it is the direction our order manager has moved since.

How to test the fix

Unit tests deliver events in every permutation: ack before REST, fill before ack, REST before all, duplicate fills, late REST response after fill. Final state must be the same in all cases. A property-based test generates random interleavings for multiple concurrent orders.

Confirming in production

• No unknown order errors since deployment over several weeks.
• Fills are reflected in the bot's state within milliseconds, instead of waiting for reconciliation in the race cases.

Regression guard

• The permutation and property-based tests in CI.
• A metric for unmatched execution reports, with an alert if it rises above the low background level from manual orders.

What we would do differently

Model the order lifecycle as a state machine with explicit transitions before writing handlers. The original code was written as two independent handlers, one for REST responses and one for websocket events, each assuming the other had already run. Drawing the state diagram, including the order in which events can arrive, would have exposed the race on paper. Our architecture guide now requires that diagram for any change to order handling, and the permutation tests are generated from it.

Security implications

Missed or late fills cause the bot to misjudge its position, which affects risk limits. Making state updates order-independent and idempotent is a correctness and risk control. Client order ids should not contain sensitive information, since they appear in exchange records.

Prevention checklist

• Assign client order ids before sending, and key orders by them.
• Register orders in the state store before the request goes out.
• Make state transitions idempotent and monotonic.
• Never drop events for unknown orders; buffer and reconcile.
• Test all event orderings, not just the common one.

References

• Binance user data stream, execution report fields — https://developers.binance.com/docs/binance-spot-api-docs/user-data-stream
• Binance Spot API, new order and client order id — https://developers.binance.com/docs/binance-spot-api-docs/rest-api/trading-endpoints
• Hypothesis, property-based testing for Python — https://hypothesis.readthedocs.io/