Trading bot missed fills overnight: websocket dropped at 24 hours and nobody reconciled

In the morning, our bot's dashboard showed two open sell orders and the matching inventory, but the exchange's order history showed both filled at 02:47. The bot had not known for five hours, so it did not place the follow-up buy orders its logic required, and its position report was wrong.

The connection log:

2025-06-10 02:46:58 INFO  ws user stream: connection closed by server code=1000
2025-06-10 02:47:01 INFO  ws user stream: reconnecting (attempt 1)
2025-06-10 02:47:03 INFO  ws user stream: connected, subscribed

The fills happened during those few seconds.

Reproducing it

The connection had been opened exactly 24 hours earlier. We closed the connection manually in staging while an order was filling on the testnet, and the bot missed the execution report in the same way.

What we ruled out

• Exchange not sending the event. The order history API showed the fill with a timestamp inside our disconnect window.
• A parsing error. No exceptions in the message handler.

Root cause

Exchanges limit the lifetime of websocket connections; Binance closes connections after 24 hours. Messages sent while you are disconnected are not replayed when you reconnect. Our client handled the disconnect correctly at the transport level, reconnected in two seconds and resubscribed, but it treated the stream as a complete record of events. Anything that happened in the gap was lost from the bot's view of the world, permanently.

What is happening under the hood

Websocket streams are a live feed, not a durable log. Disconnections happen for many reasons: scheduled maximum connection lifetime, server maintenance, network changes, ping timeouts if your client does not answer pings, and load balancer idle timeouts. Most exchanges document a maximum lifetime and a ping/pong requirement: the server pings periodically and expects a pong within a time limit, or closes the connection.

The only safe assumption is that every reconnect may hide missed events. After reconnecting, a client must re-establish state from an authoritative source, usually REST endpoints for open orders, recent trades and balances.

Environment

Minimal reproduction

Open a user data stream, place an order on testnet that fills shortly, drop the connection during the fill, reconnect, and observe that the execution report never arrives.

The fix

1. Reconnect with backoff and proactive rotation. We rotate connections ourselves before the 24-hour limit, overlapping the new connection with the old one so there is no gap for market data, and use exponential backoff with jitter on unexpected disconnects.

2. Reconcile after every (re)connect. As soon as a new user stream connection is subscribed, the bot fetches authoritative state over REST and merges it:

async def reconcile(client, state):
    open_orders = await client.get_open_orders()                 # REST
    open_ids = {o["clientOrderId"] for o in open_orders}
    for cid, local in list(state.open_orders.items()):
        if cid not in open_ids:
            remote = await client.get_order(orig_client_order_id=cid)
            state.apply_remote_order(remote)                    # FILLED, CANCELED, EXPIRED...
    for o in open_orders:
        state.upsert_open_order(o)
    trades = await client.get_my_trades(since=state.last_trade_time - 60_000)
    state.apply_trades(trades)                                  # idempotent by trade id
    state.balances = await client.get_balances()

Applying trades is idempotent by trade id, so overlapping data from REST and the stream cannot double count.

3. Event ordering during reconciliation. Messages arriving on the new stream while reconciliation runs are buffered and applied afterwards, then de-duplicated by execution id.

4. Periodic reconciliation anyway. Every 5 minutes, even without disconnects, the bot compares its open orders and balances with REST. Differences raise an alert and are corrected from the exchange's data.

5. Pause on uncertainty. If reconciliation fails (for example, REST errors), the bot stops placing new orders until it succeeds.

Alternatives and trade-offs

• Poll REST only. Simpler state, but slower reactions and rate limit pressure.
• Use exchange features that replay or query by sequence where available. Some venues provide sequence numbers or recovery endpoints; use them when offered.

How to test the fix

In staging, a chaos test closes the user stream connection at random intervals while orders fill on the testnet. After 500 forced disconnects, the bot's state matched the exchange's records every time.

Confirming in production

• Reconciliation corrects about one or two order states per week, each corresponding to a disconnect; before, these were silent drifts.
• No more stale "open" orders in the morning report.

Regression guard

• Metric and alert for reconciliation differences found per hour.
• Alert when the user stream has been disconnected for more than 30 seconds.
• The chaos test runs in CI against the testnet weekly.

What we would do differently

We would have designed for reconciliation from the start instead of adding it after an incident. Any system that mirrors external state from a stream needs three parts: the stream for speed, an authoritative source for truth, and a regular comparison between them. Our new services start with all three, and the stream is treated as an optimisation of the reconciliation loop rather than the other way around.

Security implications

A bot that does not know its true positions can exceed risk limits, for example by re-placing orders it thinks are missing. Reconciliation and the pause-on-uncertainty rule are risk controls. API keys for the bot cannot withdraw and are IP-restricted.

Prevention checklist

• Treat websocket streams as best-effort, never as the source of truth.
• Reconcile open orders, trades and balances via REST after every reconnect.
• Rotate connections before exchange-imposed lifetimes.
• Make state updates idempotent by exchange ids.
• Stop trading when state cannot be confirmed.

References

• Binance WebSocket market streams, connection limits and ping/pong — https://developers.binance.com/docs/binance-spot-api-docs/web-socket-streams
• Binance user data stream documentation — https://developers.binance.com/docs/binance-spot-api-docs/user-data-stream
• Binance Spot REST, query order and account trade list — https://developers.binance.com/docs/binance-spot-api-docs/rest-api/account-endpoints
• RFC 6455, The WebSocket Protocol — https://www.rfc-editor.org/rfc/rfc6455