Local order book drifts from the exchange: diff updates applied without a correct snapshot sync

Our quoting service keeps a local order book per symbol, built from the exchange's diff depth websocket stream, and uses it to place passive orders. After running for a few hours, sanity checks started firing:

WARN BTCUSDT local book crossed: best_bid=67412.80 best_ask=67411.95
WARN BTCUSDT level 67380.00 qty=1.204 present locally, absent in REST snapshot

A crossed book (best bid above best ask) cannot exist on the exchange. Our quotes were being priced off a corrupted view.

Reproducing it

We ran our book builder alongside periodic REST snapshots and compared the top 20 levels every minute. Divergence appeared within 30 minutes on busy pairs, and immediately after any websocket reconnect.

The builder's logic, simplified:

book = fetch_snapshot("BTCUSDT", limit=1000)     # REST
async for event in ws_depth("btcusdt@depth@100ms"):
    apply(book, event)                           # apply every diff, no checks

What we ruled out

• Exchange data errors. A second client implementing the documented procedure stayed in sync.
• Float issues in prices. Prices were stored as strings and Decimals.

Root cause

Three mistakes:

1. No buffering before the snapshot. We fetched the REST snapshot, then opened the stream. Events that happened between the snapshot and the stream start were lost.
2. No alignment by update id. We applied events that were already included in the snapshot and did not check that the first applied event bridged the snapshot's lastUpdateId.
3. No gap detection. Each diff event carries a first and last update id (U and u). We never checked that each event continued from the previous one, so a dropped message during a network blip or reconnect silently corrupted the book forever.

What is happening under the hood

A diff depth stream sends changes, not full books. Each event says "these price levels now have these quantities" (quantity 0 means remove the level). To build a correct book you need a starting point (the REST snapshot) and an unbroken sequence of changes after it. The documented procedure for spot markets is, in short:

1. Open the websocket and start buffering events. Note U of the first event.
2. Fetch a depth snapshot. If its lastUpdateId is lower than the first buffered event's U, fetch again.
3. Discard buffered events whose u is less than or equal to lastUpdateId.
4. The first event to apply must satisfy U <= lastUpdateId + 1 <= u.
5. From then on, each event's U must equal the previous event's u + 1. If not, events were missed: start over from step 1.
6. Quantities are absolute values for the level; 0 removes the level.

Futures streams use a slightly different continuity field (pu, the previous event's final update id), so read the procedure for the market you consume.

Environment

Minimal reproduction

Fetch the snapshot first, open the stream a second later, and apply all events: on a busy symbol, the book diverges within minutes because the events in between are missing.

The fix

We rewrote the builder as a small state machine following the procedure exactly:

from decimal import Decimal

class BookSync:
    def __init__(self, symbol):
        self.symbol, self.buffer, self.last_u = symbol, [], None
        self.bids, self.asks, self.synced = {}, {}, False

    def on_event(self, ev):
        if not self.synced:
            self.buffer.append(ev)
            return
        if ev["U"] != self.last_u + 1:
            raise GapDetected(self.symbol, expected=self.last_u + 1, got=ev["U"])
        self._apply(ev)

    def initialise(self, snapshot):
        last_id = snapshot["lastUpdateId"]
        if self.buffer and last_id < self.buffer[0]["U"]:
            raise SnapshotTooOld()                       # caller fetches a new snapshot
        self.bids = {Decimal(p): Decimal(q) for p, q in snapshot["bids"]}
        self.asks = {Decimal(p): Decimal(q) for p, q in snapshot["asks"]}
        pending = [e for e in self.buffer if e["u"] > last_id]
        if pending and not (pending[0]["U"] <= last_id + 1 <= pending[0]["u"]):
            raise GapDetected(self.symbol, expected=last_id + 1, got=pending[0]["U"])
        self.last_u = last_id
        self.synced = True
        self.buffer.clear()
        for ev in pending:
            self.on_event(ev)

    def _apply(self, ev):
        for side, levels in ((self.bids, ev["b"]), (self.asks, ev["a"])):
            for p, q in levels:
                p, q = Decimal(p), Decimal(q)
                if q == 0:
                    side.pop(p, None)          # removing an unknown level is normal
                else:
                    side[p] = q
        self.last_u = ev["u"]

The supervisor opens the stream first, waits for the first event, fetches the snapshot, and calls initialise. On GapDetected or SnapshotTooOld, or on any websocket reconnect, it discards the book and restarts the procedure. While resyncing, the quoting service treats the symbol as unavailable and cancels its resting quotes.

Sanity checks remain as a last line of defence: a crossed book triggers an immediate resync.

Alternatives and trade-offs

• Partial book depth streams (top N levels pushed in full every 100 ms). No sync logic needed; only the top levels, and no update ids to detect gaps in the same way. Good enough for many strategies.
• Periodic full resync every few minutes as a safety net. Adds REST load; not a substitute for gap detection.

How to test the fix

We replay recorded websocket sessions with injected faults (dropped events, duplicated events, delayed snapshot) and assert that the builder either matches a reference book or raises and resyncs. A live soak test compared the local top 20 levels with REST snapshots every 10 seconds for 72 hours: zero mismatches outside resync windows.

Confirming in production

• No crossed books in a month across six symbols.
• Resyncs happen a few times a day, mostly after websocket reconnects, and complete in under a second.
• Quote rejection for "would cross" errors dropped to zero.

Regression guard

• Metrics: resync count per symbol, time spent unsynced, and a periodic REST comparison mismatch counter.
• Replay tests with fault injection in CI.

Security implications

A corrupted book is a risk-control failure: decisions are made on prices that do not exist. The service cancels quotes and stops quoting a symbol while its book is not synced. API keys for quoting have trading permission only, no withdrawals, and an IP allowlist. This post covers data correctness, not trading strategy.

Prevention checklist

• Open the stream and buffer before fetching the snapshot.
• Align the first applied event with lastUpdateId using the documented rule.
• Check continuity of update ids on every event.
• Resync on any gap or reconnect, and stop trading the symbol meanwhile.
• Compare against REST snapshots periodically as a monitor.

References

• Binance Spot WebSocket streams, how to manage a local order book correctly — https://developers.binance.com/docs/binance-spot-api-docs/web-socket-streams
• Binance USD-M futures, how to manage a local order book correctly — https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/How-to-manage-a-local-order-book-correctly
• Binance depth REST endpoint — https://developers.binance.com/docs/binance-spot-api-docs/rest-api/market-data-endpoints