The first version of most trading bots is a loop: fetch prices, compute a signal, place an order, sleep. It works on a quiet afternoon. Then the websocket disconnects during a fill, an order times out, the process restarts with orders open, or two signals arrive at once, and the bot no longer knows what it owns.

Over several years of running automated trading systems, we have converged on an architecture that keeps state correct under those conditions. This guide describes it. It is about engineering structure; it does not cover which strategies to run.

Separate the components

market data  ->  strategy  ->  risk  ->  order manager  ->  exchange gateway
     ^                                         |                   |
     |                                         v                   v
     +---------------- event bus / log <---- fills, acks, errors --+
                              |
                         persistence + reconciliation
  • Exchange gateway: talks to the exchange (REST, websockets), handles authentication, rate limits, retries for reads, and normalises messages into internal events. Knows nothing about strategy.
  • Market data: maintains order books, candles, mark prices, with sequence checks and resync (see our post on order book sync).
  • Strategy: consumes market data and position state, emits intents ("target position 0.5 BTC", "place limit buy at X"). It never calls the exchange directly.
  • Risk: checks every intent against limits before it reaches the order manager.
  • Order manager: turns intents into orders, owns the order state machine, and is the only component that places or cancels orders.
  • Persistence and reconciliation: an append-only event log, periodic snapshots of state, and regular comparison with the exchange.

The separation makes each part testable alone and limits the damage of bugs: a strategy bug produces bad intents, which risk can block; it cannot corrupt order state.

The order state machine

Each order has a lifecycle with explicit states and allowed transitions:

PENDING_NEW -> NEW -> PARTIALLY_FILLED -> FILLED
            \-> REJECTED
            \-> UNKNOWN -> (resolved by query) -> NEW | FILLED | REJECTED | NOT_FOUND
NEW / PARTIALLY_FILLED -> PENDING_CANCEL -> CANCELED | FILLED

Rules that matter:

  • Orders get a client order id before sending, so every later event can be matched (see our post on duplicate orders).
  • Unknown is a first-class state for timeouts and disconnects, resolved by querying, never by guessing.
  • Transitions are idempotent by exchange event ids; receiving the same fill twice changes nothing.
  • Events can arrive out of order (a fill before the placement acknowledgment); the state machine must accept them (see our post on fill-before-ack races).

Positions come from fills

Position and balance state is derived from fills and ledger entries, not from intents or order requests. The strategy's view of "my position" is a projection maintained by the order manager from confirmed executions. That one rule prevents a whole family of bugs.

Risk checks before every order

Simple, fast, independent checks:

  • Maximum position per symbol and in total.
  • Maximum order size and notional.
  • Price sanity: limit price within a band around the mark or mid price.
  • Rate limits on orders per second per symbol.
  • Daily loss limit; when hit, only risk-reducing orders are allowed.
  • Kill switch state (see our post on kill switches).

Risk has its own configuration and owner, and changes to it are reviewed separately from strategy changes.

Persistence and restarts

On restart, a bot must not assume a clean slate. The startup sequence:

  1. Load the last snapshot and replay the event log since then.
  2. Connect to the exchange, subscribe to user data streams, and buffer events.
  3. Reconcile: fetch open orders, recent trades and balances over REST; resolve differences in favour of the exchange.
  4. Apply buffered events.
  5. Only then let the strategy run.

A bot that starts trading before reconciling can double its exposure after a crash.

Reconciliation, continuously

Every few minutes, compare open orders, positions and balances with the exchange. Differences are logged as events, corrected, and alerted if they exceed thresholds. Reconciliation is what turns silent drift into visible, fixable incidents.

Observability

Per order: client id, intents that caused it, timestamps for each state change, and latency between decision and acknowledgment. Per component: event lag, reconnects, resyncs, risk rejections. A dashboard that shows "what does the bot think it owns" next to "what the exchange says it owns" is the single most useful view.

Testing

  • Unit tests for the state machine with out-of-order, duplicate and missing events.
  • Replay tests from recorded sessions.
  • Chaos tests against exchange testnets: kill connections, delay responses, restart processes mid-order.
  • A paper or minimal-size live phase for any change to order handling.

Deployment and operations

Run each bot as its own process with its own credentials and sub-account, so failures and restarts are contained. Deploy changes to order handling behind a flag, first to a testnet environment with chaos tests, then to production with minimal size. Keep the previous version deployable for fast rollback. Configuration that affects risk (limits, symbols, sizes) is versioned and reviewed like code, and the bot logs its full effective configuration at startup so incidents can be analysed against what was actually running.

Scaling the design

The same structure works for one bot or twenty. As the number of strategies grows, the gateway, risk and reconciliation components become shared services, and strategies become clients that emit intents. That makes cross-strategy risk limits possible: total exposure across bots, not just per bot. It also concentrates exchange-specific code in one place, which matters when exchanges change their APIs.

Common mistakes

Mistake Why it hurts Do this instead
Strategy calls the exchange directly Bugs bypass risk and corrupt state Strategy emits intents only
Positions from intents State diverges after failures Positions from confirmed fills
No unknown state Duplicates or lost orders on timeouts Unknown plus query-based resolution
Trading before reconciliation on restart Double exposure Reconcile, then start strategy
Websockets as source of truth Missed events Periodic REST reconciliation

Key takeaways

  • Separate gateway, market data, strategy, risk and order management.
  • Model orders with an explicit, idempotent state machine including unknown.
  • Derive positions from fills and ledgers, never from intentions.
  • Reconcile on startup and continuously, in favour of the exchange.
  • Test with failures, not just with happy paths.

References