Automated trading systems fail differently from most software. A web service bug degrades a page; a trading bug places orders, and every second it runs can cost money. The incidents we have written about, duplicate orders on retry, missed fills after disconnects, wrong market types, test configuration on mainnet, all have one thing in common: the damage depended on how quickly something stopped the bot.

This guide is the set of monitoring and stopping mechanisms we require before any bot trades real money. It is engineering guidance for operating automated systems safely, not advice about what to trade.

Principle: assume the bot will misbehave

Design controls that work when the bot's own logic is wrong. That means the most important limits live outside the bot: in an independent risk process, in exchange-side features, and in human-operable switches.

Heartbeats

Every bot publishes a heartbeat every few seconds with its state: connected, synced, last market data time, last account event time, open orders, position. A watchdog process checks:

  • Heartbeat age (process alive).
  • Market data freshness per symbol.
  • Account event freshness and recent reconciliation status.

If any check fails, the watchdog moves the bot to a safe state (see below) without waiting for a human.

Exchange-side protections

Use what the exchange offers, because it works even if your infrastructure is down:

  • Cancel-on-disconnect or dead man's switch features: some venues cancel all open orders if your session drops or if you stop sending a periodic signal. Configure them where available.
  • Reduce-only and post-only flags on orders to limit what a bug can do.
  • Account and sub-account limits: fund each bot's sub-account only with what it needs.
  • Key permissions and IP allowlists: no withdrawal permission, fixed IPs.

Limits in an independent risk process

The risk process sits between bots and the exchange gateway (or subscribes to their activity and can act on their accounts). It enforces:

Limit Example
Max position per symbol and total Notional and quantity caps
Max order size Per order, per symbol
Order rate Orders per second and per minute per bot
Price band Limit price within X percent of mark
Daily loss / drawdown Stop increasing exposure after a loss threshold
Unexpected activity Trades in symbols the bot is not configured for

Breaching a soft limit blocks new risk-increasing orders; breaching a hard limit triggers the kill switch for that bot.

The kill switch

A kill switch must be fast, simple and tested. Ours does, in order:

  1. Set the bot's state to HALTED in the shared coordination store; the order gateway rejects all its new orders immediately.
  2. Cancel all open orders for the bot's accounts on every venue (using the exchanges' cancel-all endpoints).
  3. Optionally flatten positions, depending on configuration and only for venues and instruments where that is safe; otherwise leave positions for a human.
  4. Notify the on-call channel with the reason and current positions.

It can be triggered by the risk process, the watchdog, a chat command, or a command-line tool, and it works even if the bot process is hung, because it acts on the gateway and the exchange directly.

tradectl halt --bot mm-eth --reason "manual: unexpected fills" --cancel-all

Test it monthly on testnet, and once a quarter in production with a small, controlled scenario. A kill switch that has never been used is a hope, not a control.

Alert design

  • Page for: hard limit breaches, kill switch activations, heartbeat loss with open orders, reconciliation failures, account events stale while orders are open.
  • Notify for: soft limit hits, frequent reconnects, rising rejection rates, latency increases.
  • Report daily: PnL, fees, funding, reconciliation summary, limit utilisation.

Every page links to the runbook and to a dashboard showing the bot's view of positions next to the exchange's view.

Runbook: the first five minutes

  1. Stop the bleeding: if in doubt, halt the bot and cancel orders. Halting is cheap; unexplained activity is not.
  2. Compare views: bot positions and open orders versus exchange records.
  3. Check recent changes: deploys, configuration changes, exchange announcements, maintenance.
  4. Decide on positions: leave, reduce or close, by a person with authority to make that decision.
  5. Record: timeline, actions and observations, for the postmortem.

Practising

Every quarter we run a game day: on testnet, someone injects a failure (a bot that places orders in a loop, a stale market data feed, a lost user stream) without telling the on-call engineer which one. We measure time to detection, time to halt, and whether the runbook was enough. The first game day took 11 minutes to halt a misbehaving test bot; the most recent took under 90 seconds. The improvements came from small things: a clearer alert title, a chat command for halting, and a dashboard that shows bot and exchange positions side by side.

Common mistakes

Mistake Why it hurts Do this instead
Limits only inside the bot A bug in the bot disables its own limits Independent risk process
Kill switch never tested Fails when needed Monthly drills
Relying on process liveness only A live but desynced bot keeps trading Freshness and reconciliation checks
Ignoring exchange-side protections No safety when your infra is down Cancel-on-disconnect, reduce-only, sub-accounts
Pages without runbooks Slow, inconsistent responses Link a short runbook to every alert

Key takeaways

  • Put critical limits outside the bot, in an independent process.
  • Use exchange-side protections such as cancel-on-disconnect and scoped accounts.
  • Build a fast, simple kill switch and test it regularly.
  • Monitor freshness and reconciliation, not just liveness.
  • Keep a short runbook and halt first when in doubt.

References