Automated trading fails fast: a bug can place hundreds of orders before a human notices. This guide covers the monitoring and stopping mechanisms we run for every bot: heartbeats, position and loss limits, cancel-all on disconnect, a tested kill switch, and runbooks for the first five minutes.
Our nightly invoice export stopped running and nobody noticed for nine days. One Job had hung on a network call with no timeout, and with concurrencyPolicy Forbid every later schedule was skipped. Deadlines on the Job, timeouts in code and an alert on last success fixed it.
Our customer portal went dark at 07:12 when its Let's Encrypt certificate expired. Certbot's timer had run twice a day for a month and failed every time, because a firewall change blocked port 80. Here is how we traced it, fixed it and made expiry impossible to miss.