Trading bot started on mainnet with test sizing after a config refactor

This is a blameless postmortem of a configuration incident. At 10:42 on a Tuesday, the risk monitor paged: an account used by an experimental strategy had placed four orders of 2.0 BTC notional equivalent each, while its production limit was 0.2. The risk process blocked the fifth order and cancelled the open ones. Two orders had filled.

The bot's startup log:

10:41:58 INFO config loaded: env=default profile=experimental
10:41:58 INFO exchange base_url=https://api.binance.com
10:41:58 INFO max_order_notional=20000 (from profile)

The developer believed they were starting it against the spot testnet.

Reproducing it

The configuration had been refactored the previous week from one file per environment into a layered system: defaults, then an environment layer, then a strategy profile. The new defaults file contained the production base URL. The testnet environment layer was selected by an environment variable BOT_ENV=testnet. The developer's shell had ENV=testnet set from the old system. With BOT_ENV unset, the loader silently used the defaults: production endpoints.

The bot had credentials for production because the developer's local secret file contained both testnet and production keys, with the production key also loaded by default.

What we ruled out

• A code change in the strategy. The strategy's logic was unchanged; it used parameters intended for testnet experiments.
• Exchange-side issues. Orders were placed exactly as requested.

Root cause

Several latent conditions lined up:

1. Production as the default. Missing environment selection fell back to production, the most dangerous choice.
2. Silent rename. The variable changed from ENV to BOT_ENV without failing on the old one.
3. Keys available everywhere. A developer machine could load production trading keys.
4. Profiles not tied to environments. An "experimental" profile with large test sizes could be combined with production endpoints.

The risk process limited the damage because it runs independently and enforces per-account limits.

What is happening under the hood

Configuration systems often fail open: when a value is missing, they use a default. That is convenient in development and dangerous when the default points at production. Trading systems need to fail closed: if the environment is not explicitly and consistently specified, refuse to start.

Separation of environments should exist at several layers at once: configuration, credentials, network access and account permissions, so that one mistake cannot connect test intentions to production money.

Environment

Minimal reproduction

Start the bot with no BOT_ENV set: it loads production URLs from defaults.

The fix

1. No default environment. The loader requires BOT_ENV to be one of testnet or production and exits otherwise. The old ENV variable now causes an error explaining the rename.

VALID = {"testnet", "production"}

def resolve_env() -> str:
    if "ENV" in os.environ and "BOT_ENV" not in os.environ:
        sys.exit("ENV is no longer used; set BOT_ENV=testnet or BOT_ENV=production")
    env = os.environ.get("BOT_ENV")
    if env not in VALID:
        sys.exit(f"BOT_ENV must be one of {sorted(VALID)}, got {env!r}")
    return env

2. Startup assertions that cross-check layers. After loading, the bot verifies that the base URL, the key's account and the profile all belong to the same environment:

assert cfg.env == "production" or "testnet" in cfg.base_url, "testnet env with non-testnet URL"
assert cfg.env == "testnet" or "testnet" not in cfg.base_url, "production env with testnet URL"
account = client.account_info()
assert account_tag(account) == cfg.env, "API key belongs to a different environment"
assert cfg.profile.allowed_envs and cfg.env in cfg.profile.allowed_envs, "profile not allowed in this env"

3. Production keys only on production hosts. Developer machines no longer have production trading keys at all. Production keys exist only in the secret manager, readable by production workloads, and are IP-restricted to production egress IPs, so even a leaked key would not work from a laptop.

4. Production starts are deliberate. Starting in production requires a deployment through the pipeline, which runs a dry-run mode first (orders built and checked against risk limits, not sent) for a minute, then switches to live after confirmation.

5. Risk limits stay independent. The risk process caught this incident; its limits are configured separately and cannot be overridden by strategy profiles.

Alternatives and trade-offs

• Separate builds per environment. Removes runtime selection entirely; adds build complexity.
• Distinct accounts and exchanges for testing only reachable from test networks. We already used the testnet; network-level separation for developer machines is the remaining layer, which IP allowlists now provide.

How to test the fix

Tests start the bot with each combination of missing or mismatched settings and assert it exits before connecting. A canary in the pipeline attempts to start a testnet profile with production variables and expects failure.

Confirming in production

• The two filled orders were closed within minutes; the loss was small and recorded in the incident report.
• Since the changes, startup checks have blocked three misconfigurations in development, each failing loudly in under a second.

Regression guard

• Startup assertion tests in CI.
• Quarterly review of where production keys can be read from.
• The independent risk process is a required component; bots cannot start without registering with it.

Security implications

Environment separation is a security boundary. Production credentials on developer machines expand the attack surface and the blast radius of mistakes. Restricting keys to production hosts and IPs, failing closed on configuration, and keeping risk limits independent are controls that protect against both errors and compromise.

Prevention checklist

• Never default to production; require an explicit environment.
• Cross-check endpoint, key account and profile at startup.
• Keep production keys off developer machines and restrict them by IP.
• Gate production starts through a pipeline with a dry run.
• Keep an independent risk process with its own limits.

References

• Binance Spot testnet — https://testnet.binance.vision/
• The Twelve-Factor App, config — https://12factor.net/config
• Google SRE book, postmortem culture — https://sre.google/sre-book/postmortem-culture/