Once more than two services call LLM providers, each one reinvents timeouts, retries and key handling, usually badly. This guide covers what a small internal LLM gateway should do, the timeout and retry rules that work, and how to fail over without making outages worse.
During a volatile minute, our place-order request timed out and the bot retried. Both requests reached the exchange, and we held twice the intended position. A timeout does not mean the order failed. Client order ids, query-before-retry and an unknown state fixed it.
A routine database failover lasted 20 seconds, but our API stayed down for 41 minutes. Three layers each made up to three attempts with no delay, multiplying load on the database by 27. Here is the math, the fix and how we proved it.