A field guide to receiving webhooks reliably: verify signatures on the raw body, respond fast, queue the work, deduplicate by event ID, tolerate out-of-order delivery, replay safely and monitor the whole pipeline. With code you can adapt.
During a volatile minute, our place-order request timed out and the bot retried. Both requests reached the exchange, and we held twice the intended position. A timeout does not mean the order failed. Client order ids, query-before-retry and an unknown state fixed it.
A slow accounting API pushed our PayPal webhook handler past the nginx timeout. PayPal saw a 504 and retried, while the first request was still running, and 41 wallets were topped up twice. Postmortem plus the idempotent design we moved to.
On November 3, 2024, our 01:30 billing run fired at 01:30 CDT and again at 01:30 CST, charging 1,284 subscriptions twice. The in-house scheduler matched local wall-clock time. We moved schedules to UTC and added idempotency keys, a run ledger and a unique constraint.
Our onboarding page created two draft workspaces per visit in development. StrictMode was blamed and nearly disabled, but the real bug was a non-idempotent POST inside an effect with no cleanup, and it was also happening in production.