Third-party APIs, webhooks, payment gateways, OAuth providers.
A field guide to receiving webhooks reliably: verify signatures on the raw body, respond fast, queue the work, deduplicate by event ID, tolerate out-of-order delivery, replay safely and monitor the whole pipeline. With code you can adapt.
We launched local pricing in JPY, KRW and VND and a customer in Osaka was charged 398,000 yen for a 3,980 yen course. Our code multiplied every price by 100 before sending it to Stripe. How we found it, refunded it and built a per-currency exponent table with tests.
Our partner OAuth server matched redirect_uri by prefix, and our login page followed any next URL. Chained together with a legacy implicit flow, an access token could land on an outside domain. We moved to exact matching, relative-only next and code plus PKCE.
Sign in with Google worked locally and on staging but failed in production with redirect_uri_mismatch. Behind the TLS proxy our FastAPI app built an http:// callback because Uvicorn did not trust the proxy's headers. How we traced and fixed it.
After we tightened DMARC to quarantine, Gmail started filing our password resets and receipts as spam. Both SPF and DKIM passed, but for amazonses.com, not our domain. Here is how we read the headers, aligned both, and kept it that way.
A slow accounting API pushed our PayPal webhook handler past the nginx timeout. PayPal saw a 504 and retried, while the first request was still running, and 41 wallets were topped up twice. Postmortem plus the idempotent design we moved to.
Our PDF invoices and the amounts charged through Stripe disagreed by a cent on 214 invoices in one month. Floats, toFixed and rounding at every step were to blame. We moved to integer minor units with one explicit rounding point.
After a harmless-looking Express refactor, every Stripe webhook failed verification and paid bookings stayed pending. A global JSON parser consumed the body before constructEvent saw it. How we found it, fixed the route and replayed the missed events.
On November 3, 2024, our 01:30 billing run fired at 01:30 CDT and again at 01:30 CST, charging 1,284 subscriptions twice. The in-house scheduler matched local wall-clock time. We moved schedules to UTC and added idempotency keys, a run ledger and a unique constraint.