Our checkout service went down for 47 minutes because the transactional email vendor had an outage. Placing an order does not need an email to be sent at that moment, but the order handler called the email API synchronously, the calls hung until timeout, the worker pool filled up and every other request queued behind them. A dependency that should have caused "confirmation emails are late" instead caused "nobody can buy anything". That incident made us rewrite how we think ab...
Design for failure on AWS: static stability, cells and blast radius
An email provider outage stopped customers from placing orders. Nothing about orders needed email. We rebuilt the service around three ideas: static stability, cell-based isolation and a blast-radius budget per dependency. Here is the worked design.
This is a paid story
Subscribe to read the full content, access all paid stories, and support the authors.
Add a label
Embed this post
<iframe src="https://hibug.me/design-for-failure-aws-reliability" width="700" height="420" frameborder="0" loading="lazy"></iframe>
I am a Senior Software Engineer with over a decade of hands-on experience building, scaling, and maintaining complex systems across the full stack—spanning frontend, backend, and DevOps. Over the last 10 years, I have navigated diverse architectures and untangled some of the most frustrating, elusive bugs in production. I believe that every production incident is a massive learning opportunity. Having discovered hibug.me, I am excited to contribute to this knowledge base, breaking down root causes, and sharing verified fixes to help fellow developers debug faster and build more resilient systems.
Related Stories
A catch-all exception handler reported a CRM sync as successful for five weeks
Retries done right: which errors to retry, retry budgets, and not stacking retries across hops
Comments
0No comments yet. Be the first to comment!
Sign In to post a comment.