AWS us-east-1 outage of October 2025: DynamoDB DNS race and lessons for our systems

On October 19 and 20, 2025, many applications built on AWS in the N. Virginia (us-east-1) Region degraded or went down for hours. Ours was among them: a background pipeline that writes to DynamoDB stalled, and our customer-facing API, hosted in a different region, still failed logins for about an hour because of a dependency we did not know was regional. A few days later AWS published a detailed summary of the event.

This post is our reading of that official summary, written for engineers who run their own services. The first half restates what AWS reported, in our words, without adding speculation. The second half is separate on purpose: the lessons we took for our own architecture and runbooks. Times below are in PDT, as in AWS's report.

What happened, per the official AWS summary

The trigger: a race in DynamoDB's DNS automation

DynamoDB manages the DNS records for its regional endpoints with an automated system that has two parts. A DNS Planner watches load balancer health and capacity and produces versioned DNS plans. A DNS Enactor applies those plans to Route 53. For resilience, three independent Enactors run in three Availability Zones, each applying plans with Route 53 transactions.

According to the report, one Enactor became unusually slow while applying a plan. Meanwhile the Planner kept producing newer plans, and a second Enactor applied a much newer one and then started a clean-up step that deletes plans considerably older than the one just applied. The Enactor only checks that its plan is newer than the applied plan once, before it starts. Because of the delay, that check was stale by the time the slow Enactor wrote its older plan, overwriting the newer one at about the same moment the clean-up removed that older plan as obsolete.

The result was an empty DNS record for the regional endpoint dynamodb.us-east-1.amazonaws.com, and a system left in an inconsistent state that blocked further automatic updates. AWS states that manual operator intervention was needed.

Timeline of the DynamoDB phase

The cascade: EC2 instance launches

The longer tail came from EC2. AWS describes a subsystem called DropletWorkflow Manager (DWFM) that manages the physical hosts ("droplets") behind EC2 and maintains leases on them, with state checks that depend on DynamoDB. While DynamoDB was unreachable, those leases gradually timed out. When DynamoDB came back at 2:25 AM, DWFM tried to re-establish leases across the whole fleet at once. The report says it entered congestive collapse: work timed out and was queued again faster than it could complete. Engineers throttled incoming work and selectively restarted DWFM hosts from 4:14 AM, and by 5:28 AM all droplets had leases again.

New instances then needed their network configuration propagated by a separate Network Manager, which now had a large backlog. Propagation delays started around 6:21 AM and returned to normal by 10:36 AM. AWS relaxed the request throttles it had applied from 11:23 AM, and EC2 recovered fully at 1:50 PM.

Network Load Balancer health checks

Between 5:30 AM and 2:09 PM, some Network Load Balancers had increased connection errors. AWS explains that health checks ran against new instances whose network state had not finished propagating, so checks alternated between failing and passing. That increased load on the health check subsystem, degraded it, and triggered automatic Availability Zone DNS failover, which removed capacity that was actually healthy. Engineers disabled automatic failover at 9:36 AM, restoring capacity, and re-enabled it at 2:09 PM once EC2 had recovered.

Other services named in the report

AWS lists impact on many services, including Lambda (an SQS polling subsystem that did not recover automatically, then under-scaling after NLB-driven terminations, with full recovery at 2:15 PM), ECS, EKS and Fargate (container launch failures until 2:20 PM), Amazon Connect, STS, IAM sign-in for the console, Redshift (some clusters took until 4:05 AM on October 21), and the AWS Support console. The report also describes effects outside the region: console sign-ins through signin.aws.amazon.com failed for customers logging in to other regions, and a Redshift defect that routed IAM calls through us-east-1 meant Redshift users in all regions could not run queries with IAM user credentials (clusters using local database users were unaffected). Both are reminders that "regional" services can have global edges.

Remediation AWS announced

• Disabled the DynamoDB DNS Planner and Enactor automation worldwide until the race is fixed, with added protection against applying incorrect plans.
• For NLB, a velocity control that limits how much capacity a single load balancer can remove when health checks trigger AZ failover.
• For EC2, an additional test suite for DWFM recovery workflows, and improved throttling that rate limits work based on queue size.

Lessons for our own systems

Nothing below is a claim about how AWS should have done things. These are patterns from the report that map onto mistakes we could make, and in some cases had already made.

Check-then-act on shared state is a race, even with "versioned" data

The Enactor compared versions once and then wrote much later. Many of our own jobs do the same: read a config version, do slow work, write. The fix pattern is a conditional write that re-checks at commit time, for example a DynamoDB condition expression or a SQL WHERE version = :expected:

aws dynamodb put-item \
  --table-name routing_plans \
  --item file://plan-1742.json \
  --condition-expression "attribute_not_exists(pk) OR plan_generation < :g" \
  --expression-attribute-values '{":g":{"N":"1742"}}'

And a clean-up job should never be able to delete the thing that is currently live. We added a guard to ours that refuses to delete any record referenced by the active pointer.

Recovery is its own load test

The longest impact was not the DNS bug but the stampede of work when the dependency came back. Our own workers retry with exponential backoff, but they also all reconnect the instant a database returns. We now add jitter to reconnects, cap concurrent retry work by queue depth, and practise "dependency comes back after 2 hours" in game days, not just "dependency goes away".

Automatic failover needs a speed limit

Health-check-driven failover removed healthy capacity. In our Kubernetes clusters, the equivalent is a readiness probe that fails when a downstream dependency is slow, so every pod goes unready at once. We changed readiness probes to check only the pod's own health and moved dependency checks to alerting. Where automation removes capacity, it now has a floor (for example a PodDisruptionBudget or a minimum healthy percentage).

Map dependencies, including the hidden regional ones

Our API ran in eu-west-1, yet logins failed because a third-party identity feature we used called an STS endpoint in us-east-1. We now keep a dependency map per service with the region of every endpoint, and we pin SDKs to regional STS endpoints. A quick audit of where your code actually connects is worth an afternoon:

# Distinct AWS endpoints contacted by a service over 24h, from VPC DNS query logs
grep -hoE '[a-z0-9.-]+\.amazonaws\.com' route53-resolver-queries-*.log | sort | uniq -c | sort -rn | head -30

Decide region strategy with real numbers

Multi-region active-active is expensive and complex. For our pipeline we chose a cheaper middle ground: DynamoDB global tables with a second region we can switch writes to with a runbook, tested quarterly. For the customer API we accepted single-region risk but removed every cross-region dependency we found. Write the decision down with the expected recovery time so it is not re-argued during the next incident.

Status pages and support channels can share your failure

The AWS Support console itself was impaired for part of the event. Our incident runbook now lists an out-of-band channel to reach our account team and a status page hosted outside our primary provider.

Key takeaways

• Per AWS's summary, a check-then-act race in DNS automation produced an empty record for the DynamoDB regional endpoint, and recovery load caused a longer EC2 and NLB tail.
• Re-check preconditions at write time; never let clean-up delete what is live.
• Design and test recovery paths for the thundering herd that follows a dependency's return.
• Put rate limits on automation that removes capacity.
• Audit hidden regional dependencies such as STS and identity endpoints.
• Write down your region strategy and its expected recovery time before you need it.

References

• AWS: Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region — https://aws.amazon.com/message/101925/
• Amazon DynamoDB Developer Guide: Condition expressions — https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html
• Amazon DynamoDB Developer Guide: Global tables — https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GlobalTables.html
• AWS Well-Architected Framework: Reliability pillar — https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html