On the night of 19 October 2025, the name dynamodb.us-east-1.amazonaws.com
stopped resolving to anything. DynamoDB itself was healthy: its servers were up
and its data was intact. But nothing could find it, and a surprising amount of
AWS needs to find DynamoDB before it can do its own job. The DNS record was
fixed in under three hours. The region took fourteen and a half to recover,
because the systems that depend on DynamoDB had piled up work while it was gone
and could not get through it when it came back.
It is the best recent example of two ideas interviewers love: a race condition in a control plane, and the gap between "the root cause is fixed" and "the system has recovered".
At a glance
- When: 19–20 October 2025, 11:48 PM to 2:20 PM PDT
- Where: AWS us-east-1 (Northern Virginia)
- Trigger: A latent race in DynamoDB's DNS automation wrote an empty record for the regional endpoint
- Amplifier: EC2's host manager kept its leases in DynamoDB, then collapsed under its own retry backlog
- Root cause fixed: 2 h 37 min
- Fully recovered: 14 h 32 min
What users saw
First, errors from anything that called DynamoDB in us-east-1: "could not resolve host" rather than a throttle or a 500. That included AWS's own services, so IAM console sign-in, STS tokens, Redshift queries and Support Center cases failed alongside customer applications. Global tables replicas in us-east-1 fell behind on replication.
Once DynamoDB came back, a second wave started. Already-running EC2 instances were fine, but new ones would not launch ("insufficient capacity"), and the ones that did launch could take hours to get working networking. Then Network Load Balancers began dropping healthy targets, so services that had been fine all night started returning connection errors in the morning. Lambda, ECS, EKS, Fargate and Amazon Connect all felt some part of this.
Timeline
All times PDT, starting the evening of 19 October.
The bar is the lesson. The phase most people remember, the DNS record going empty, is the thin slice at the start. Most of the outage happened after the cause was fixed.
The root cause
DynamoDB serves each region from a large fleet of load balancers, and DNS is how clients are spread across them. Hundreds of thousands of records per region are kept up to date by automation split into two parts:
- The Planner watches load balancer health and capacity and periodically writes a new plan: which load balancers, with which weights, should sit behind each endpoint. Plans are numbered generations.
- The Enactor applies the latest plan to Route 53. There are three, one per Availability Zone, running independently so that losing a zone does not stop DNS updates. Each one applies a plan inside a Route 53 transaction, so a single endpoint never ends up half old and half new.
Before applying, an Enactor checks once that its plan is newer than the one already live. It then walks every endpoint, retrying any it is blocked on. When it finishes, it deletes plans that are many generations old, since nothing should need them any more.
That design has three safe pieces that together were unsafe:
- Enactor A picked up a plan and ran into unusual delays, retrying endpoint after endpoint.
- Meanwhile the Planner kept producing newer plans, and Enactor B applied one of them quickly across every endpoint.
- A finally reached the regional endpoint and applied its plan, which was now many generations stale. Its "is this newer?" check had passed when it started, a long time earlier, so nothing stopped it.
- B, having finished, ran its clean-up. A's plan was far older than B's, so B deleted it, and with it every address for the regional endpoint.
The record was now empty, and the system's own bookkeeping was inconsistent (the live plan no longer existed), which blocked any Enactor from applying a new one. Automation designed to recover from failures could not recover from this one. A human had to.
This is a time-of-check to time-of-use bug. The check was correct when it ran; the world changed before the write. The fix is to make the check and the write one atomic step: a conditional write that only succeeds if the live generation is still lower than mine, the same way you would guard a row in a database with a version number.
Why it spread
The DNS fault lasted under three hours. Three things stretched it to fourteen.
A hidden dependency. EC2's DropletWorkflow Manager (DWFM) looks after the physical hosts, called droplets, that run instances. It holds a lease on each droplet and renews it with a state check every few minutes. Those checks use DynamoDB. With DynamoDB unreachable, leases quietly expired all night. Running instances were unaffected, but a droplet without a lease cannot take a new instance, so launches failed with "insufficient capacity" even though the capacity was sitting there.
Congestive collapse. When DynamoDB came back at 2:25 AM, DWFM tried to re-establish every lease at once. There were so many that the work took longer than its own timeouts. Timed-out work was queued to retry, the queue grew, and the system spent its time on retries that would also time out. It was busy and getting nowhere. There was no runbook for this; engineers had to throttle incoming work and restart DWFM hosts to clear the queues.
A health check that removes capacity. The component that pushes network configuration to new instances was now hours behind. Load balancer health checks saw newly launched instances without working networking, marked them unhealthy, then healthy, then unhealthy again. The flapping overloaded the health check system, which slowed checks down further, which triggered automatic failover that removed whole zones of load balancer nodes from DNS. A safety mechanism designed to route around bad capacity was removing good capacity. Engineers switched automatic failover off at 9:36 AM.
How it was fixed
- DNS: operators corrected the DynamoDB DNS state by hand, restoring all records at 2:25 AM.
- EC2: incoming work was throttled and DWFM hosts restarted selectively, so leases could be renewed at a rate the system could finish. Throttles were then relaxed gradually through the late morning.
- Networking: load on the propagation system was reduced until it caught up at 10:36 AM.
- Load balancers: automatic health check failover was disabled until EC2 was healthy, then turned back on.
AWS's follow-ups map one to one onto the amplifiers: the DynamoDB DNS automation was switched off worldwide until the race is fixed and a guard against applying a wrong plan is added; load balancers get a limit on how much capacity a health check failover can remove; DWFM gets a test suite for its recovery path; and EC2's propagation throttling will scale with queue depth.
Design lessons
Make check-and-write atomic. A freshness check that runs once at the start of a long operation is stale by the end. Use a conditional write, a version number or a lease with fencing tokens, so the write itself fails if the world moved. This is the conditional write on the DynamoDB page and the fencing token on the ZooKeeper page.
Clean-up is a write, and a dangerous one. Garbage collection that deletes "old" state must check that the state is not in use right now, not merely that it is old.
Recovery is its own workload. After a long outage, every client returns at once with a backlog. Design the recovery path for that load: admission control, queue-depth-based throttling and jittered retries, as in retries, backoff and dead letter queues and backpressure. A system that cannot drain its retry queue faster than it fills is in congestive collapse; load shedding is how you get out.
Bound what automation can remove. Health checks that pull capacity need a velocity limit, or a flapping signal empties the pool. See health checks: if more than a set fraction of targets look unhealthy at once, the checker is more likely wrong than the targets.
Know your dependency graph, including the control plane. EC2 depending on DynamoDB is invisible from the outside. A DynamoDB table is a fine place for leases, but then the lease holder's recovery plan has to include "the table was unreachable for three hours".
The interview angle
This incident turns up as a follow-up to any design with leader election, leases or a config pipeline: "What happens if two workers both think they are current?"
Two workers apply routing config to the same DNS record. How do you stop an old config overwriting a new one?
Each worker checks the version before it starts applying, and skips anything older than what's live.
That check is exactly the one that failed at AWS. A worker can pass it, stall for minutes, and then write. Nothing ties the check to the write.
Two workers apply routing config to the same DNS record. How do you stop an old config overwriting a new one?
I'd make the version check part of the write. Each config has a monotonically increasing generation, and the write is conditional: set this record to generation 42 only if the stored generation is below 42. A stalled worker's write then fails instead of landing. Same idea as a fencing token on a lock.
Clean-up gets the same rule: never delete the generation that is currently live, however old it is. And I'd keep the last known-good config somewhere the workers can always fall back to, so an empty or invalid record is never something they publish.
And if it goes wrong anyway?
Then recovery is the next risk. Everything that depends on the record has been retrying, so I'd bring traffic back behind admission control and watch queue depth, not just error rate.
Names the race, makes the guard atomic, covers deletion as a write, and notices that recovery has its own load.
Sources
- AWS, Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (us-east-1) Region, October 2025.
Retold in our own words from the companies' public postmortems listed above. System Design Atlas is not affiliated with or endorsed by any company named on this page. Spotted a mistake? Email tulparapps@gmail.com.