Outage case study10 min
Strongly consistent coordination store for leader election and membership.Core featuresLinearizable writes via quorumEphemeral nodes tied to a sessionWatches push change notificationsSmall data only, not a databaseetcd and Consul are the alternativesWhen to useElecting exactly one leader per shardCluster membership and failure detectionAssigning partitions to workersStaying up and correct when parts of the system fail.Metrics, logs and traces that show what a system is doing.Keeping hot data in fast storage to skip slow lookups.
PremiumRoblox: 73 hours on one Consul cluster
October 2021. Two bugs in the one Consul cluster everything depended on took Roblox offline for three days.
Premium
This case study is on the Premium plan
Readers with Lifetime Access read it in full. Already bought it? Sign in from the top bar.
What’s inside
- At a glance
- What users saw
- Timeline
- The root cause
- Why it spread
- How it was fixed
- Design lessons
- The interview angle
- Sources