On 18 November 2025, a large share of the web started returning Cloudflare's own error pages. So much of the web sits behind Cloudflare that for a few hours "the internet is down" was close to literally true. The trigger was not an attack, a bad code deploy or hardware. It was a permissions change on an internal database, which changed the output of a query, which doubled the size of a configuration file, which crashed the proxy that reads it. Cloudflare called it its worst outage since 2019.
It is the cleanest example there is of a lesson every config pipeline needs: a file your own system generated is still input, and input can be wrong.
At a glance
- When: 18 November 2025, 11:20 to 17:06 UTC
- Where: Cloudflare's global network: CDN, security, Workers KV, Access, Turnstile, dashboard
- Trigger: A ClickHouse permissions change made a metadata query return duplicate rows
- Amplifier: The resulting file was pushed to every machine within minutes, and the proxy crashed on a hard limit
- Core traffic restored: 14:30 UTC, about three hours in
- Fully recovered: 17:06 UTC
What users saw
Visitors to sites behind Cloudflare got HTTP 5xx error pages instead of the site. Turnstile, Cloudflare's CAPTCHA alternative, failed to load, which meant logging in to anything protected by it failed too, including Cloudflare's own dashboard. Workers KV returned errors, Cloudflare Access refused logins to applications behind it, and the email security product briefly lost a reputation data source.
The odd part was the rhythm. For the first couple of hours the errors came and went every few minutes: sites would recover, then fail again. To the people inside Cloudflare that looked like a large DDoS attack, not a bug, and by coincidence Cloudflare's status page, which is hosted entirely outside its own network, went down at the same time.
Timeline
The root cause
Cloudflare's Bot Management assigns each request a bot score using a machine learning model. The model's inputs, called features, are listed in a feature file that is regenerated every five minutes from a ClickHouse query and pushed to every proxy machine on the network. Speed is the point: bots change tactics quickly, so the file must change quickly too.
The query read column names for one table from ClickHouse's system.columns
metadata, filtering by table name but not by database. That was harmless while
users could only see one database's metadata. The permissions change, made for
a good reason (running distributed queries as each user's own account), also let
those users see the same tables in a second database, r0, where the
underlying shards live. The query now returned every column twice.
The proxy's bot module had a limit of 200 features, against roughly 60 in use,
because it preallocates memory for them rather than allocating per request. The
doubled file exceeded the limit. The new Rust proxy, FL2, treated "too many
features" as an error and then called unwrap() on that error, which panics the
worker thread. Every request through it failed.
Customers still on the older proxy did not get errors. Their proxy loaded the bad file and produced a bot score of zero for every request, so anyone with a rule that blocks low-scoring traffic started blocking real users. Same bad input, two failure modes, depending on which version of the code read it.
Why it flickered. The permissions change was rolling out across the ClickHouse cluster gradually. Each five-minute run of the query landed on some node: an updated node produced a bad file, an old one a good file. Whichever file was generated went everywhere within minutes, so the whole network flipped between broken and working. Once every node had the change, every file was bad and the outage stopped flickering and stayed down.
Why it spread
One file, every machine, minutes apart. The feature file is built for fast global propagation, which is the right call for bot defence and exactly the wrong property for a bad file. There was no staged rollout to catch it on a handful of machines first.
A hard limit with no graceful path. The limit itself was sensible. What turned it into an outage was the reaction to crossing it: crash the thread rather than keep the last good file, drop the extra features, or skip bot scoring for a while. Bot Management is one module among many in the request path, but its failure took the whole request down with it.
Shared infrastructure, hidden coupling. Turnstile, Workers KV and Access all depend on the same core proxy. When it failed, they failed, and because the dashboard login uses Turnstile, customers could not log in to see what was happening.
The wrong theory cost time. Flickering errors, the recent Aisuru DDoS attacks and a status page that went down at the same moment all pointed to an attacker. The team also initially read Workers KV errors as the primary symptom. Neither was the cause.
Observability made it worse. The systems that attach debugging detail to uncaught errors used enough CPU under the error flood to add latency across the network.
How it was fixed
- At 13:05, Workers KV and Access were routed around the failing proxy, which reduced their impact while the cause was still unknown.
- Once the feature file was identified, generation and propagation of new files were stopped at 14:24.
- A known-good older file was put into the distribution queue and the core proxy was restarted. By 14:30 most traffic was flowing again.
- The remaining services that had got stuck were restarted over the next two and a half hours, and the dashboard's control plane was scaled up to absorb the backlog of logins.
Cloudflare's commitments: treat internally generated configuration with the same suspicion as user input, add more global kill switches for features, stop error reporting and core dumps from being able to overwhelm a machine, and review how every core proxy module fails.
Design lessons
Validate generated config like user input. The file should have been checked against its limits before it was published, not when it was loaded. A size or row-count check in the generator, plus a rule that a file wildly different from the previous one is held for review, would have stopped this at the source.
Fail to the last known good, not to a crash. A consumer that cannot load a new config should keep running on the previous one and raise an alarm. That is graceful degradation applied to configuration, and it is the difference between a stale bot model and a global outage.
Stage every rollout, including data. Code goes through canaries; config often goes straight to everyone because it "isn't code". A feature file, a routing table or a rules list can break a fleet just as well. Roll it out in rings, the same way as a deploy in deployment strategies, even if each ring only bakes for a minute.
Isolate optional modules from the request path. Bot scoring is valuable, but a request can be served without it. Treat optional work like a dependency with a circuit breaker and bulkhead: if it fails, skip it and serve the request.
Queries should say what they mean. "All columns of this table" was really "all columns of this table in this database". Under-specified queries are correct only as long as the environment around them stays the same.
Keep the status page and the debugging tools off the failing system. Cloudflare did this for its status page, which is why its simultaneous failure was a coincidence rather than a dependency. The CDN page covers why the edge is shared by everything behind it.
The interview angle
Any design with a CDN, an API gateway, a feature flag service or a rules engine invites the follow-up: "How does configuration reach your fleet, and what stops a bad config taking everything down?"
Your edge servers load a rules file that's regenerated every few minutes. What could go wrong?
We'd version it and push it out through a pub/sub channel, so every server picks it up within seconds.
Describes how to spread the file fast, which is exactly the property that made this outage global. Nothing about what happens when the file is wrong.
Your edge servers load a rules file that's regenerated every few minutes. What could go wrong?
The file is the most dangerous thing we deploy, because it changes more often than code. Three guards. The generator validates the output before publishing: schema, size, row counts, and a diff against the last file, holding anything that changes by more than a set amount. The rollout goes to a canary slice first, and only proceeds if their error rate holds. And each server, if a new file fails to load, keeps the previous one and alerts rather than crashing.
Then a global kill switch, so we can turn the feature off everywhere in one action if all three guards miss.
Why not just validate on load?
We should, but by then the bad file is already on every machine. Validating at the source stops it once; validating on load stops it on each server.
Moves validation upstream, stages the rollout, defines a safe failure on the consumer and adds a kill switch. That is Cloudflare's own remediation list.
Sources
- Cloudflare, Cloudflare outage on November 18, 2025, November 2025.
Retold in our own words from the companies' public postmortems listed above. System Design Atlas is not affiliated with or endorsed by any company named on this page. Spotted a mistake? Email tulparapps@gmail.com.