On November 18, 2025, a large share of the web returned error pages for several hours because Cloudflare's network could not deliver core traffic. Cloudflare published a detailed postmortem on its blog the same day. We read it as a team because it describes a failure mode that almost every company with generated configuration is exposed to, including ours.

This post has two clearly separated parts. The first summarises what happened, strictly according to Cloudflare's official report; we have not added anything that is not in that report, and where the report gives exact figures we use them. The second part is our own: the lessons we are taking for the systems we run, which are far smaller but built from the same kinds of parts. None of it is about blame. The report itself is unusually candid, and the useful response is to check whether the same chain could happen to us.

What happened, per the official report

The trigger: a database permission change

Cloudflare runs ClickHouse clusters that, among other things, feed data into its Bot Management system. According to the report, engineers were improving the security and reliability of distributed queries so that they would run under the initial user's account, which allows query limits and access grants to be evaluated in a more fine-grained way. As part of that work, at 11:05 UTC a change made access to the underlying tables in a database called r0 explicit, so users could now see the metadata of those tables as well as the tables in the default database.

The query that changed its answer

A job that builds Bot Management's "feature file" read column metadata with a query that filtered on table name but not on database name. The report shows it as:

SELECT name, type FROM system.columns
WHERE table = 'http_requests_features' order by name;

Before the permission change, this returned columns only from the default database. After it, the same query also returned the matching columns from r0. The report describes this as more than doubling the rows in the response. Nothing in the query was wrong syntactically; it was relying on an implicit assumption about what the user could see.

From a bigger file to a crashing proxy

The feature file is regenerated every few minutes and pushed across Cloudflare's whole network so that the bot detection model has current data. The proxy module that consumes it preallocates memory for a maximum number of features. The report gives the numbers: the limit was 200 features, and normal usage was around 60. With duplicated rows, the file exceeded 200. The limit check worked as intended, but the error it returned was not handled gracefully. In FL2, Cloudflare's newer Rust-based proxy, the result was a panic:

thread fl2_worker_thread panicked: called Result::unwrap() on an Err value

Customers on FL2 received HTTP 5xx errors. Customers on the older FL proxy did not see errors, but bot scores were not generated correctly and everything got a score of zero, so rules that block on bot score produced false positives.

Why it looked like an attack

The ClickHouse permission change was rolling out gradually across the cluster. Because the feature file was generated every five minutes on whichever node ran the query, some runs produced a good file and some a bad one. The network flapped between failing and recovering. The report says this pattern, together with Cloudflare's status page going down at the same time (it is hosted entirely off Cloudflare's infrastructure, and the report says the timing was a coincidence), initially led the team to suspect a large DDoS attack. The report states explicitly that the incident was not caused, directly or indirectly, by a cyber attack.

Timeline (UTC, from the report)

Time Event
11:05 Database access control change deployed
11:20 Network begins experiencing significant failures delivering core traffic
11:28 First errors observed on customer HTTP traffic
11:31-11:35 Automated detection fires; incident call created
13:05 Workers KV and Cloudflare Access bypassed to an earlier proxy version, reducing their impact
13:37 Work focuses on rolling back the Bot Management configuration file to a last-known-good version
14:24 Generation and propagation of new feature files stopped; a known good file tested
14:30 Known good file deployed globally; main impact resolved
17:06 All services restored

Affected products

The report lists the core CDN and security services (5xx errors), Turnstile (failed to load), Workers KV (elevated 5xx), the dashboard (logins mostly unavailable because Turnstile and Workers KV were failing, with a second period of elevated latency from a backlog of login attempts and retries after recovery), Email Security (email processing and delivery were unaffected, but a temporary loss of access to an IP reputation source reduced some detections, and some Auto Move actions failed) and Access (authentication failures until the bypass). It also notes significant increases in latency on top of the errors, caused by large amounts of CPU consumed by debugging and observability systems that automatically enhance uncaught errors with additional debugging information.

Remediation Cloudflare committed to

The report lists four follow-ups: hardening ingestion of Cloudflare-generated configuration files in the same way as user-generated input; enabling more global kill switches for features; eliminating the ability for core dumps and other error reports to overwhelm system resources; and reviewing failure modes for error conditions across all core proxy modules. Cloudflare called it its worst outage since 2019.

Lessons for our own systems

Everything below is our interpretation for our own stack. It is not a claim about how Cloudflare should operate.

Generated config is code, and it needs a release process

Most teams have careful rollouts for binaries and almost none for data files their own systems generate: feature lists, routing tables, allow-lists, model weights, pricing tables, translation bundles. We found three such files in our platform that are regenerated by a cron job and copied to every node with no staged rollout. After reading the report we put all three behind the same pipeline as config changes:

  • Validate the generated artifact against a schema and hard bounds before publishing (row count, file size, unique keys).
  • Compare it with the previous version and stop if it changed by more than an expected amount. A file that doubles overnight is a signal, even if every row is valid.
  • Roll it out to a canary slice first and watch error rates before going global.
  • Keep the last-known-good version on every node and make "revert to it" a single command.

A check as simple as this would catch the class of problem described in the report, in our context:

def validate_features(new: list[dict], previous: list[dict], hard_limit: int = 200) -> None:
    names = [f["name"] for f in new]
    if len(names) != len(set(names)):
        raise ValueError("duplicate feature names in generated file")
    if len(new) > hard_limit:
        raise ValueError(f"{len(new)} features exceeds hard limit {hard_limit}")
    if previous and len(new) > 1.5 * len(previous):
        raise ValueError(f"feature count jumped from {len(previous)} to {len(new)}")

Queries should state their assumptions

The metadata query was correct for the permissions it was written against. When permissions changed, its meaning changed. We audited our own queries against information_schema and catalog tables and added explicit schema filters everywhere. The general rule: if a query's correctness depends on what the current user can see, say so in the query rather than in the grant.

Limits must fail soft where they can

Hard limits are good engineering; preallocation and bounds are how you keep a hot path fast. The question is what happens when the bound is hit. For the components we run, we now ask for each limit: can we keep serving with the old data, a truncated list, or the feature disabled, instead of crashing? In Rust that means treating unwrap() on input-derived values in request paths as a code review flag. In other languages it means the equivalent: an unchecked exception in a worker that handles every request.

Kill switches need to exist before the incident

The report mentions enabling more global kill switches. We had feature flags for user-facing features but none for internal enrichment steps like our fraud score or geo lookup. We added flags so that any optional enrichment can be turned off globally in one action, and we test them during game days. A kill switch that has never been flipped in production is a hypothesis.

Watch for misleading coincidences

The flapping behaviour and the unrelated status page outage pointed responders toward an attack. Our takeaway is procedural: early in an incident, explicitly list recent changes (deploys, config, permissions, data jobs) next to the external hypotheses, and assign someone to rule them in or out. Permission and data changes are easy to forget because they do not show up on deploy dashboards.

Observability must not make things worse

The report notes that debugging and observability systems consumed significant CPU during the incident. We checked our own error paths: our exception reporter captured full request context and local variables on every error, synchronously. During an error storm that is exactly the wrong behaviour. We now sample error reports above a threshold and send them asynchronously with a bounded queue.

Key takeaways

  • Per Cloudflare's report, a permission change made a metadata query return duplicate rows, which more than doubled a generated feature file, which exceeded a 200-feature limit and caused the FL2 proxy to panic.
  • Files your own systems generate deserve schema validation, size and diff checks, canary rollout and instant rollback, just like code.
  • Queries against catalogs and metadata should filter explicitly instead of relying on what the user is allowed to see.
  • Decide in advance how each hard limit fails, and prefer degrading over crashing in request paths.
  • Build and rehearse global kill switches for optional processing steps.
  • Make sure error reporting is sampled and bounded so it cannot amplify an outage.

References