9 min read ·
A config change is a deploy
On November 18, 2025, a large piece of the internet went down because a file got too big. It wasn't a code deploy. Somebody made a sensible change to database permissions, and before the day was out Cloudflare was writing the words "worst outage since 2019."
It was the third giant outage in a month. Between October 19 and November 18, AWS, Microsoft Azure and Cloudflare each took a large slice of the web offline for hours. I read all three postmortems. None of them begins with a bad code deploy. Each one begins with a change that nobody treated like one.
A deploy, as most teams use the word, is when you ship new code. It gets ceremony. Somebody reads the diff. Tests run. The new version goes to a few servers first, a canary, and if errors climb there's a button that rolls it back.
Configuration is everything that changes how the code behaves without changing the code. Feature flags. DNS records. Firewall rules. The list of signals a model reads. The text of a prompt. Config usually skips the ceremony, because it's "just data". And it ships fast on purpose. The whole reason you move a setting out of code is so you can change it without a release.
That speed is the trap.
AWS first, because it's the shortest. Late on October 19, Pacific time, the automation that manages DNS records for DynamoDB in AWS's Northern Virginia region left an empty record for the service's main endpoint. DNS is the internet's phone book. An empty entry means nobody can find the number. Two copies of the automation raced, and an old plan overwrote a newer one. I wrote about that kind of race in load-bearing latency. The point here is simpler: AWS's summary names no deploy. Automation edited config the way it did every day, and the event ran for about fourteen and a half hours.
Then Azure, on October 29. Azure Front Door is the global layer that sits in front of websites and routes their traffic. Two more terms. The control plane is the part of a system that accepts your settings. The data plane is the part that serves traffic with them. According to Microsoft's incident review, the outage began with an incompatible configuration change. A sequence of valid customer changes, made across two different versions of the control plane, produced a description of those settings that hit a bug nobody knew was in the data plane, and the servers that answer requests crashed. From 15:41 UTC until 00:05 the next morning, sites behind Front Door saw timeouts and failed DNS lookups.
Azure had safeguards. The rollout was staged. Health checks ran, and they passed, because the crash happened later, in a background process. After about five minutes of healthy signals, the change moved on to the rest of the fleet. By then the bad config had also been saved as the "last known good" snapshot, the version you roll back to.
The rollback button pointed at the bomb.
Microsoft then froze customer configuration changes on Front Door until November 5. Three weeks earlier, on October 9, Front Door had a smaller outage whose trigger, in Microsoft's lessons-learned post, was "a manual cleanup of stuck tenant metadata" that "bypassed our configuration protection layer."
A cleanup. The most harmless word in operations.
Cloudflare's outage is the one worth knowing step by step, because every step is ordinary.
Cloudflare gives every request a score for how likely it is to come from a bot. The model behind that score reads a feature file, a list of the signals it should look at. A job rebuilds the file every five minutes from a ClickHouse database and pushes it to every machine on the network, so bot detection keeps up with attackers.
At 11:05 UTC, Cloudflare started to roll out a permissions change on that database cluster. It made some existing access explicit so limits could be enforced more precisely. Good hygiene.
The query that builds the feature file asked for the columns of a table by name and never said which database. Before the change, it saw one copy of those columns. After it, it saw the same columns twice. The file doubled in size, past a hard limit of 200 features, on a system that normally used about 60. The proxy, the program on each Cloudflare server that handles requests, reserves memory up front for 200 features and no more, so it refused the file and crashed with called Result::unwrap() on an Err value.
Then it got strange. The permissions change was still rolling out across the cluster, so some machines had it and some didn't. The job ran every five minutes, and each run might land on either kind. Bad file, good file, bad file. The network fell over, came back, and fell over again. That flapping looked so much like an attack that Cloudflare's team suspected one at first.
Core traffic was mostly flowing again by 14:30, after the team stopped generating new files and put a known good one into the distribution queue by hand.
The easy villain is that unwrap. In Rust, unwrap() means "I'm sure this worked, and if it didn't, crash." Plenty of people online called it the bug.
I don't buy it. The proxy had a rule, at most 200 features. The input broke the rule, and the proxy refused to carry on with it. You can argue it should have kept the old file and kept serving, and Cloudflare now argues that too. I'm less sure: a crash you can see beats a fallback you can't. But the decision that hurt came earlier. A file built by one job, from one half-migrated cluster, went to the whole fleet in minutes, and nothing in between asked whether it looked anything like the file from five minutes before.
Cloudflare's fix list says it in one line. They will harden ingestion of Cloudflare-generated configuration files "in the same way we would for user-generated input."
A company built to absorb the worst traffic on the internet has decided its own files deserve the suspicion it gives to strangers.
Seventeen days later, Cloudflare went down again. December 5, about 25 minutes, roughly 28% of its HTTP traffic. They were rushing out protection for a critical security hole in React, a popular web framework. The main change went through their gradual deployment system. A second change, switching off an internal testing tool that didn't support the new setting, went through the global configuration system, which in their own words "does not perform gradual rollouts." It hit a bug in the older proxy that had sat undetected for years.
The careful change took the careful path. The quick "just turn it off" took the fast one.
That's the pattern. The fast path exists for emergencies, and emergencies are exactly when people are tired and in a hurry.
Line it up against what a code deploy gets.
A diff somebody reads. A config diff is one line in a settings file or one database row, and when a machine generates it, nobody reads it at all.
A canary. Cloudflare's config system, Quicksilver, got a change to 90% of servers within seconds. Great when you're blocking an attack. A liability every other day.
Health gates and rollback. Azure had both, which is why its story scares me most. A gate is only as good as its timing, and "last known good" only as good as your definition of good.
Cloudflare and Microsoft have since landed on the same answer. Cloudflare's resilience plan named "controlled rollouts of our configuration, just as we already do for software releases" its most important workstream. This May it reported that "in most cases" its internal config changes "no longer reach our network instantly and are instead rolled out progressively with real-time health monitoring," and that services now fall back to the last known good config where possible. They call it "fail stale".
Microsoft wrote that "any cleanup must flow through the same guarded stages and health checks as standard configuration changes," and described a process it calls the Food Taster: an isolated worker on every server that loads new configuration first and serves no traffic, so if the config is poison, only the taster dies.
It took two of the biggest infrastructure companies on earth a very bad autumn to arrive at a sentence your team could adopt this week.
Every team has a path like this. I've spent the last year putting models into operations work, and in my world the config is the prompt. Change one sentence in a system prompt, the standing instructions a model reads before every run, and every run after it behaves differently. It often lives in a text box on a dashboard, goes live on save, and has never seen a review. Same for the feature-flag console, the rules file a script rebuilds every night, the threshold somebody tweaks at 6 pm on a Friday.
So ask the deploy questions about it. Who reads the diff? Does it go everywhere at once, or to a few machines first? What does the code that reads it do with a file twice the usual size? Is your last known good actually known to be good, or just the last thing that didn't crash in the first minute?
If a file can take your service down, it deserves the same ceremony as the code that reads it.
And stop writing "configuration error" in postmortems as if it were the weather. It was a deploy that skipped the checks.