One line of regex knocked Cloudflare offline

transcript

One line of regex took down Cloudflare's whole network. And the rule wasn't even supposed to block anything.

Cloudflare sits in front of websites. Its firewall rules spot attacks. 13:42 UTC. A new rule ships in simulate mode. Log matches, block nothing. But even a watching rule has to run. Every CPU core serving web traffic maxes out. Worldwide. Visitors get 502 errors. About 80 percent of traffic, lost.

The villain? This tiny piece of the pattern. Look at it. Dot star parts, side by side.

A request comes in. The regex engine tries one way to match. It fails. Steps back. Tries another way. With dot stars side by side, it goes round again. And again. The longer the input, the more the tries explode. CPU gone.

But why everywhere at once? Normal code at Cloudflare ships in stages. WAF rules didn't. They went global in seconds, to fight new attacks fast. And the safeguard capping regex CPU? Removed by mistake, weeks earlier.

14:07. They hit global terminate. A kill switch for one component, worldwide. 14:09. Traffic is back. 27 minutes. Cloudflare's first global outage in six years.

Cloudflare promised staged rule rollouts. A fast path only for active attacks. And a regex engine with run-time guarantees. Because config and rules are code too. Ship them like code.

Full breakdown in the lecture.