Cloudflare sits in front of its customers' websites and filters their traffic. Part of that filter is a web application firewall (WAF): a list of rules that spot attacks in incoming requests. Many of those rules are regular expressions, tiny pattern-matching programs.
On 2 July 2019 at 13:42 UTC, Cloudflare deployed a new WAF rule to catch cross-site scripting
attacks. It went out in "simulate" mode, meant to log matches and block nothing. But even in simulate
mode a rule has to run, and this one contained a pattern with the piece .*(?:.*=.*).
That piece triggers catastrophic backtracking. The regex engine tries one way to match, fails, steps
back and tries another; with .* parts side by side, the number of tries explodes as input grows.
Cloudflare's post says the rule exhausted the CPU "on every CPU core that handles HTTP/HTTPS traffic on
the Cloudflare network worldwide". Visitors to Cloudflare sites got 502 error pages. Cloudflare lost
about 80% of its traffic.
Why everywhere at once? Normal code releases at Cloudflare go out in stages. WAF rules didn't: they were pushed globally in seconds, because, as the post says, Cloudflare needs to "react as quickly as possible to new attacks". A safeguard that limited regex CPU use had also been removed by mistake during a refactor weeks earlier.
At 14:07 they used a "global terminate", a kill switch that disables one component worldwide. By 14:09 traffic was back. Cloudflare said it was its first global outage in six years. It promised to move to a regex engine with run-time guarantees (re2 or Rust's), and to roll out rule changes in stages like other software, keeping a fast path only for active attacks.
- 27 minutes of outage, 13:42 to 14:09 UTC
- About 80% of traffic lost
- CPUs serving HTTP/HTTPS near 100%, on servers across the whole network
- First global outage in six years
Why it matters: "Config" and "rules" are code too. Anything that can reach every server in seconds needs the same staged rollout, limits and kill switch as a software release.