One customer's valid setting broke 85% of Fastly

transcript

One customer changed one setting. A perfectly valid one. And 85 percent of Fastly's network started returning errors.

12 May 2021. Fastly rolls out new software. Testing misses a bug. For almost four weeks, it sits quiet. 8 June, early. A customer pushes a configuration change. 09:47 UTC. Global disruption.

Fastly is a CDN. Each customer controls its site with settings. The bug needed one specific configuration, under specific circumstances. Here's the catch. The change was valid. Normal. Allowed. One customer's legal setting. Errors across 85 percent of the network.

Now, the smart move. They didn't fix the bug first. They found that setting, isolated it, and switched it off.

Monitoring spotted it in one minute. Within 49 minutes, 95 percent of the network was normal. The permanent bug fix? It only started rolling out at 17:25.

Fastly's own words: we should have anticipated it. On a shared platform, the most dangerous input can be perfectly legal. Test configs, not just code. And contain each tenant.

Full breakdown in the lecture.