Fastly is a content delivery network (CDN): a worldwide set of servers that keep copies of its customers' websites close to readers, so pages load fast. Each customer controls how Fastly handles its site through configuration settings.
On 12 May 2021, Fastly began rolling out new software. It contained a bug that could be triggered by one specific customer configuration, under specific circumstances. Fastly's testing didn't catch it.
Early on 8 June, a customer pushed a configuration change. Fastly stresses that the change was valid: a normal, allowed setting. But it contained exactly the conditions that set off the bug, and 85% of Fastly's network started returning errors.
Here is the timeline from Fastly's own post (UTC):
- 09:47 global disruption begins
- 09:48 Fastly's monitoring spots it
- 09:58 a public status post goes up
- 10:27 engineers identify the customer configuration involved
- 10:36 services start to recover
- 11:00 most services recovered
- 12:35 incident mitigated
- 17:25 the permanent bug fix starts rolling out
The first move was to stop the trigger, not fix the bug: Fastly found the configuration, isolated it and disabled it. Within 49 minutes, 95% of its network was operating as normal. The permanent fix for the bug itself only started rolling out later that day.
Fastly published its summary the same day. It promised to find out why its quality checks had missed the bug, and said: "Even though there were specific conditions that triggered this outage, we should have anticipated it."
- Bug shipped 12 May; triggered 8 June, about four weeks later
- 85% of the network returned errors
- 1 minute from start to detection
- 49 minutes until 95% of the network was normal
Why it matters: A shared platform's most dangerous input can be a perfectly legal one. Test configurations as well as code, and build ways to contain one tenant's settings so they can't take everyone else down.