Joyent ran a public cloud. For routine jobs (upgrades, provisioning, maintenance) its data centers had tooling that could run a command remotely on one server, or on many. It was mostly used by automated processes, but operators used it too.
On 27 May 2014, an operator was upgrading some new capacity in the us-east-1 data center. The command to reboot the selected new servers "was mis-typed, and instead specified all servers in the data center". The tool didn't have enough input validation to stop it without extra steps, so it sent a reboot to every server in us-east-1 at once, without delay. At 20:13 UTC, all of its API systems and customer instances went down together.
Then everything tried to boot at the same moment. Servers start by asking a boot server over the network for what to load, and every server's request arrived at once. The boot system's normal throttles were now slowing every machine down, so the team removed them. Most instances came back over the next 20 to 30 minutes.
Some stragglers were older machines whose network card driver has a known bug: it fails to get a network address on boot about 10% of the time; they needed another reboot. The API took longer still: its stateful parts are built to survive a couple of failures, but with every one down at once, they couldn't rebuild themselves and needed manual recovery. Joyent notes this was partly by design: it would rather systems stay down than come up "split brain" and lose data. The API was back at 21:30 UTC.
Joyent's fixes: stricter validation that won't let all servers, or all control plane servers, be rebooted at once, and fewer "full power" tools for routine tasks.
- Every server in us-east-1 rebooted at once, at 20:13 UTC
- 80% of customer instances back within 32 minutes, over 90% within 59
- Longest instance downtime 149 minutes; API down 1 hour 17 minutes
- About 10% boot failure rate on the older network card driver
Why it matters: The scariest tools are the ones that can touch everything. Make "all" hard to type: confirm big blast radii, cap how many machines one command can hit, and keep power tools out of routine work.