One mistyped command rebooted Joyent's entire data center

transcript

One typo. Every server in a data center, rebooted. All at once. No confirmation. No delay.

27 May 2014. A Joyent operator is upgrading some new servers. The tool can run a command on one server. Or on many. The reboot command is mistyped. It says all servers. The tool doesn't question it. 20:13 UTC. Everything goes down.

Now every server wakes up. Each asks the boot server what to load. All the requests arrive together. Normal throttles slow every machine down. So the team rips the throttles out. Machines start coming back.

80 percent of customer instances were back within 32 minutes. Over 90 percent within 59. The unluckiest waited 149 minutes.

But here's the catch. Some older machines had a buggy network driver. It fails about 10 percent of boots. Reboot them again. And the API's stateful parts couldn't rebuild themselves. Humans had to.

That part was partly on purpose. Joyent would rather stay down than come up split brain and lose data.

Joyent's fix: strict validation. No rebooting every server at once. If one command can hit everything, make all hard to type.

Full story in the article.