Roblox was down for 73 hours because its service directory choked

transcript

About 50 million people play Roblox every day. For 73 hours, they couldn't. The culprit? The one tool everything else depends on.

Roblox runs its own hardware. Over 18,000 servers. 170,000 containers. They find each other through Consul. A shared service directory. Consul unhealthy? No new containers. No production secrets. And the monitoring? It ran on Consul too. Remember that.

28 October 2021. Consul starts to slow down. New hardware. Less load. A full cluster reset. Nothing holds. And the data to debug it? Missing. Monitoring needed Consul.

Roblox and HashiCorp engineers found two problems. Stacked. One: a newer feature called streaming. Turned on for traffic routing the day before. Under heavy load, it caused heavy contention inside Consul.

Two: BoltDB, the small database where Consul saves its data. Take a write of 16 kilobytes or less. Along with it, BoltDB wrote a 7 point 8 megabyte free-space list. Every time.

30 October. Streaming off everywhere. Median writes drop to 300 milliseconds. But here's the catch. The caches are cold. Let everyone in at once? A flood could knock it over again. So Roblox steered players with DNS. Opening access about 10 percent at a time.

31 October, 16:45 Pacific. Every player is back. Roblox says no data was lost. Monitoring must not depend on the thing it watches. And after a total outage? Let traffic back in slowly.

Full breakdown in the lecture.