Roblox was down for 73 hours because its service directory choked.

Roblox runs on its own hardware in its own data centers: more than 18,000 servers and 170,000 containers. Its services find each other through Consul, a tool from HashiCorp that keeps a shared directory of which service lives where, runs health checks and holds a small key-value store. In Roblox's words, when Consul is unhealthy, "the system cannot schedule new containers or retrieve production secrets".

On 28 October 2021, Consul began to slow down. Engineers tried replacing hardware, reducing load, resetting the cluster. Nothing held. Worse, Roblox's monitoring itself depended on Consul, so when Consul was sick, the data needed to debug it was missing too.

Roblox and HashiCorp engineers eventually traced it to two problems stacked on each other. First, a newer Consul feature called streaming. Roblox had been switching it on gradually for months and, on 27 October, one day before the outage, turned it on for the service that routes traffic. Under heavy load, streaming caused heavy contention inside Consul. On 30 October they turned streaming off everywhere, and the typical (median) time for a write fell to 300 ms.

Second, the small database Consul uses to save its data, BoltDB, had a pathological pattern: for every write of 16 kB or less, it was also writing out a 7.8 MB list of free space to disk.

Fixing Consul was only half the job. Caches were cold, and Roblox did not want a flood of players to knock the system over again. So it let players in gradually, steering them with DNS and opening access in steps of roughly 10%. At 16:45 (Pacific time) on Sunday 31 October, 73 hours after the start, every player had access again. Roblox said no user data was lost.

Why it matters: Monitoring must not depend on the thing it watches, and recovery from a total outage needs a plan: let traffic back in slowly.