Fortnite hit 3.4 million players at once, and a 100 ms cache timeout blocked sign-ins

transcript

3 point 4 million people playing Fortnite. At the same time. Then a 100 millisecond timeout blocked sign-ins.

February 2018. Fortnite hits a peak far beyond Epic's wildest expectations. 3 point 4 million concurrent players. Six incidents in two days. One of them starts with a cache.

Every sign-in call goes through one proxy. Nginx. Its shortcut: answer token checks from a cache. Everything else passes straight to the app.

But here's the catch. Sunday, 18:30 UTC. The cache starts failing. Nginx waits on every cache call. 100 milliseconds, then timeout.

The cache is drowning in connections. Nginx runs out of free workers. It can't answer anything. Not even the load balancer's health checks. So the load balancer pulls every node. Full downtime.

Players already in a match? Only moderately hit. Signing in? Mostly blocked. The launcher signed players out with an error. Until 20:20.

The twist? The Java app had its own overload protection. It never helped. The traffic couldn't get through to it.

Epic's fix. Send everything but token checks straight to the app. Add much more cache capacity. Then remove the pair entirely. Because one stuck helper can fail the health checks for everyone.

Full story in the article.