AWS added a few servers. Kinesis broke for most of a day.

transcript

AWS added servers to Kinesis. And Kinesis stopped working. Not for minutes. Until late that night.

Every Kinesis request hits a front-end server first. It needs a map of where each stream lives. The shard-map. To build it, each server opens one thread for every other server.

2:44 a.m. AWS adds a small batch of front-end servers. 5:15 a.m. Alarms. Reads and writes are failing. The errors are all over the place. Nearly four and a half hours of hunting.

Here's the twist. Nothing was broken. A limit was hit. Every server now needed more threads than the operating system allowed. So no server could finish its map. None could route a request.

The fix was a restart. But the fleet is many thousands of servers. Building a map steals power from serving requests. So only a few hundred per hour. Kinesis was fully normal at 10:23 p.m.

And services built on Kinesis fell with it. Cognito logins. CloudWatch metrics. Even AWS's status page was slow. Its posting tool used Cognito.

AWS's fixes: bigger servers, so fewer threads each, and isolated cells. If it grows with your fleet, it has a ceiling. Know where it is.

Full story in the article.