AWS added a few servers to Kinesis, and Kinesis stopped working for most of a day.

Amazon Kinesis carries streams of real-time data, for customers and for other AWS services. Requests first hit a front-end fleet of servers, which routes each one to the right back-end cluster using a map of who owns what, the "shard-map".

To keep that map fresh, each front-end server talks to every other one, and it creates an operating system thread for each of the other servers in the fleet. Add servers, and every server needs more threads.

On 25 November 2020, from 2:44 to 3:47 a.m. PST, AWS added what its summary calls "a relatively small addition of capacity" to that front-end fleet in us-east-1. At 5:15 a.m. the first alarms fired for errors reading and writing records. The errors were so varied that cause and side effect were hard to tell apart. Only at 9:39 a.m. did they confirm it: the new capacity had pushed every server in the fleet past the maximum number of threads allowed by an operating system setting. The shard-maps never finished building, so servers couldn't route requests at all.

The new capacity had already been removed, so the fix was a restart. But it had to be slow. A server building its map competes with the requests it serves, and the fleet is "many thousands of servers", so AWS could only bring back a few hundred per hour. Kinesis was fully normal at 10:23 p.m. PST.

Services built on Kinesis fell with it, including Cognito logins and CloudWatch metrics. Even AWS's status page was slow to update, because its posting tool used Cognito.

AWS's fixes: bigger servers (so fewer threads each), alarms on thread use, and splitting the front-end into isolated cells.

Why it matters: Anything that grows with the size of the fleet (threads, connections, gossip) has a ceiling. Know where it is, alarm before you reach it, and split big fleets into cells so one limit can't hit every server at once.