A network blip pushed DynamoDB's errors to 55%

transcript

A brief network blip. Then more than half of DynamoDB requests failed. The villain was a timeout. The answers had quietly grown right up to it.

DynamoDB storage servers ask a metadata service: which partitions are mine? The answer is a membership list. Get it in time, and you serve traffic. Too slow? The server retries, and takes itself out of service.

Then customers rush to a new feature. Global Secondary Indexes. Each index brings its own partitions. A big table's share can double or triple. Answers creep up to the time limit. AWS isn't watching list size closely.

Sunday, 2:19 a.m. A brief network blip. Many servers ask for their lists at the same time. Answers run past the limit. They retry. Now even healthy servers time out and drop out. 2:37 a.m. About 55 percent errors. The worst in three years.

But here's the catch. AWS tries to add capacity. It fails. The metadata service is too overloaded to accept the admin requests.

5:06 a.m. They pause requests to the metadata service. The retries drop. Now capacity goes in. 7:10 a.m. Errors are low enough for most customers.

AWS now monitors membership size, and gives check-ins more time. Know how close you run to every timeout. Before a blip finds out for you.

Full story in the article.