DynamoDB splits every table into partitions spread over many storage servers. Which partitions a server holds is its membership, and a central metadata service keeps track of it. Storage servers check in regularly to confirm their membership, and again after any network trouble. If the answer doesn't arrive within a set time, the server retries and takes itself out of service until it gets one.
In the months before, customers had rapidly adopted a new feature, Global Secondary Indexes, which let you look up a table by another key. Each index has its own partitions, so a big table's share of the membership lists could quickly double or triple. Answering them took longer and longer, creeping up to the time limit. AWS said it lacked detailed monitoring of membership size, and enough metadata capacity for these heavier requests.
On Sunday 20 September 2015 at 2:19 a.m. PDT, a brief network disruption hit some storage servers in US-East. Many asked for their membership at the same time. The big answers slowed down further and ran past the time limit. Those servers dropped out and retried. The retries loaded the metadata service so much that healthy servers doing routine check-ins timed out too, and dropped out as well. By 2:37 a.m., DynamoDB's error rate was far beyond anything in three years, settling at about 55%.
AWS couldn't even add capacity: the metadata service was too overloaded to accept the admin requests. At 5:06 a.m. the team paused requests to it, which cut the retries, then added capacity and let the servers back in. By 7:10 a.m. errors were low enough for most customers.
The fixes: more metadata capacity, monitoring of membership size, fewer check-ins with more time allowed, and many metadata services, each serving part of the fleet.
- Network blip at 2:19 a.m. PDT; error rate about 55% by 2:37 a.m.
- Highest DynamoDB error rate in 3 years
- Requests to the metadata service paused at 5:06 a.m.
- Error rates low enough for most customers at 7:10 a.m.
Why it matters: A timeout is a contract with a number in it. Watch how close you're running to it, because when load pushes you over, the retries can take down everything else too.