Almost everything at Google that needs you signed in depends on one system: the User ID Service. It holds a unique ID for every account and handles the credentials behind login tokens and cookies. Its data lives in a distributed database where one "leader" coordinates writes. For security, the service rejects requests when it sees that its data is out of date.
Google uses automated tools to give each service a quota: how much of a shared resource, like storage, it's allowed to use. In October 2020, as part of a migration, the User ID Service was registered with a new quota system. But parts of the old system were left in place, and they reported the service's usage as 0.
Nothing happened at first, because quota cuts had a grace period. When it expired, the automation did what it was built to do: it saw a service using nothing and lowered its quota. Google's safety checks didn't cover the case of zero reported load for a single service, so none of them stopped it.
On 14 December 2020, the quota for the account database dropped. The leader could no longer write. Soon most reads were out of date, so the service did what it was designed to do and refused them. From 03:46 to 04:33 US/Pacific, credential issuance and account lookups failed for all Google user accounts. Google served errors on "virtually all authenticated traffic": Gmail, Calendar, Docs, Drive, the Cloud Console and Cloud APIs. Google's own internal tools were hit too, which slowed its status updates.
Alerts paged engineers at 03:48. They found the cause by 04:08, turned off quota enforcement in one data center at 04:22, then everywhere at 04:27. Errors were back to normal at 04:33.
- 47 minutes of failed sign-ins, 03:46 to 04:33 US/Pacific
- Usage reported as 0 by the leftover quota system
- Engineers paged at 03:48; root cause found at 04:08
- About 75% of BigQuery streaming requests failed during the incident
Why it matters: Automation that acts on a metric is only as safe as the metric. Treat "this service suddenly uses nothing" as a likely bug, not a reason to take its resources away.