Google's sign-in died for 47 minutes because a quota system read zero

transcript

One wrong number. Zero. And every Google sign-in failed for 47 minutes.

14 December 2020. You open Gmail. Docs. Drive. Errors. Everywhere. Anything that needs you signed in is down.

Google sign-in runs through one system. The User ID Service. October 2020. It's moved to a new quota system. But pieces of the old one stay. They report its usage as zero. A grace period hides it. Until it expires.

The automation sees a service using nothing. So it cuts the quota. The account database's leader can't write anymore. Reads go out of date. And for security, stale data is refused.

But wait. Google had safety checks for quota changes. None covered zero reported load for a single service.

03:48. Engineers are paged. 04:08. Cause found. They switch quota enforcement off. 04:33. Errors back to normal. 47 minutes, end to end.

Automation is only as safe as the number it reads. When a busy service suddenly uses nothing, that's a bug. Not a saving.

Full story in the article.