~learn.

Fighting a Christmas Day attack, a cache rule showed Steam users other people's account pages.

Fighting a Christmas Day attack, a new caching config cached Steam's logged-in pages. About 34,000 users' pages may have been seen.

Valve (Steam) · 2015 · 2 min read · a true story, sources below

Early on Christmas morning 2015 (Pacific time), in the middle of Steam's holiday sale, the Steam Store came under a denial-of-service attack. Attacks on Steam are routine and usually invisible to users. This one wasn't: traffic to the store rose 2000% above the sale's already high average.

To protect Steam's servers, caching rules managed by Valve's web caching partner were deployed. A cache keeps a copy of a page at the network's edge and hands that copy to the next person who asks, so the real servers do less work. That's perfect for a public store page that looks the same for everyone.

Then a second wave of the attack came, and a second caching configuration went out. This one had an error: it cached pages for logged-in users. A page built for one person, with their details, was stored and served to whoever asked next.

What people saw varied. Some got the store's front page in the wrong language. Others saw someone else's account page. Between 11:50 and 13:20 PST on 25 December, requests for about 34,000 users may have been seen by others. Depending on the page, that included a billing address, purchase history, email address, the last four digits of a Steam Guard phone number and the last two digits of a credit card. Valve said full card numbers and passwords were not included, and nobody could log in or buy anything as another user.

Once the error was found, Valve shut the Steam Store down. It stayed down until every caching configuration had been reviewed, the fixed ones were on all partner servers, and all cached data on the edge servers had been purged.

The numbers
  • Store traffic up 2000% over the Steam Sale average during the attack
  • About 34,000 users' pages possibly shown to others
  • 90 minutes of exposure: 11:50 to 13:20 PST on 25 December

Why it matters: A cache must know whose page it is holding. Never cache personalised responses at a shared edge, and treat emergency config changes as risky as any deploy.

Sources

the idea behind it · W04 · DAY 01Caching Patternsread the lesson →