Before match night, Hotstar attacked itself with a fake crowd

transcript

Before match night, Hotstar attacked its own platform. With a fake crowd. So big, it swamped CDN edges and hit other customers.

Where does your system break? You want to know before the match does. So Hotstar ran game days. Full rehearsals, with match-shaped traffic, at the real platform.

By 2019, an in-house tool called Project HULK. More than 3,000 load machines. 36 virtual CPUs each. 108,000 vCPUs in all. 216 terabytes of memory, sending out 200 gigabits a second.

At first, the crowd fired from one region. It overwhelmed CDN edges, and affected other customers. So HULK spread across eight AWS regions. And logs from real matches shaped the fake traffic.

On top came chaos engineering. A notification storm. A tsunami of viewers, then a sudden drop. An edge failure. Even 50 extra milliseconds on one content API.

But here's the twist. One test found a system burning over 70 percent of its compute. On a feature Hotstar wasn't even using. Tuning it freed massive headroom. Zero extra servers.

Match night, 10 July 2019. The real peak: 25.3 million. The team was ready for 50 million. About twice that.

Rehearse with traffic shaped like the real night. Then break things on purpose. Find your breaking point before the crowd finds it for you.

Full breakdown in the lecture.