A cleanup script deleted 883 Atlassian sites in 23 minutes

transcript

A cleanup script deleted 883 customer sites. In 23 minutes. And it was only meant to remove one old app.

Atlassian folded an old app into Jira Service Management. Time to remove it. One team asked for the deletion. Another team ran it. But the list didn't hold app IDs. It held whole site IDs. And the delete API took either kind. Give it a site, it deletes the site.

But here's the catch. This script passed every check. The review looked at the endpoint. Not what the IDs pointed to. Staging? Those IDs didn't even exist there. A first run on 30 sites worked fine. Its IDs predated the mix-up.

07:38 UTC. The second run starts. It looks like a standard delete. So monitoring sees nothing. At 07:46, a customer opens the first ticket. 08:01. Done. 883 sites, 775 customers, deleted. Even contact details were gone. So customers couldn't file tickets normally.

Atlassian could restore one site. Hundreds at once? Never built. About 70 steps each. Restoring everyone that way would take three weeks. So they built a faster way. Some customers still waited up to 14 days.

The twist? No customer lost more than five minutes of data. The data survived. Getting hundreds back at once was the hard part.

Atlassian's first fix? Soft deletes, everywhere. Deleted data is kept for a while. Make destructive APIs ask: site or app? And practise mass restores. Because a delete you can't undo needs more than a code review.

Full breakdown in the lecture.