In 2021, Atlassian folded a standalone app, "Insight – Asset Management", into Jira Service Management. Engineers used an existing script to remove the old app from customer sites. A site is a customer's container for products like Jira and Confluence.
Two things went wrong. The team that asked for the deletion and the team that ran it miscommunicated: instead of the IDs of the app, the list held the IDs of the entire sites. And the deletion API accepted either kind of ID and assumed the input was correct. Give it a site ID, and it deleted the site, with no warning asking which was meant.
The script had been peer-reviewed, but the review focused on which endpoint was called and how, not on what the IDs pointed to. It was tested in staging, where those IDs didn't exist. A first production run on 30 sites worked fine, because its IDs had been gathered before the mix-up.
The second run started at 07:38 UTC on 5 April 2022 and finished at 08:01: 883 sites, belonging to 775 customers, deleted. Monitoring saw nothing, because sites were deleted through a standard workflow. A customer opened the first support ticket at 07:46. Some customer contact details were deleted too, so affected customers couldn't file tickets the usual way.
Atlassian could restore a single site, but had never built a way to restore hundreds at once. Its first method took about 70 steps per site. With it, restoring everyone would have taken three weeks, so on 9 April it proposed a faster one that re-used the old site IDs. The first customers were back on 8 April; all sites by 18 April. No customer lost more than five minutes of data.
Atlassian's first fix: universal "soft deletes". Deleted data is kept for a retention period, so it can be recovered quickly, and only then removed for good.
- 883 sites, 775 customers, deleted in 23 minutes
- Up to 14 days without access for some customers
- About 70 steps per site in the first restore method
- No customer lost more than five minutes of data
Why it matters: A delete that can't be undone needs more than a code review. Check what an ID points to, make destructive APIs ask "site or app?", and practise restoring many customers at once.