The FAA had three backup databases. They didn't save the day.

transcript

The FAA had three backup databases. One morning, every US departure was paused anyway.

Pilots check NOTAMs before they fly. Closed runways. Birds. Volcanic ash. Late on 10 January 2023, the FAA's NOTAM system went unreliable.

Engineers try switching to a backup database. There are three. By early morning it seems restored. But formatting problems persist. 7:15 a.m. Eastern. The morning rush. Every US departure is paused. By about 9 a.m., it is lifted. The damage is done.

The cause? Contractors were fixing the sync between primary and backup. They unintentionally deleted files. Then the FAA's written answers to senators add a twist. The deletion didn't break the sync. It was already broken. Before the outage, sync hadn't been reaching all the backups.

More than 1,300 flights cancelled. More than 9,500 delayed.

The FAA's fix? A one-hour delay before syncing to the backup. So bad data can't reach the backup right away. And no one works on the database alone.

A backup that has quietly stopped syncing can't take over. And one that copies every change instantly copies every mistake. So add a delay or a snapshot. And check that it's current.

Full story in the article.