The FAA paused every US departure after files were deleted while fixing a database sync

transcript

One morning, every single departure in the United States was paused. It started with contractors fixing a database sync.

Pilots check NOTAMs before they fly. Closed runways. Birds. Volcanic ash. Late on 10 January 2023, the FAA's NOTAM system went unreliable.

Engineers try switching to a backup database. There are three. All night, a phone hotline keeps users updated. 7:15 a.m. Eastern. The morning rush. Every US departure is paused. By about 9 a.m., it is lifted. The damage is done.

The cause? Contractors were fixing the sync between primary and backup. They unintentionally deleted files. But here's the catch. Changes sync across the primary and the backups. So the backup was hit by the same deletion.

More than 1,300 flights cancelled. More than 9,500 delayed.

The FAA's fix says it all. A one-hour delay before syncing. Now bad data can't reach the backup right away. And two people must be present.

A replica that copies every change instantly copies every mistake too. A real backup needs a delay, a snapshot, or independence.

Full story in the article.