One server's clock ran 5 seconds fast. The audit log told the wrong story.

transcript

Your audit log has every event. Nothing missing. Nothing edited. And it still tells investigators the wrong story. Because of one clock.

Server A writes an event at 10 o'clock and 2 seconds. Then another, one second later. Server B's clock runs 5 seconds fast. Its event is stamped 10:00:07. It really happened at 10:00:02.

In reality, B's action came before A's second one. Sorted by timestamp, B lands after it. For an investigation, that's the wrong story.

The fix: don't trust the producer's clock alone. Keep its event time, for reference. And have the consumer stamp when it received the event. Investigators sort by received time, for a consistent order. Store the drift too.

And sync every server with NTP. Keep drift under 100 milliseconds. Alert when any clock drifts past a second. Clock sync is infrastructure. Monitor it.

A producer's timestamp is a claim. Its clock might be wrong. Keep the claim. But order by a clock you control.

Full breakdown in the lecture.