The disk got slow, and a log line stalled every request

transcript

The disk gets slow. Then every request gets slow. The culprit? A log line.

Rule one of logging: never slow down the code that's logging. So the async sink drops each record on a bounded queue. No waiting. And the request thread returns at once. One worker thread does the writing. The slow disk is its problem.

But here's the catch. The disk stays slow. The queue fills up. Now what does write do?

Block until there's room? Now the request thread waits. A slow disk just became a stuck service. Instead, drop the record. Count it. Return at once. A counted drop shows up in metrics. A blocked request is an outage.

And with room in the queue, nothing's lost. 8 threads log 500 lines each. After close: 4,000 written. None dropped. Close flushes the queue before the worker stops.

Losing a log line is better than an outage caused by logging.

Full breakdown in the lecture.