Slack's job queue filled up, and then it couldn't even empty itself

transcript

Slack's job queue filled up. Then it couldn't even empty itself. And when the queue stops, sending messages can stop too.

Every message post makes jobs. Push notifications. URL unfurls. Calendar reminders. Billing. 1 point 4 billion jobs on the busiest days.

The villain: database contention. Every job slows down. Jobs pile up in Redis. Until it hits its memory limit. Here's the twist. Dequeuing a job needed a bit of free memory. So the queue stayed locked up. Even after the database was fine.

The fix: put Kafka in front of Redis. Jobs land on disk first. Kept for two days. A relay feeds Redis only as fast as workers can take. Workers slow? Turn the dial down. Redis memory stays flat.

And the code that enqueues and runs jobs? It didn't change. Slack called it the minimum viable change.

A backlog should become lag. Not an outage. A log absorbs. A queue schedules.

Full breakdown in the lecture.