The health check that pulled every server at once

transcript

Your database blips. Just for a moment. Your load balancer pulls every single server. Nowhere to send traffic.

A load balancer spreads traffic across your servers. It probes each one every 10 to 30 seconds. Are you healthy? Server 3 stops answering. It's marked unhealthy, and gets nothing.

So what should that check test? A shallow check asks one thing. Is the process up? A deep check tests every dependency. Database, Redis, an external API. Thorough. Sounds safer, right? Then the database refuses connections. The deep check answers 503. Unhealthy.

But here's the catch. That database is shared by every server. So every server's deep check fails. At the same moment. The whole pool is marked unhealthy. Nowhere to send traffic.

The fix: the load balancer gets the shallow check. Traffic keeps flowing. The deep check goes to monitoring and alerting. So you know what broke.

The load balancer needs one answer. Is this server alive? Ask it about every dependency, and one blip empties the pool.

Full breakdown in the lecture.