99.8% success. The codes were four minutes late.
transcript
Your dashboard says 99.8 percent success. Support is on fire. Sign-in codes are taking four minutes. Users can't sign in.
Five past seven. Push success: 99.8 percent. Errors are flat. Every graph is green. Rohan's code? Four minutes late. And he isn't the only one.
Here's the catch. Success rates only count attempts. A message stuck in a queue hasn't failed. Not yet. So a four-minute queue looks perfect. Until it times out.
And the worker's timer? Picked up to sent: 200 milliseconds. The user's clock starts at creation. Four minutes. Same message, two very different numbers.
Graph the oldest message's age in each lane. And p99 latency from creation time, per class. Either one jumps within a minute.
The usual cause? A campaign sharing the OTPs' lane, workers or quota. The runbook's first move: pause the campaign.
Success rates can't see a queue. Measure the user's clock. Not the worker's.
Full breakdown in the lecture.