p99 jumped from 200 ms to 2.3 seconds, with zero extra errors

transcript

You're paged. p99 latency is 2 point 3 seconds. Normally 200 milliseconds. But the error rate is normal. Nothing is failing. It's just slow.

Start with the dashboard. p99 at 2 point 3 seconds. It started 10 minutes ago. And only on the orders endpoint. That's your scope.

Open a slow trace. The order service takes 2 point 2 seconds. Inside it, a database query, get products: 2 seconds. That's where.

Now the logs, for that same trace ID. That query returned 500 rows. That's a lot.

Is that new? The metrics say yes. Rows returned at p99 used to be 10. Now 500, matching the spike. Right after a new customer was onboarded.

Mitigate now: a query timeout, or a limit. Long term: pagination, or optimise the query. And prevent it: alert on abnormal row counts. Load test with realistic orders.

The trace said where. The logs said what. The metrics said when. Together, they told you why.

Full breakdown in the lecture.