One customer's export report took down 500 other customers

transcript

Monday, 9 AM. One customer runs one report. 500 other customers can't use your product.

API latency: 200 milliseconds, to 15,000. Database CPU: 98 percent. Error rate: from 0 point 1 percent, to 45.

The top query: every event for Acme Corp, since 2020. Running for 847 seconds. 2.3 billion rows scanned. Still going. Someone at Acme ran an export all data report. Nobody attacked you.

But you have per-tenant rate limits! A tenant can send just 5 requests a minute. Each can scan billions of rows. Rate limits count requests. Not what each one costs.

So check the query before it runs. EXPLAIN estimates its rows, without running it. Over 10 million rows? Only Enterprise may run that. Anyone else gets an error. Add filters, or upgrade. Everything else runs, with the plan's timeout as the backstop.

Rate limits catch volume. One cheap-looking request can still melt the database. So guard the cost, before it touches shared CPU.

Full breakdown in the lecture.