A new client's first two requests got two rate-limit buckets

transcript

A brand-new client sends its first two requests. Same instant. For a moment, it gets twice its rate limit.

The registry maps each client and endpoint to a limiter. A token bucket, say. Both requests ask for this client's limiter. Both hear: none yet. Request A creates a bucket. Request B creates another, and overwrites A's. Two buckets. Double the limit.

Lock the whole registry on every request? Then everyone waits on everyone. The fix is to look twice.

First, look without the lock. Almost always, it's there. Missing? Take the registry lock. And look again. Another thread may have won. Still missing? Only now create it. Exactly once.

Then each limiter has its own lock. Watch 20 threads fight one bucket. 2,000 attempts on a bucket of 50. No time passing. Exactly 50 allowed. Not 51.

One limiter per key, each with its own lock. Look, lock, look again. Created exactly once.

Full breakdown in the lecture.