A million new products, invisible to search for 3 hours

transcript

A vendor uploads a million new products. Shoppers search for them. Search says they don't exist. For nearly 3 hours.

A million new products, arriving at once. The normal change capture path indexes about 100 products a second. A million over 100 a second is 10,000 seconds. Nearly 3 hours.

So push harder? Here's the catch. Faster, but never at the cost of three things. Don't overwhelm the cluster. Don't starve the live updates. And don't spike search latency for shoppers.

So bulk work gets its own lane. A separate bulk indexer reads PostgreSQL directly. Batches of 1,000. 4 parallel workers. A token bucket capped at 5,000 documents a second. Meanwhile, live changes keep flowing through their own pipeline.

And a bulk job is tracked. After every batch, it saves a checkpoint: the last processed ID. A 6 hour job crashes at 70 percent. Start over? No. It resumes after the checkpoint. A few documents sent twice? Harmless. Indexing by ID just overwrites.

The live pipeline keeps search fresh. The bulk lane loads fast, under a hard rate limit. Keep them apart. A million-row import must never starve the stock update a shopper relies on.

Full breakdown in the lecture.