3 AM, Black Friday, search is melting. The fix was 24 hours ago.

transcript

3 AM, Black Friday. Your phone rings. Search is melting. The real fix? It needed to happen 24 hours ago.

50,000 queries a second. Five times normal. Elasticsearch CPU at 95 percent. p99 latency: 2.5 seconds. The target is 200 milliseconds.

The rule of thumb: one core per about 150 simple queries a second. 50,000 divided by 150. About 333 cores. At 16 per node, about 22 data nodes. Normally you run 6. At peak, CPU sets the size.

So add 16 nodes now? New nodes have to take shards over. Relocating shards adds load, and takes time. Exactly when you can least afford it.

So the plan pre-scales. A day before: double the data nodes. Add a replica. At the start: the full peak. 22 data nodes, 10 coordinators. After: scale down slowly, draining nodes first. Halfway at 6 hours, normal at 24.

Still bad? Degrade on purpose. Warning level: spell check and personalisation off. Critical: no facets, no highlighting, simpler queries, 20 results. What never goes? Search itself. Slow search beats no search.

It costs more. About 17,000 dollars a month at peak scale, against 4,760. But reactive scaling is too slow. The fix was a day earlier.

Full breakdown in the lecture.