A short note on every chunk cut search failures by 67 percent

transcript

On Anthropic's test, plain search missed 5 point 7 percent. A short note on each chunk cuts that by 67 percent.

Cut a chunk from its document. The company's revenue grew by 3 percent. Which company? Which quarter? The chunk doesn't say. So an LLM reads the whole document first. It prepends a 50 to 100 token note. Then you index it.

Anthropic measured it. Plain chunks failed 5 point 7 percent. Contextual embeddings: 3 point 7. Add contextual BM25: 2 point 9. Add a re-ranker: 1 point 9. 67 percent fewer failures.

Here's the twist. That's an LLM call for every chunk. So MegaMart repriced it with Claude Haiku 5 point 5.

All of MegaMart's documents: about 31 dollars. The prompt cache holds each one. Three million ticket summaries: about 310 dollars. Ten times more.

Policies and contracts lose meaning when cut. Worth it. Ticket summaries are written to stand alone. Skip them. Add context only where chunks lose it.

Full breakdown in the lecture.