Same two words. A 45-word review beats a 150-word guide, nearly 3 to 1.

transcript

Two pages match running shoes. One is a 45-word review. One is a 150-word guide. The short one wins. Nearly three to one. Here's why.

Elasticsearch scores with BM25. One: a word that appears more often counts more. With diminishing returns. Ten mentions aren't ten times one. Two: rare words matter more than common ones. Three: a long document shouldn't win just by having more words.

The word the is in 90 percent of documents. Its weight: about 0 point 1. It says nothing. Shoes, in 10 percent: 2 point 3. Running, in 5: 3 point 0. Nike, in just 2 percent: 3 point 9. The rarer the word, the more a match means.

Now the review. 45 words, running 3 times, shoes twice. Score: about 8. The guide is 150 words, three times the average. Each term once. Length normalisation pulls it down. About 2 point 9.

And that diminishing return protects you. A description that repeats waterproof 20 times doesn't get 20 times the score.

Rare words count more. Repeats count less and less. And length alone never buys relevance. That's BM25 working as intended.

Full breakdown in the lecture.