More documents made the model worse than none at all

transcript

You give the model 20 documents. The answer is in there. And it does worse than if you'd given it nothing. Because where you put the answer matters.

A 2023 study moved the answer around the prompt. At the start or the end, models found it. Buried in the middle, GPT 3 point 5 Turbo dropped more than 20 percent. Worst case? Below its closed-book score of 56 point 1 percent.

But more documents must help, right? Past 20, it gained about 1 and a half percent. For a much longer prompt.

Now the bill. 20 chunks of 800 tokens is 16,000 a turn. Over 6 turns, 96,000. Four times a whole conversation. Re-rank down to 8 chunks of 400. Just 3,200 a turn.

So re-rank, and keep about 8. Put the best evidence first, where models look hardest. Those were 2023 models. Measure yours, and stop when more stops helping.

More context isn't more knowledge. Send less. Put the best first.

Full breakdown in the lecture.