212 answers. Perfect faithfulness. All of them wrong.

transcript

212 answers. Every one scored a perfect 1.0 on faithfulness. And every one was wrong. The metric never noticed.

MegaMart cuts the return window for opened electronics. It's 10 days now. But the index still holds the old 30-day rule. 212 answers cite it. Every citation looks right. Nothing goes red.

Faithfulness asks one thing. Is each claim backed by the retrieved text? Nothing more. The text was stale. So the answers matched it perfectly. So faithfulness scores 1.0. Every single time.

Faithful means it matches what was retrieved. Whatever was retrieved. Correct means it matches the real policy. A different question, and a different score.

So score correctness too, against a reference answer. Then 212 wrong answers show up as 212 failures. And a stale document is an ingestion bug. Fix it in the pipeline, not the prompt.

Faithful to the wrong text is still wrong. Faithful isn't correct.

Full breakdown in the lecture.