No evidence isn't “made it up”: three verdicts for an LLM judge

You want to know whether your assistant makes things up. You hand past conversations to another model, the judge, and ask it. The problem: every time your knowledge base lacks the fact, the judge answers "yes, it made it up". It ends up punishing the bot for the gaps in your base.

Two questions that look like one

"Did it make it up?" bundles two different questions:

1. Does the answer state something the base doesn't back?
2. Is the base at hand to check it?

If the second is no, the first has no answer: without the base, the judge only measures whether something sounds true. The verdict needs three boxes:

Verdict	What it means
Didn't make it up	Every concrete fact in the answer is in the base fragments
Made it up	It states a price, opening hours or a policy that isn't in the fragments, and says which
No evidence	The search returned nothing, or failed: it can't be judged

The path

1. Search again with the customer's question. The judge gets what that search returns, not the whole base.
2. With the same search the chat uses. Same threshold, same mode, same scopes. If the evaluator searches differently from the bot, it measures something else.
3. Without fragments, the judge is never asked whether it made something up. It gets a different instruction and grades what it can: whether it resolved, whether it handed off correctly, the tone.
4. The third state is set by code, not by the judge. Even if the model answers "made it up", without fragments the result is "no evidence". It doesn't depend on the model obeying.

Two more details. A "made it up" that doesn't say which statement lacks support doesn't count as a verdict. And since today's base may not be the one from the day of the chat, the window is short (up to 14 days) and the result says when the base was queried.

Test it in red

A rule like this breaks without anyone noticing: someone simplifies and treats any search response as evidence. The tests stay green if none of them looks at that case.

The way to know is to break it on purpose. In CognitionDesk's quality evaluator, forcing every search to count as evidence makes the two tests guarding the rule fail: empty search and failed search. With the real code, all 19 pass.

What to take away

If you measure hallucinations with a judge: give it the fragments found by the bot's own search, and a box for "nothing to check against". When the search didn't fail, a recurring "no evidence" isn't a bot error: it's a question your base has nothing for.

Start by reviewing the “made it up” verdicts in your current evaluator and check how many had fragments behind them. This is the criterion behind CognitionDesk's weekly quality check: if you want to apply it to your conversations, get in touch.