How to test retrieval quality in an AI knowledge base

Test retrieval quality by checking whether the system finds the right evidence for representative questions, not just whether the final answer sounds correct. An answer can be accidentally right while citing an irrelevant source, and a capable model can fail when the needed passage never reaches it.

Build a question-to-source set

Choose a small collection of business questions with known authoritative answers. For each, record the relevant document, section and any qualifying conditions. Include questions whose wording differs from the source, because real users rarely repeat a document heading exactly.

Add questions with no answer in the collection. The system should distinguish missing evidence from a negative answer.

Inspect the retrieval stage separately

• Did the correct document appear among the retrieved results?
• Was the relevant passage included with enough surrounding context?
• Were current versions preferred over retired material?
• Were access restrictions applied for this requester?
• Did the answer cite evidence that actually supports its claim?

A policy sentence without its exception may be misleading even if the document title is correct. Review what text the model saw, not only which file was returned.

Use a deliberate distractor

Create a synthetic document with similar vocabulary but a different scope. For example, a partner onboarding rule should not answer an employee onboarding question. If the system selects by word overlap alone, improve metadata filters, source structure or retrieval logic.

Keep test expectations independent of the generated answer. Letting the same system decide what evidence counts as correct can hide the failure you are trying to measure.

Track errors by cause

Separate absent source, poor ranking, incomplete passage, stale version and unsupported synthesis. Each category suggests a different fix. Rewriting the prompt will not recover a document that was never indexed. After a change, rerun the same set plus new examples from ordinary use. This makes retrieval maintenance a repeatable practice rather than an occasional demo.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.