Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Retrieval Evals for Company Memory: Test Recall Before Launch

Learn how to test whether an AI agent recalls the right company fact before launch, using a repeatable retrieval eval that still holds after a model change.

Be Human typographic cover with the title: Retrieval Evals for Company Memory: Test Recall Before Launch
Conceptual illustration created with AI · Be Human

A superintelligent agent retrieval evaluation is a fixed set of real company questions, each paired with the fact the agent must find and the source that holds it, run before launch and again after every change. It answers a question that a fluent answer hides: did the agent retrieve the right company fact from memory, or did it write something plausible from the wrong passage? Because the test is built on facts and their owners rather than on one model, it keeps giving a useful measurement when a stronger model replaces the current one.

Key takeaways

  • Score whether the agent retrieved the governing fact, not whether its answer sounds right.
  • Write each test case with its source and its owner before the agent runs.
  • Separate retrieval failures from answer failures so you know which part to fix.
  • Store the cases outside the model so a model swap is a measured change, not a guess.
  • Re-run the same frozen cases after every change and read every miss with the owner.

What a retrieval eval measures in company memory

Company memory is everything an agent can look up to answer a question: policies, service terms, product documentation, runbooks, and customer history. Retrieval is the step where the agent selects some of that material and places it in front of its answer. A retrieval eval tests that step directly. For each question, it asks whether the passage that governs the answer was among the material the agent actually received, and whether the final answer relied on it.

This is different from grading the answer alone. A fluent answer can be wrong in a way that is hard to notice, and a correct answer can be correct by accident, drawn from a note that happens to match the wording of the question. Measuring retrieval on its own shows where the system breaks: the index, the way documents were split into pieces, the ranking, the instructions that shape the answer, or the source material itself. Each of those needs a different fix, and a prompt edit only helps with one of them.

How to build the case set from facts with owners

Each case needs four fields: the question as a staff member or customer would phrase it, the fact that must govern the answer, the source document and version that holds it, and the person who confirmed that the fact is still current. Teams often skip two more fields that matter. The first is the passages that must not be used, usually superseded versions of a policy. The second is the set of questions the memory should decline or escalate instead of answering.

Write the set with the people who own the processes, not only with the engineer who builds the index. If operations wrote a rule, operations is the authority on what it says. When the eval expects a fact the owner would not stand behind, the eval is wrong, and any agent that passes it is wrong too. Start with the questions where a wrong answer costs the most, then widen the set over time. The mechanics of turning real work into labeled cases are covered in how to build an eval set from real business cases.

A step-by-step plan for the first retrieval eval

Use this sequence for one role before any launch. With the owners available, the first pass is usually a working week of effort, and the steps are the same whatever the size of the company.

  • Name the single role and the memory sources it may read, with one named owner for each source.
  • Collect the questions staff actually ask that role, from tickets, chat logs, or short interviews.
  • Write each case with the question, the governing fact, the source version, and the owner who confirmed it.
  • Add negative cases: superseded passages the agent must not use, and questions it should escalate.
  • Run retrieval only against the current index, and record the passages returned for each question.
  • Score each case against the pass rule the owner agreed to before the run started.
  • Review every miss by hand and label its cause before changing anything.
  • Freeze the case set, store it outside the agent's configuration, and re-run it on every change.

Keep the first run small enough that one person can read every miss. A large list that nobody reads hides the pattern behind the volume, and a small list that is read every week teaches the team more than a large one that is archived.

Score recall apart from the final answer

For each case, record four things: whether the governing passage appeared in the material the agent received, whether the source it cited is the current version, whether the answer matches the fact in that passage, and whether the agent declined when it should have. Keep these as separate results. An agent can retrieve the right passage and still write a wrong answer, and it can write a right answer from a superseded source. Those are different problems with different owners.

Write the pass rule in plain words before the run, for example: the governing passage must appear for every case in the notice-window group, and only the owner of that policy may waive it. Changing the rule after seeing the results turns the eval into a description of what the system already does. When a case is hard to grade by exact match, an automated judge can help, but the judge needs its own checks against human labels, as explained in LLM judges for AI agent evals.

When a case fails, use this order of checks:

  • If the governing passage was never retrieved, check the index, the way documents were split, and the query the agent sent before touching the prompt.
  • If the passage was retrieved but ignored, the answer instructions or the order of the supplied context are the likely cause.
  • If a superseded version was retrieved, archive that source before tuning anything else.
  • If the owner rejects the expected fact, correct the source and the case together, and record the change.

Illustrative scenario: a team runs an agent that answers questions about changing a booking inside a notice window. The new policy document is in the index, but an older handbook still sits beside it. The eval shows that questions using the word reschedule retrieve the handbook, while questions phrased as move my appointment retrieve the new policy. A prompt edit could make the answers look better for one phrasing and hide the gap for the other. The retrieval log points to the duplicate source, which is the actual fix.

Keep the measurement useful after a model change

A stronger model often writes more fluent answers, and that can hide a retrieval gap. It can reason its way to a plausible response from partial context, so the end-to-end score rises while the retrieval miss stays in place. For that reason, run the eval in two modes. Retrieval-only runs should not change when the model changes, because the model is not part of that step. End-to-end runs can change for good or bad reasons, so they need to be read case by case rather than as a single average.

When the model is replaced, re-run both modes on the frozen set, compare each case with its previous result, and review every case whose status changed. Record which model version, index version, and source versions produced each result. If the index is rebuilt or a source is edited, run retrieval first, since that is the earliest point where a change can be seen. The scoring stays the same across runs, so any movement in the results reflects a change in the system rather than a change in the yardstick.

Be Human's managed approach treats this as part of operations rather than a one-time test. An engagement begins by defining one role, its memory sources, their owners, and the approval boundaries around them. The retrieval eval is written with those owners before launch and stored with the agent's operating notes. After launch, recurring evaluations re-run the same cases after integration changes or model updates, permission reviews check which sources the agent can read, and misses are logged where the people who own the facts can see what changed. For a broader view of how this fits into routine upkeep, see a monthly routine for healthy agents.

Decide the pass rule before the first run

Before the first run, agree in writing on three things: which facts count as governing for the role, who is allowed to change them, and who reviews misses each week. Those decisions matter more than the tooling. Teams that skip them tend to adjust the agent until the eval passes, and at that point the eval no longer tells anyone whether company memory is accurate. Make the owners part of the process from the start, and the eval will keep its value as the company, its documents, and its models change. If you are not sure which role should be first, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

How many cases do I need before launch? Start with the questions that carry the most risk for one role rather than aiming for a fixed total. A set small enough for a person to read every miss is more useful than a large set nobody reviews. Add cases after each real incident or staff correction so the set grows with actual work.

Does a retrieval eval replace checks on answer quality? No, it tests a different step. The retrieval eval shows whether the governing fact reached the agent, while answer checks show whether the response is faithful to that passage and whether tone and permissions are right. You need both, and each should have its own pass rule.

Can I use an automated judge to grade the results? You can, particularly for cases where exact matching fails, but the judge must be checked against labels written by people first. Without that check, you are measuring the judge's opinion rather than the company fact. Keep the judge's own errors in the review log, just as you would keep errors from the agent.

What should I do when the model changes? Re-run the frozen case set in both retrieval-only and end-to-end modes, compare each case with its previous result, and review every case whose status moved. Keep the pass rule unchanged so the comparison stays honest. If a case flips, check the index and the source versions before crediting or blaming the new model.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →