Memory Recall Metrics for Superintelligence Readiness: A Practical Guide

An AI agent's memory is working when it returns the right stored fact for a given question and the agent uses that fact correctly. The way to check this is to measure recall against a fixed set of questions with known correct answers, then track five signals: whether the approved fact came back, whether outdated or unrelated memories came back, whether conflicts were resolved, whether the answer points to a checkable source, and how these scores move after each change. Together, these measures show whether the memory layer is compounding or simply growing.

Key takeaways

• Measure recall against a fixed test set of questions with known correct answers, not against impressions from daily use.
• Track whether the approved fact was retrieved, whether stale or unrelated items were retrieved, and whether the answer cites a source staff can open.
• Log each miss by type (missing, stale, conflicting or misfiled) so the cause is visible, not hidden inside one score.
• Re-run the same test set after every approved memory change, so the trend shows whether the layer is improving.
• Give one named owner the authority to approve changes to expected answers, and review the results on a fixed schedule.

What does it mean for an agent to recall the right fact?

Recall has two steps. The memory layer returns a relevant item, and the agent uses that item correctly in its answer. Either step can fail on its own. An agent can retrieve the approved policy and still answer from a general guess, or it can answer confidently from a retired note that was never removed. A recall metric has to look at the retrieved context, not only the final text, because a fluent sentence can hide a wrong source.

For each test question, separate three kinds of memory items. The expected fact is the approved answer that should come back. Distractors are related items that are wrong, outdated or about a different customer, site or product, and they should not come back. The absence case is a question the company has never documented, where the correct behavior is to say so or escalate. Teams that skip the absence case tend to reward agents for inventing answers, because the invented answer looks like success.

Which five metrics should leaders track?

Each metric answers one question a leader can act on.

• Expected-fact recall: in how many test questions did the retrieved context include the approved fact? A low result usually points to capture, storage or indexing problems.
• Retrieval precision: of the items returned, how many were relevant to the question? A low result suggests the store is too broad or that the context is crowded with material that does not help.
• Stale-fact exposure: how often was a superseded or expired item retrieved? Any occurrence in a sensitive area deserves a review before the next change.
• Conflict handling: when two memories disagree, did the agent use the one the written rules name as current? This metric is meaningless until a precedence rule exists.
• Source attribution: does the answer point to a record that a staff member can open and check?

Report each metric as a count of passing questions out of the total in the test set, not as a bare proportion. A count keeps the size of the test set visible, so a small set cannot produce a misleadingly clean picture. For a closer look at the retrieval step on its own, see How to test retrieval quality in an AI knowledge base.

How do you build a fixed recall test set?

A test set is useful only if it stays stable long enough to show change. Build it in this order.

• Pull the questions staff actually ask from inboxes, tickets, call notes and handoff messages, rather than inventing tidy examples.
• For each question, write the approved answer and name the document or record it comes from, so the expected fact can be checked by someone other than the author.
• Add deliberate traps: one question whose old answer was retired, one where two sources disagree, and one the company has never documented.
• Mark which questions the agent must escalate instead of answering, and write down who receives the escalation.
• Assign one owner who can approve changes to expected answers, and freeze the set between scheduled reviews so results stay comparable.
• Run the full set before and after each approved memory change, and store the results next to the change record.

The set does not need to be large. It needs to cover the decisions the agent supports and to be small enough that the owner can review it. Add questions when real work produces a kind of question the set does not yet cover.

How do you read a drop in recall?

A falling score is a prompt to look at a specific cause, and the metric breakdown tells you where to look.

• If the approved fact is missing from the retrieved context, check whether it was stored, indexed, and retrievable with the wording used in the question. Capture and indexing are the usual causes.
• If a retired fact returns, check its status and expiry. The item was probably never marked as superseded. Stale context explains how to stop this.
• If the right fact is retrieved but the answer uses the wrong one, check the precedence rule. An agent cannot follow a rule nobody wrote down. Conflicting memories describes how to set one.
• If precision falls while recall holds, retrieval is pulling too much. Narrow the scope of what is loaded, rather than adding more material.
• If the agent answers an undocumented question with confidence, the absence case is failing. Add an explicit escalation rule and add that question to the test set.

Record the cause next to each failed question. Over several reviews, the causes tell you whether the weak point is capture, retention, precedence or scope, which is far more useful for planning than the single score.

Illustrative scenario: a replaced policy document

Illustrative scenario: a hypothetical professional services firm uses an agent to answer staff questions about travel reimbursement. The test set includes a question about the approved mileage rule. After the finance team replaces the policy document with a new version, expected-fact recall for that question falls, and stale-fact exposure rises. The old version remains indexed next to the new one, so the agent sometimes quotes the retired rule. The fix does not involve retraining anything. The owner marks the old version as retired, re-runs the same test set, and sees both measures return to their earlier levels. The change record shows the cause, the fix and the two runs side by side, which is the evidence the owner needs for the next review.

How does Be Human handle recall measurement in managed operations?

In Be Human's managed model, recall measurement begins in the first engagement, when the agent's role, its memory categories and its success criterion are defined together. That keeps the test set tied to a real decision rather than to a general impression of quality. During managed operations, the test set is re-run on a schedule and after each approved memory change, and results go to a named owner. Permission review checks that the agent does not retrieve records its role should not see, and incident response covers cases where a recall failure exposed information it should not have. Changes to expected answers are approved by the owner, not made by the agent itself.

Is the memory layer compounding over time?

A single score says little. Compounding means each accepted change leaves the store with fewer unresolved misses and no growth in clutter. Watch three lines over time: expected-fact recall, stale-fact exposure, and the amount of material loaded for each answer. If recall rises while the load keeps growing, the layer is accumulating. If recall rises while the load stays flat or falls, the layer is compounding. Context budgets for AI agents covers how to decide what to load and what to skip.

Keep every run in the same structure so that runs can be compared: the question, the retrieved items, the answer, the pass or fail result, the failure type and the identifier of the change being tested. The OpenTelemetry concepts for traces and metrics describe a shared way to structure this kind of observability data. Set the threshold that triggers a review before the first run, not after the numbers arrive, so the owner is not tempted to explain away a poor result. Finally, if the team is unsure which recurring question would make the most useful first test, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

What is the best single metric for AI memory recall? There is rarely one. Expected-fact recall is the most useful starting point because it asks whether the approved answer came back, but it should be read alongside stale-fact exposure, since a high recall score can coexist with outdated items in the same store. Read both together and look at the trend across several runs rather than any one result.

How many test questions do I need? Enough to cover the decisions the agent supports, and no more than the owner can review and keep current. A smaller set that is reviewed and frozen between cycles is more useful than a large one that nobody maintains. Add questions whenever real work produces a kind of question the set does not yet cover.

Can I measure recall without engineering work? Partly. A staff member can write the questions, the approved answers and the source references in a spreadsheet, then compare the agent's answers against them by hand. Automated runs and retrieval logs need technical setup, which managed operations can include, so the manual version is a good way to start before committing to tooling.

How often should recall be measured? Run the full set after every approved memory change, and on a fixed schedule agreed in the first engagement, such as a monthly review. Add an extra run when a policy changes, when a new system is connected, or when staff report a wrong answer. The schedule matters less than the habit of acting on the results and recording what changed.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.