Skip to content
be-human.ai
← Back to the journal

Context & memory

Memory Retrieval Ranking for Superintelligent Agents: Source and Owner

Rank retrieved agent memories by source authority, owner and recency so the agent cites the right fact first. Includes a step-by-step method and checklist.

Be Human typographic cover with the title: Memory Retrieval Ranking for Superintelligent Agents: Source and Owner
Conceptual illustration created with AI · Be Human

When an agent retrieves several notes that answer the same question, the ranking decides which one it quotes first. A dependable order puts the record with the highest source authority first, then the record owned by the person accountable for that fact, then the most recently confirmed version, and only then the similarity score assigned by the search. Ranking by source and owner is what lets the agent's first citation be one your staff can check, defend and correct.

Key takeaways

  • Similarity scores find text that looks like the question. They do not show which text governs the answer.
  • Rank by source authority first, placing approved policies and system records above meeting notes and chat messages.
  • Use the named owner as the next signal, and as the person who answers for any disputed fact.
  • Treat recency as confirmation, not as a reason to override a higher authority source that is still in force.
  • Write source, owner and confirmation date into each memory record when it is saved, not while the agent searches.

Why similarity alone picks the wrong fact

Vector search is good at finding passages that share meaning with a question. That is useful, but it treats a draft proposal, a signed agreement and a support comment as equal candidates whenever they use similar words. The agent then cites whichever passage is phrased closest to the question, which is often not the one a manager would accept.

Illustrative scenario: a customer asks about the cancellation terms of a service plan. The memory holds the signed contract clause, an internal FAQ maintained by the customer success lead, a sales email from a pilot offer, and a support note that says the team usually waives the fee. All four match the question. Ranked by similarity alone, the sales email may win because its wording mirrors the customer's question. Ranked by source and owner, the contract clause comes first, the FAQ second with its owner named, and the support note appears only as context that nobody has approved.

The retrieval step found the right material. The ranking step decided what the agent would say, and it needs explicit rules to do that reliably.

The three signals: source, owner and recency

Source authority describes the kind of record a fact came from. A signed agreement, an approved policy in the system of record, a validated product specification, a reviewed staff correction, a meeting note and an unreviewed chat message are different kinds of evidence. Decide the list of source types your company uses and rank them once, in writing. Keep the list short enough that a person can apply it without debate.

Owner describes who answers for the fact. Each memory record should name a role, such as the head of customer success or the finance lead, rather than a department or a shared inbox. The owner is the person the agent can point to when a citation is challenged, and the person who approves changes. A fact with no owner should rank below a fact with one, because nobody is in a position to confirm it.

Recency describes when a fact was last confirmed, not when the file was created. A policy written long ago but reconfirmed by its owner in the most recent review should carry that confirmation date. A newer document that nobody has reviewed is not more trustworthy simply because it is newer. Recency is most useful as a tie breaker between versions that share the same source tier and owner.

Build an authority ladder your team can read

The ranking works only when people agree on it. Draft the ladder in a short working session with the operations lead and the owners of the main knowledge areas. Keep it to a few tiers, and write one sentence explaining each tier so that a new team member understands why it sits where it does.

Use this checklist when you set the ladder:

  • Name every source type the agent can retrieve from, and assign each one to exactly one tier.
  • Mark which tiers may be quoted directly to a customer and which may only inform the agent's internal reasoning.
  • Assign an owner role to every record class, not to individual files, so ownership survives staff changes.
  • Decide what happens to a record with no owner or no confirmation date: it ranks lowest, or it is not retrieved at all.
  • Store the ladder as a versioned document and reference it in the agent's instructions, so every change is visible.
  • Schedule a review of the ladder whenever a new source type appears, such as a new call summary format or a new policy system.

The ladder should be short enough to fit on one page. If it needs a long appendix, the tiers are probably too fine to apply consistently.

Scoring: the order of operations

Avoid a single opaque number that mixes every signal together. A transparent approach sorts candidates in a fixed sequence. First, remove any record the requesting role is not allowed to see, because a restricted record should never enter the candidate list. Second, sort by source tier. Third, prefer records with a named owner and a current confirmation. Fourth, use recency. Finally, use similarity to break remaining ties. Because each step is explicit, an operator can see which step let a weaker record through when the agent gets something wrong.

Some teams prefer a blended score. That can work, but the weights become hard to explain and easy to drift over time. If you choose a blend, keep authority dominant enough that a low tier record can never outrank a high tier record on similarity alone, and log the component values for every retrieval. Anthropic's guidance on context engineering for agents makes a related point: the smallest set of high signal information usually performs better than loading everything a search can find. Ranking is how you enforce that smallness in practice.

Whatever method you choose, log the top candidates with their tier, owner, confirmation date and final position. The NIST AI Risk Management Framework encourages organizations to document how an AI system behaves and who is accountable for it, and a retrieval log is a practical way to meet that expectation for memory.

Conflicts, stale facts and the cited answer

Ranking is also how conflicts resolve. When two records disagree and both are current, the higher tier wins, and the agent should say that a conflict exists and name the owner of the record it used. When the top record is older than the confirmation window your company sets, the agent should flag it for review rather than quietly quote it. Detailed rules for disagreeing memories are covered in When AI Agent Memories Conflict: Rules for Which Fact Wins, and the handling of old context is covered in How to stop stale context from misleading your AI agent.

The citation itself should expose the ranking. A good answer names the document or record, its version, its owner and its confirmation date. Staff can then check the source in seconds, and a wrong citation becomes a specific, fixable problem rather than a vague feeling that the agent is unreliable. The same discipline is described in Cite the Exact Document Version: Sources Staff Can Check in Seconds.

How a managed setup keeps the ranking current

Be Human designs the authority ladder, the owner fields and the retrieval order during the first engagement, and defines the approval boundaries before the agent cites anything to staff or customers. Memory records are written with source, owner and confirmation fields, and permissions are reviewed as roles change. In managed operations, the team monitors which records the agent cites, runs recurring evaluations on questions whose correct sources are known, investigates incidents where a lower tier fact was quoted, and changes the ladder only through a logged review. The goal is a ranking that a business owner can explain and inspect, not one that has to be trusted on faith. If you are not sure which of your agent's questions would benefit most from this ordering, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

Should recency ever come before source authority? Usually not. A recent note from a team chat should not outrank an approved policy that was confirmed earlier, because the policy carries the authority of its owner and approval. Recency is most valuable as a tie breaker among versions of the same record, or as a trigger for review when a confirmation has expired. Set that rule explicitly so the agent does not infer it on its own.

Who should own the ranking rules? A named operations leader should own the ladder, with the owners of each knowledge area confirming the tiers that apply to their records. Engineering or the team running the agent implements the rules and keeps the logs. Splitting ownership this way prevents a situation where nobody can explain why a particular fact was quoted.

Does ranking replace embeddings or keyword search? No. Embeddings and keyword search still find the candidate passages, and they remain useful for breaking ties at the end of the sequence. Ranking decides which of those candidates the agent may treat as authoritative and in what order it presents them. For a broader view of how search and memory fit together, the Google Cloud guidance on retrieval-augmented generation architectures describes the retrieval and grounding steps in more detail.

How do we test whether the ranking works? Write a set of questions for which the correct source is already known, including a few where a tempting but unapproved record exists. Run them on a schedule, record the top citation for each, and review every case where the agent chose a lower tier source. Add each failure to the test set so that a future change to the ladder or the memory does not quietly reintroduce the same mistake.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →