Skip to content
be-human.ai
← Back to the journal

Context & memory

Memory Retrieval Ranking for Superintelligence: Which Notes Come First

Rank stored agent notes by recency, source and owner so the agent reads the most reliable record first. Includes a scoring plan and review steps.

Be Human typographic cover with the title: Memory Retrieval Ranking for Superintelligence: Which Notes Come First
Conceptual illustration created with AI · Be Human

Memory retrieval ranking decides which stored notes an AI agent reads first when it answers a question or prepares an action. For a company agent, the practical rule is to score each note by how current it is, where it came from and who is accountable for it, then load the highest scoring records before anything weaker. Done well, the agent stops treating an old chat remark as equal to an approved policy.

Key takeaways

  • Rank notes by three signals together: recency of confirmation, source quality and a named owner.
  • Put approved records and system-of-record data above staff corrections, and both above conversation summaries and agent inferences.
  • A note without an owner or a confirmation date should load lower, and it should never silently override a newer, owned record.
  • Ranking is a starting order, not a final verdict. High-stakes conflicts need a human decision and a written rule.
  • Test the ranking with real questions, log every wrong top result, and adjust the scoring rule on a fixed review schedule.

Why the order of notes changes the answer

Retrieval does not return one note. It returns a set of candidates, and the agent builds its answer from whatever fits in its working context. Context windows are finite, so the set is trimmed, and the candidates that survive are usually the ones that look most relevant on the surface. A two-year-old note about a supplier price and a note confirmed last week can both match the same question with similar wording. If nothing in the system says which one is more reliable, the agent will often pick the one that reads more confidently, which is not the same as the one that is correct.

This is why retrieval quality is partly an ordering problem. The Anthropic engineering team describes context engineering as choosing the smallest set of high-signal information that gives the model the best chance of the right behavior. Ranking is the step that decides what counts as high signal. Without it, the agent treats every stored sentence as equally trustworthy, and your staff spend their time correcting answers that were retrieved correctly but ranked wrongly.

Three signals to rank by: recency, source and owner

Recency should measure when a fact was last confirmed, not when the note was created. A note written eighteen months ago that someone re-verified last month is more reliable than a note written last week that copied a draft. For facts that change, such as prices, opening hours, service intervals or staff assignments, add an expiry date so the note drops in rank once it is due for review.

Source answers the question of where the fact came from. A workable order for most companies is this: system-of-record data (the scheduling tool, the accounting ledger, the signed contract) first; approved internal documents with a version number second; staff corrections that have been reviewed third; conversation summaries and agent inferences last. Each class should be labeled in the memory store so the ranking can read the label rather than guess from the text.

Owner answers who is accountable if the note is wrong. Every policy-type or customer-facing note should name a person or team. When the owner leaves or changes role, the notes they own should be flagged for reassignment, otherwise they keep their rank while nobody is responsible for them. An unowned note can still be useful as background, but it should not outrank an owned record on the same subject.

How to build the scoring rule

You do not need a complex model to start. A transparent rule that a manager can read in one minute is more useful than a clever one that nobody can explain. The following plan works for a first engagement:

  • Step 1: List the question types the agent will handle, such as refund eligibility, appointment changes or warranty terms, and name the owner of each topic.
  • Step 2: For each topic, identify the system of record and the approved documents that cover it, and label them as the highest source class.
  • Step 3: Set a review interval for each class of fact, and store the last confirmation date with every note.
  • Step 4: Assign a simple order: source class first, then whether the note is within its review interval, then whether it has a named owner, then recency as the final tie-break.
  • Step 5: Exclude notes past their expiry from the top of the list, and flag them for review rather than deleting them silently.
  • Step 6: Write the rule on one page, approve it with the owners, and version it like any other operating document.

Keep the rule short enough that an operations lead can apply it by hand to a sample of ten notes. If the rule needs a diagram to explain, it is probably too complicated for the first version.

Illustrative scenario: a refund policy and a chat remark

The following scenario is illustrative and does not describe a real client. A small service company has an approved refund policy document, updated by the operations manager, and a conversation summary from six months earlier in which a technician promised an exception for a customer. A customer asks whether a refund is possible for a cancelled visit. Without ranking, both notes match the question closely, and the agent may quote the exception as if it were policy.

With the scoring rule above, the approved policy ranks first because it is a document with a version, a named owner and a recent confirmation. The conversation summary ranks last because it is a summary, has no owner and is past its review interval. The agent answers from the policy, and the exception note is flagged to the operations manager, who decides whether it should become a documented rule. The customer gets a consistent answer, and the manager gets a clear record of the disagreement rather than a silent guess.

Where ranking should stop: conflicts and human approval

Ranking is good at ordering notes that mostly agree. It is not a reliable way to settle a real contradiction, especially when the answer affects money, safety, a regulated obligation or a customer's personal data. When two high-ranked notes disagree, the agent should say so, cite both, and route the question to the owner. The conflict handling rules in When AI Agent Memories Conflict: Rules for Which Fact Wins describe how to decide which fact should win and who signs off.

The NIST AI Risk Management Framework treats measurement and management of AI risk as ongoing work, which fits this pattern. The ranking rule is one control among several, and it should sit inside a documented process with owners, review dates and a way to record decisions. Ranking also needs protection against bad input. OWASP's guidance on prompt injection explains that content retrieved from stored or external sources can carry instructions an agent should not follow. A note that looks authoritative because it is highly ranked should still be treated as data, not as a command that changes the agent's permissions or actions.

How to check that the ranking works

Test the ranking the way you would test a new employee on the job. Write thirty or so real questions from recent tickets or emails, and for each one record which note the agent used and whether that note was the correct, current, owned record. This is a retrieval check, separate from whether the final answer sounds right. An answer can read well and still come from the wrong note.

Keep a short log of misses. Each miss should name the question, the top-ranked note, the note that should have ranked first, and the reason. Most misses fall into a few patterns: a missing owner, an expired note that was not flagged, a summary that outranked a document, or a question that needed two sources and got one. Fix the rule or the note, then rerun the same questions. Repeat this after any change to the knowledge base and on the review schedule you agreed with the owners. For more on keeping the content itself current, see AI Knowledge Base Maintenance: Owners and a Review Routine.

How Be Human handles this in managed operations

Be Human designs the memory and tools an agent needs for one useful role, and defines the ranking rule as part of that design: the source classes, the owners, the review intervals and the approval boundaries for conflicts. In managed operations, Be Human can run the recurring retrieval checks, review expired or unowned notes with your team, maintain the integrations that feed the memory, and adjust the rule through controlled changes that are logged and approved. The agent keeps its identity, memory and permissions under your control, and the ranking is something your team can read, question and change.

If you are not sure which of your agent's notes should rank first today, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

How is memory retrieval ranking different from search? Search finds notes that match a query, while ranking decides the order in which the matched notes are shown to the agent and how much of each one fits into its working context. Both matter, but ranking is where reliability is decided, because a relevant note that is out of date or unowned can still lead to a wrong answer. A good setup uses search to gather candidates and a written rule to order them.

Should the most recent note always win? No. Recency helps only when the newer note is confirmed by an owner or comes from a higher source class. A recent chat summary can be less reliable than an older approved policy, so recency should be the final tie-break rather than the first rule. Expiry dates for time-bound facts are a safer use of recency.

Do we need a vector database or a knowledge graph to rank notes? Not necessarily. Ranking by source class, review status, owner and date can be done with the metadata stored alongside each note, whatever store holds it. Richer structures can help later, but a first version works with clear labels and a short rule that people can read and check.

How often should the ranking rule be reviewed? Review it on the same schedule as the content it governs, with an extra check after any major change to policies, staff roles or integrations. Keep a log of retrieval misses so each review starts from real examples rather than memory. Changes should be versioned and approved by the named owners, so the agent's behavior can be traced back to a decision someone made.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →