Skip to content
be-human.ai
← Back to the journal

Context & memory

Customer Facts vs Staff Opinions in AI Agent Memory: A How-To

Learn how to label confirmed customer facts apart from staff impressions in AI agent memory, with a step-by-step plan and recall rules that show each type.

Be Human typographic cover with the title: Customer Facts vs Staff Opinions in AI Agent Memory: A How-To
Conceptual illustration created with AI · Be Human

To keep customer facts apart from staff impressions in agent memory, store each item with a memory type, a source and a confirmation date. Confirmed records can be stated as facts, staff impressions are stored as labeled judgments, and open questions are kept as gaps the agent must not answer from memory. When recall returns the type along with the content, the agent can speak plainly about what is on record and carefully about what a person only thought.

Key takeaways

  • Give every memory item one of three types: confirmed record, staff impression or open question.
  • Attach a source and a confirmation date to each confirmed record, so a person can open the original.
  • Give each staff impression an author, a date, a reason and a review date.
  • Make recall show the type, so the agent never presents an impression as a record.
  • Review both kinds on a schedule, because a confirmed fact can go stale and an impression can expire.

Why facts and opinions blur together in agent memory

Most companies do not begin with a memory design. They begin with notes. A sales colleague writes that a buyer seems ready to expand, an account manager records a contract term, and a support lead notes that the customer was frustrated about a late delivery. All three end up in the same text field or the same search index, often with the same timestamp and no marker of who wrote what or on what basis.

Retrieval then treats them as equals. When the agent is asked about the account, it pulls the passages that look most similar to the question and blends them into one answer. A contract term and a remark about mood arrive with the same grammatical confidence, and the reader has no way to tell which sentence rests on a signed document and which rests on one person's reading of a call. The fix is not a larger model or a longer context window. It is a label that survives retrieval and stays visible in the answer.

Define three memory types before you write a single memory

Start with a small vocabulary. Three types cover most business cases, and each one changes how the agent may speak.

  • Confirmed record: a fact backed by a document, a system of record, or a statement the customer made in writing. It carries a source and a confirmation date.
  • Staff impression: a person's judgment about the customer, such as perceived urgency, tone or the likely next step. It carries an author, a date and a reason.
  • Open question: something the company does not know yet, such as whether a site visit happened or who approves changes above a set amount. It is a gap, not a memory.

Keep the list short on purpose. Every extra type is another decision each reviewer must make and another way for an entry to be filed wrongly. If you want a fuller method for handling gaps, the approach in AI Agent Knowledge Gaps: How to Log Unanswered Questions shows how to log them so they are not filled in later by guesswork.

A step-by-step plan to label existing and new memory

Work through existing memory first, then apply the same rules to every new entry. This sequence keeps the cleanup small enough to finish.

  • Step 1: Export a current sample of customer memory, including staff notes and any summaries the agent produced.
  • Step 2: Tag each item as a record, an impression or a question. If one item mixes two types, split it into two entries.
  • Step 3: For every record, add the source (document name and version, system name, or the customer's written message) and the date the fact was confirmed.
  • Step 4: For every impression, add the author, the date and a short reason, then set a review date after which the impression expires unless someone renews it.
  • Step 5: Write one recall rule per type: records may be stated with their source, impressions must be framed as staff notes, and questions must trigger a handoff instead of a guess.
  • Step 6: Make the type field required at write time, so an unlabeled item cannot enter memory.
  • Step 7: Test recall with real questions from the past month and check that the type and source appear in each answer.

Agent-written summaries need the same labels, because a summary is a new claim about older events. The checks in Checking AI Agent Conversation Summaries Before They Become Memory are a practical companion to step two.

Illustrative scenario: one customer, three kinds of memory

Illustrative scenario: a software reseller has an account where a renewal discussion is underway. The memory holds three items for that customer. The first is a confirmed record: the signed order form names the current seat count and the end date, and the entry cites the document version. The second is a staff impression: an account manager wrote during the last call that the operations director sounded skeptical about the new module, and the entry is flagged for review. The third is an open question: nobody has recorded whether finance must approve a change to the tier.

When a colleague asks the agent to draft the renewal email, the agent states the seat count and end date as the signed terms. It says that a staff member noted skepticism during a previous call and that this has not been confirmed. It then tells the colleague that the approval path is unknown and should be checked with the customer before the email promises anything. The colleague now knows which sentence to rely on and which one to verify.

How recall should phrase each type

The label only helps if recall shows it, so phrasing rules belong in the design, not in a reviewer's memory. Keep the patterns short and fixed, and review them like any other output.

  • Record: state the fact and name its source, for example: the signed order form on file sets the current seat count.
  • Impression: name the author, mark it as a judgment and say it is unconfirmed, for example: a staff note from the last call records that the director seemed skeptical. This is an impression, not a confirmed position.
  • Question: say that the item is not on record and name the next step, for example: this is not recorded, and the account owner can confirm it with the customer.
Confirmed by the signed order form. Staff impression from the last call, not verified.

Avoid two common errors. The first is softening a record into an impression to sound cautious, which hides facts the team should use. The second is hardening an impression into a fact to sound helpful, which is how guesses become commitments. Both errors are what this labeling is meant to prevent.

Who may change a label, and when

Labels change over time. An impression becomes a record when a source arrives, a record becomes stale when a newer document replaces it, and an open question closes when someone answers it. Each change needs an owner. In many teams the account owner can promote an impression after checking a source, while only the owner of the underlying document can change a record. Changes should be logged with the previous label, so staff can see how an entry evolved.

When two items disagree, do not let recall pick whichever text is newer. Apply a written rule for which source wins, such as the one described in When AI Agent Memories Conflict: Rules for Which Fact Wins. The NIST AI Risk Management Framework is a useful reference for assigning these responsibilities and documenting them, which matters more as the memory grows.

Where to start with your own customer memory

How Be Human approaches this in managed work: the first engagement defines the memory types, source fields and recall phrasing alongside the one role the agent will perform, the approval boundaries for changing a label, and the success criterion that shows whether the labels are working. In managed operations, the team reviews a sample of recall answers on a recurring schedule, checks that records still match their sources, expires impressions nobody renewed, and routes conflicts and open questions to the named owner. Changes to labels are recorded so they can be traced to a person, and the company keeps control of the agent's memory and permissions throughout.

If you are unsure which customer memory is worth labeling first, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Should staff impressions be stored in agent memory at all? Yes, when they are labeled as impressions with an author, a date and a review date. Staff judgments often carry useful context, such as which contact tends to make the final decision. The risk comes from storing them without a label, because recall then presents them as facts.

How is a confirmed record different from a summary the agent wrote? A confirmed record traces back to a document, a system or a customer statement that a person can open and read. A summary is a new statement about past events, so it needs its own check before it is stored as a record. Until someone verifies a summary against the source, treat it as an impression.

How often should impressions be reviewed or expired? Set the review date when the impression is written, based on how quickly the situation is likely to change. A sentiment about a stalled deal may need a review soon, while a stated preference may hold until the customer changes it. Expired impressions should be archived rather than deleted, so staff can still see how the account developed.

Can one memory item hold both a fact and an opinion? It should not. Split the item into a record and an impression, each with its own type, source and date. Mixed entries are the most common reason recall blends the two, so splitting them is usually the first cleanup step.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →