Skip to content
be-human.ai
← Back to the journal

Context & memory

Seeding AI Agent Memory From Company Records in Small Reviewed Batches

Load past company records into AI agent memory in small reviewed batches, with an owner check and a test question per batch, instead of one bulk dump.

Be Human typographic cover with the title: Seeding AI Agent Memory From Company Records in Small Reviewed Batches
Conceptual illustration created with AI · Be Human

To seed AI agent memory without loading noise, load past company records in small batches, have the owner of each record type approve the batch, and test a short set of real questions before the next batch goes in. Tag every item with its source, version and owner so a batch can be traced or removed. A single bulk import is faster to run, but it makes it hard to tell which old fact is causing a bad answer.

Key takeaways

  • Load past records in batches that share one record type and one job, never as one dump.
  • Give each batch a named owner who approves it before the agent can read it.
  • Run the same fixed questions after every batch and keep the results so changes are visible.
  • Tag every fact with its source document, version, owner and last confirmation so it can be traced or removed.
  • Start the next batch only after the current one passes the full question set.

Why one bulk import creates noise

Company history is mixed. It contains superseded procedures, drafts, duplicate customer records, chat threads, and one-off decisions that were never meant to become policy. When all of it loads at once, retrieval returns neighbors of the right answer, and the agent has no reliable way to tell an old template from the current one unless the metadata says so.

The problem is also one of attention. An agent works inside a limited context for each task, as explained in What Is a Context Window in AI? Why Your Agent Forgets. Every irrelevant record competes with the record that should win. When a wrong answer appears after a bulk load, nobody knows which of thousands of items caused it, so the team stops trusting the agent and starts checking everything by hand. Small batches make cause and effect visible again.

How to choose the first batch

Start from the job the agent must do, then load only the records that job needs. Use these rules to decide what waits:

  • If the agent answers questions about order status or service terms, start with the current rules and terms, not the full mail archive.
  • If it drafts internal summaries, start with the approved meeting template and the decision log for one team.
  • If a record is a draft, a duplicate, or older than the policy that replaced it, leave it out of the first batch.
  • If two sources disagree, keep the version the owner confirms as current and park the other for later review.
  • If a record holds personal data beyond what the task needs, remove those fields before loading.

A good first batch is small enough that one reviewer can read every item in it. If that is not possible, the scope is still too wide.

A step-by-step seeding plan

Run each batch through the same sequence. The order matters, because review happens before the agent can read anything.

  • Write one sentence describing the job and the questions that prove the agent handles it correctly.
  • Choose one record type and name the owner who can approve it.
  • Select the smallest set of records that covers the job, and log what was excluded and why.
  • Remove or redact sensitive fields the job does not require.
  • Tag each item with its source document, version, owner and the date it was last confirmed.
  • Load the batch into a staging memory that the agent can read but staff do not yet rely on.
  • Run the fixed question set and compare each answer with the owner's expected answer.
  • For each failure, identify the item that caused it, then correct, relabel or remove that item and run the set again.
  • Approve the batch, promote it to live memory, and write the change in the memory log.
  • Keep the batch tagged so it can be removed as a unit if a later review finds a problem.

The last two steps are what make the process reversible. A batch you cannot find and remove is a bulk import under a different name.

Illustrative scenario: a batch caught before it reached staff

Illustrative scenario: a small consulting firm wants its agent to answer questions about engagement terms. The first batch includes only the current proposal template and the operations lead's list of standard terms. The owner writes three test questions, including one about notice periods for retainers.

The agent answers the notice period question with a figure from an older signed letter. The letter was included by accident because it sat in the same folder as the current template. The test catches the error before any staff member sees it. The team tags the letter as historical and excludes it from policy questions, runs the full set again, and only then promotes the batch. Nothing else in memory changed, which is the whole point of the method.

How to test each batch before the next one

A batch is only as good as the check that approves it. Write a short fixed set of real questions with the owner, including a few the agent should refuse or send to a person. Keep the set unchanged across batches, so a drop in quality after a new load is easy to see.

For each question, record three things: whether the answer is correct, whether it cites the tagged source that should be cited, and whether it used a record outside the batch's scope. The guidance in [Anthropic's engineering article on evaluating AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) describes the same discipline of fixed tasks, repeatable checks and results you can compare over time. For the retrieval side, How to test retrieval quality in an AI knowledge base covers what to look for when the agent pulls the wrong passage.

Keep the results with the batch record. When someone later asks why a fact is in memory, the answer should be one lookup away.

How Be Human's managed approach handles seeding

Be Human designs the memory, tools and approval boundaries for one defined role first, then runs seeding as a managed process. The team proposes batch boundaries, keeps provenance tags and change logs, runs the question checks with the client's owners, and keeps the removal path ready before anything goes live. After launch, the same routine covers recurring evaluations and permission review, so a new batch does not quietly bring back records that were already corrected. Be Human works remotely, in English or Portuguese, and the client keeps decision rights over what the agent may remember.

Where to start

Most teams know which records matter. Fewer know which set should go first, and that choice shapes everything after it. The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) is a useful reference for deciding who reviews and approves changes before they reach the agent. Take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

How many records should go into one batch? There is no fixed count, and the right size depends on how many items one reviewer can read carefully in a single sitting. If the owner cannot explain each item in the batch, the batch is too large. Split it by record type or by the question set it has to pass.

Can I reuse a batch that worked for another agent? Only when the job, the owner and the source versions are the same. Otherwise the batch carries assumptions that belong to the first agent, such as a different team's policy. Reuse the method and the review checklist, and rebuild the batch for the new role.

What should I do if a bulk import is already loaded? Stop adding to it, then identify which items came from that import and tag them as unreviewed. Rerun the question set, remove the items that cause failures, and reload the remaining records as small reviewed batches. How to Delete AI Agent Memory: Correct, Expire, Remove explains how to remove or expire items without damaging the rest.

How often should new batches be loaded? Load them when a real job needs the records, not on a fixed calendar. Each new batch should go through the same review and question checks, and the memory log should show who approved it. A slower cadence with clear approvals usually produces a cleaner memory than a fast one that nobody reads.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →