Skip to content
be-human.ai
← Back to the journal

Personalized AI infrastructure

Model-Agnostic Company Memory: Swap Models Without Losing Context

Keep company memory in your own records and tools, not inside one model, so a newer model can inherit the same context on its first day of work.

Be Human typographic cover with the title: Model-Agnostic Company Memory: Swap Models Without Losing Context
Conceptual illustration created with AI · Be Human

Company memory stays usable across model changes only when it lives outside any single model. Keep the facts, decisions, permissions and working procedures in records your company controls, then connect each model to them through a defined interface. A newer, more capable model can then start from the same context on its first working day instead of relearning your business from scratch.

Key takeaways

  • Company memory is the set of records, rules and permissions an agent needs, and it should belong to your company, not to a model vendor.
  • Separate three things that are often mixed together: memory (what is known), tools (what can be done) and permissions (who approves what).
  • Store memory in plain, versioned, reviewable formats so that any model can read it through the same retrieval path.
  • Test a candidate model against your own tasks before it replaces the current one, and keep the previous configuration available until the test passes.
  • Treat a model swap as a change request with a named owner, a written success criterion and a rollback step.

What does model-agnostic company memory mean?

Model-agnostic company memory is the body of context a business relies on, kept in a form that does not depend on which model reads it. That context includes customer history, product rules, approved wording, escalation paths and the reasons behind past decisions. When this material sits inside a model's hidden state, a chat history or a vendor's proprietary memory feature, a change of model means losing it or rebuilding it by hand.

Many teams first notice the problem when an agent has learned useful habits over several months and nobody can say where those habits are stored. The agent answers well, but the knowledge is effectively tied to one provider or one session. Moving to a stronger model then feels like starting over.

The goal is not to avoid upgrades. The goal is to make upgrades cheap to evaluate and cheap to reverse. A company that can change models without losing context can adopt better capabilities when they prove useful, and can keep the current model when they do not.

Where should company memory live?

Start by listing what the agent must know to do its job. For most operating roles, the material falls into four groups: stable reference material such as policies, product definitions and procedures; current state such as open orders, active projects or open tickets; history such as prior decisions and customer interactions; and rules about what the agent may say or do. Each group has a different owner and a different update rhythm, so it should not sit in one undifferentiated pile.

Keep the source of truth in systems your company already controls or can export from: a document repository, a customer record system, a ticketing tool, or a versioned folder of procedures. The model should read these sources through retrieval, which means fetching the relevant passages at the moment of a task. The AWS guidance on comparing retrieval with fine-tuning explains why this approach lets knowledge change without retraining a model. For a business owner, the practical point is simpler: if the facts are in your records, a new model can fetch them the same way the old one did. A broader view of how these pieces fit together is in Draw Your AI Stack: A One-Page Architecture Map for Leaders.

Avoid treating a model's built-in memory as your system of record. Built-in memory is convenient, but it is usually tied to one provider and is hard to inspect or export. You can use it for convenience while keeping a separate, complete copy of anything the business depends on.

Format matters too. Use plain text, markdown, CSV or structured records with clear dates and owners. Avoid screenshots, scanned files without a text layer and notes that only one employee can interpret. If a person cannot audit a memory item in a few minutes, a model will struggle with it as well.

How do you separate memory, tools and permissions?

Most tangled agent setups blur three separate things. Memory answers the question of what is true here. Tools answer the question of what the agent can do, such as look up an order, draft an email or update a ticket. Permissions answer the question of which actions need a person to approve them first. When all three are bundled into one long prompt, a model change can quietly alter all of them at once.

Keep them as separate, named components. Memory is a set of sources, each with an owner. Tools are defined interfaces with stated inputs, outputs and error messages. Permissions are a table that maps each tool action to an access level and an approval rule. The OWASP authorization guidance makes the related point that access decisions should be enforced by the system under clear rules, not left to a model's judgment. Keeping that enforcement outside the model means a new model cannot widen its own access by accident.

A step-by-step plan for changing models without losing context

Use this sequence whenever you consider replacing the model behind a business agent.

  • Step 1: Write down the role, its three most common tasks and the sources each task reads from.
  • Step 2: Confirm that every source can be exported and has a named owner inside your company.
  • Step 3: Record each tool the agent uses and the permission level and approval rule for each action.
  • Step 4: Build a test set of realistic cases drawn from past work, including several that the current model handles poorly.
  • Step 5: Run the current model and the candidate model against the same cases with the same retrieval setup.
  • Step 6: Score the results against a written success criterion, such as citing the correct source and stopping at the approval boundary.
  • Step 7: Run the candidate in shadow mode, where it drafts responses and a person decides what is sent, for a review period agreed in advance.
  • Step 8: Switch the default only after the owner signs off, and keep the previous configuration ready for rollback.
  • Step 9: Log the change, the test results and the decision alongside other agent changes.

The same discipline applies to any change, not only to model swaps. The steps for handling change requests are described in How to Request and Approve Changes to a Managed AI Agent.

Illustrative scenario: a support role changes models

This scenario is illustrative and does not describe a real client. A small company uses an agent to draft replies to customer questions about delivery schedules and returns. The agent reads a policy folder, a shipping status export and a log of replies the team has already approved. When the company considers a newer model, the team first checks whether the policy and the reply log are in files they control. They discover that the current agent has been relying on a provider memory feature for two return exceptions, and nobody had written those exceptions down.

The team records the exceptions in the policy folder with a decision date and an owner, then runs a sample of past tickets through both models. The newer model writes a better tone but misses one exception that appeared only in the old reply log. The team adds that case to the policy folder, reruns the sample, and both models pass. The new model then drafts replies in shadow mode until the review period ends. The useful outcome is not the model choice itself but the list of hidden rules the team could finally see.

How Be Human handles model changes in managed operations

Be Human designs and operates agents so that memory, tools, permissions and operating context stay under the client's control rather than inside one model. In a managed engagement, the team documents each memory source and its owner, maps tools to permission levels, and maintains an evaluation set that runs whenever the model or the configuration changes. A model change follows the same change request path as other updates, with a written success criterion, a shadow period and a rollback plan. Monitoring and periodic permission reviews check that the context a new model receives is the context the business approved. The service is delivered remotely, in English or Portuguese, and starts with one useful role rather than a full replacement of existing systems.

How do you test that a new model inherits the context?

Inheriting context is not the same as sounding informed. Test it directly. Ask the candidate model questions whose answers exist only in your records, including questions about exceptions, the dates of decisions and the named owner of a rule. Check whether it cites the right source, admits when a source is missing, and declines actions beyond its permission level. Add a few adversarial cases as well, such as a customer message that tries to override policy with instructions hidden in the text. Prompt injection is a known failure pattern, and the OWASP prompt injection cheat sheet is a practical reference for designing those cases.

The evaluation method matters as much as the cases. The OpenAI evals guide describes writing tests that compare model outputs against expected results, which gives a business team a workable base for its own set. Keep the set small enough to run on every change and realistic enough to catch the failures your team actually fears. Each failure found in production should become a new test case before the fix is deployed, so the same gap cannot return silently after the next swap. For a wider view of how evaluation supports ongoing operations, see Feedback Loops That Make Company AI Better Every Quarter.

If you are not sure which of your agent tasks would be the safest place to start this work, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Does company memory really have to live outside the model? Not every piece, but everything the business depends on should be stored in records your company controls and can export. A model can hold temporary context for a single session, yet the durable facts, decisions and permission rules should survive any change of model. If they do not, every upgrade becomes a rebuild.

Can I rely on a vendor's built-in memory feature? You can use it for convenience, but it should not be your only copy of anything important. Built-in memory is usually tied to one provider and can be hard to inspect, which makes audits and migrations harder. Keep a complete, exportable version of the same material in your own sources.

How long does it take to test a new model before switching? The time depends on the size of your test set, how many people must review results and how long the shadow period runs. A small, well-defined role with a clear success criterion can be evaluated much faster than a broad assistant with many tools. Agree on the review period and the sign-off owner before the test begins, so the decision is not made under pressure.

Who should approve a model change for a business agent? The owner of the role should approve it, with input from whoever owns the memory sources and anyone accountable for the permissions involved. The approval should rest on the written success criterion and the test results, not on the vendor's benchmark claims. Record the decision and keep the previous configuration available until the change has proven itself in practice.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →