Skip to content
be-human.ai
← Back to the journal

Personalized AI infrastructure

Model-Agnostic Superintelligence Architecture: How to Swap Models Safely

Keep memory, tools and permissions outside the model so you can switch models as they improve without rebuilding your AI agents. A practical plan for leaders.

Be Human typographic cover with the title: Model-Agnostic Superintelligence Architecture: How to Swap Models Safely
Conceptual illustration created with AI · Be Human

A model-agnostic superintelligence architecture keeps a company's agent memory, tools, permissions and operating rules in layers the company owns, and treats the language model as one replaceable component. Because the model sits behind a defined interface, a team can move to a stronger or cheaper model later without rebuilding the workflows, records and approval rules around it. The sections below explain where to draw that boundary, how to test a candidate model, and how to run a swap so the agent keeps working for the people who depend on it.

Key takeaways

  • The model is an engine you can replace, so your agent's knowledge and business rules must not live only inside it.
  • Memory, tool definitions and permissions belong in layers you own, can export and can review.
  • Approval boundaries should be enforced by the surrounding system, not by the model's own judgment.
  • Test every candidate model against the same evaluation set before it touches live work.
  • Keep a rollback path and a written record of each change so the decision can be explained later.

Why the model should not hold your agent's memory

Many first attempts at an AI agent place almost everything inside a single vendor session: the instructions, the customer history, the list of what the agent may do, and the lessons learned from past mistakes. That works until the company wants to change models. At that point the memory is tied to one provider's format, the instructions were tuned to one model's habits, and nobody can say which rules are business rules and which were accidental prompt wording.

Memory should be treated as company data with an owner. Some of it is reference material, such as policies and product facts, which can be indexed and retrieved when needed. Some of it is operational history, such as open cases, approved exceptions and prior decisions, which needs clear retention rules. A useful test: if the model stopped being available tomorrow, could your team still find the records the agent relied on? If the answer is no, the architecture depends on the model more than it should.

The four layers to separate

A practical architecture divides the agent into layers that change at different speeds. The model changes most often. The harness and tool contracts change less. Memory grows continuously. Permissions change only when the business decides they should.

  • Model: the reasoning engine that reads context and proposes the next step, chosen and replaced as capability and cost shift.
  • Harness: the code that assembles context, calls the model, checks outputs against a schema and records what happened.
  • Memory: approved records, reference documents and summaries, each with an owner, a source and a retention rule.
  • Tools: named actions such as checking an order status or drafting a refund request, each with typed inputs, a clear description and logged results.
  • Permissions: rules stating which role, channel or workflow may use which tool, enforced by the tool layer and not by the prompt.

For a fuller picture of how these pieces relate, read the superintelligence stack explained by layer. For swapping, the key point is that only the model layer should be expected to change without a review of the others.

Tools deserve a standard interface. The Model Context Protocol describes an open way for AI applications to connect to external tools and data sources, which reduces the custom connection work each model needs. Whether you adopt that protocol or an internal interface, the tool contract should be written in your documentation, not only in a vendor's configuration.

How to make a model swap safe, step by step

A swap changes the engine, so it should follow the same discipline as any change to a live process. The steps below assume the four layers are already separated. If they are not, separating them is the first task, because every later step depends on it.

  • Write down the agent's role, approved tools and approval boundaries in one document that does not name the model.
  • Freeze the memory sources the agent uses, recording the version or effective date of each reference document.
  • Run the current model and the candidate model against the same evaluation set of real tasks, with identical context and tools.
  • Compare results on task completion, correct tool use, handling of forbidden requests and escalation behavior, not only on how fluent the answers read.
  • Confirm the candidate cannot call any tool outside its permission rules, even when the input text asks it to.
  • Route low-risk traffic to the candidate first, with the previous model ready to take over immediately.
  • Record the decision, the evidence and the rollback steps in the change log before widening the rollout.

Illustrative scenario: replacing the model behind a support agent

Illustrative scenario: a service company runs an agent that answers routine booking questions and drafts replies for a staff member to send. The team learns that a newer model handles long policy documents more reliably. In a well-separated design, the swap touches only the model setting in the harness. The policy documents stay in the memory store with named owners, the booking lookup tool keeps its schema, and the rule that the agent cannot issue refunds stays in the tool permissions. The evaluation set, built from past tickets the team already reviewed, shows where the new model is stronger and where it drafts replies that break a policy. The team keeps the old model as a fallback through a review period, then retires it through the change log.

In a tangled design, the same team would find that the refund instruction lived only in a long prompt, that the booking tool was described differently for each model, and that the old conversation history could not be read by the new one. The swap would then be a rebuild, not a configuration change.

How do you test a candidate model before it touches work?

Testing a model is different from demonstrating it. A demo shows the best answer to a friendly question. An evaluation shows what the agent does across the ordinary, awkward and forbidden cases your business actually faces. Build the set from anonymized past tasks, include the cases where staff corrected the agent, and add a group of requests the agent must refuse or escalate. The OpenAI evals guide and Anthropic's writing on evals for agents both describe this habit of testing against realistic tasks rather than single prompts.

Score each run on outcomes a finance or operations lead can follow: whether the task was finished correctly, whether any tool call fell outside permission, how often a person had to correct the output, and what each completed task cost to run. Cost belongs in the same discipline as unit economics in any operating business, where spending is read per unit of useful work rather than per request. Keep the evaluation set under version control. It is the company's record of what good looks like, and it should survive any change of model.

Keeping permissions outside the model

Permissions are the part most likely to weaken during a model change, because a new model may follow instructions differently. The safe pattern is to let the model propose an action and let the tool layer decide whether that action is allowed for this user, channel and workflow. Authorization checks belong in code, applied the way a system applies them to a person. The OWASP authorization guidance is a useful reference for this separation. The OWASP Top 10 for LLM Applications also lists prompt injection among its top risks, which is why the agent's allowed actions must never depend on what an incoming email or document asks it to do. The NIST AI Risk Management Framework offers a shared vocabulary for mapping and measuring these risks as the system changes.

Review the permission set on a schedule and check each tool against the approval boundary the business wrote down. A practical starting point is the step-by-step method for permission reviews.

How Be Human keeps the boundary in place

Be Human designs and operates this kind of layered setup for companies that want their agents to outlast any single model. The first engagement defines one useful role, the memory and tools that role needs, the approval boundaries and an observable success criterion. In managed operations, the team keeps the records of memory sources and tool contracts current, runs recurring evaluations when a new model is considered, reviews permissions, handles incidents and applies controlled changes with a written record. The goal is that a model change becomes an operational decision the client can review, not a rebuild that depends on one vendor's memory of how the agent was configured.

If you want to find the first task worth handing to an agent under this kind of structure, take the free Be Human AI diagnostic.

Frequently asked questions

Does a model-agnostic architecture mean we avoid choosing a model provider? No. You still choose a provider for each task, and you may use more than one. The difference is that the choice is reversible, because the memory, tools and permissions belong to your company and sit outside the provider's session.

How often should we consider a model swap? Consider a swap when your own evaluation set shows that a candidate performs better on your tasks, or when cost, data handling or policy requirements change. Announcements alone are not a reason to change production behavior. Each candidate should earn its place through the same test you use for the current model.

Can we swap models without retraining or fine-tuning? Often yes. When memory is retrieved at runtime and the tools are defined in your own contracts, the change usually happens in the harness and the evaluation. Fine-tuning is a separate decision with its own costs and data requirements, and it should not be the default route for a model change.

What should we ask a provider before committing? Ask whether your memory and tool definitions can be exported in an open format, whether permissions are enforced outside the model, and whether logs of each agent action are available to your team. Those answers show whether the architecture is yours or rented. For a broader checklist, see how to avoid AI vendor lock-in with business agents.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →