Personalized AI infrastructure
Multi-Model Routing for AI Agents: When to Switch Models
Learn when to send AI agent tasks to different models, how to keep the harness constant, and how to log routing decisions for review.

Multi-model routing means sending each task an AI agent receives to the model best suited to it, based on the task's risk, data sensitivity, response time and cost limits. The surrounding harness, meaning instructions, tools, memory scopes, permission checks and logs, stays the same for every route. Every routing decision should also be recorded with its reason, so a person can review it later.
Key takeaways
- Route by task class, risk and data sensitivity first, and by model reputation last.
- Keep one harness for all models: the same instructions, tool definitions, permission checks and memory scopes.
- Log each routing decision with the rule that chose the model, the model version, the tools used and the outcome.
- Test every route against a fixed evaluation set before it handles live work, and rerun that set on every change.
- Treat routing rules as configuration under change control, and review them on a schedule.
Why one agent should not depend on one model
A single model rarely fits every job an agent performs. Drafting a customer reply, classifying an inbound request, reading a long contract and deciding whether a refund needs approval place different demands on reasoning depth, speed, context length and cost. A company that pins its whole agent to one model either pays more than necessary for routine steps or accepts weak results on the hard ones. Routing treats models as components that are selected per task instead of as the identity of the agent.
The benefit is not only cost. Routing also gives the company room to respond when a provider changes pricing, retires a model or releases one that performs better on a narrow task. The risk is that an agent quietly depends on one model's habits rather than on documented instructions and tools, and then every switch becomes a rebuild. The discipline in the sections below keeps switches small. The same concern about dependence appears in how to avoid AI vendor lock-in with business agents.
Start with a routing table, not a model preference
Before choosing any model, list the task classes the agent performs and attach four properties to each: the cost of a wrong answer, the sensitivity of the data involved, how long a person will wait, and the cost per task the business can accept. Then write each routing rule in plain language. Rules based on brand loyalty or a headline benchmark are hard to defend once something goes wrong.
A practical decision guide:
- If the task can take an external action, such as sending a message, changing a customer record or publishing content, route it to the most capable model you have approved, and require a human approval step before the action runs.
- If the task is classification or extraction from a known format, route it to a smaller, faster model and validate the output against a schema.
- If the task involves personal or regulated data, route it only to a model and hosting arrangement that your data policy already approves, whatever quality gains another model might show.
- If the task requires long context, such as a full policy set or a long conversation history, retrieve only the relevant passages first. Sending less, better-chosen context is often cheaper and easier to check than sending everything to a larger window.
- If the input is ambiguous or the model reports low confidence, route to a fallback that asks a clarifying question or escalates to a person instead of guessing.
A routing table that needs a long footnote is usually two rules pretending to be one. Keep it short enough that an operations leader can read it in a few minutes and explain every line.
Keep the harness constant across models
The harness is everything around the model: system instructions, tool definitions, the memory the agent may read, permission checks, output schemas, retry behavior and logging. Keep these in one versioned place and treat the model as a parameter that the routing table sets. A small, explicit harness is easier to audit than a clever one, and Anthropic's guidance on building effective agents makes the same case for simple, composable patterns.
A checklist for keeping the harness stable:
- Tool definitions are written once, with names, arguments and descriptions that do not depend on one model's phrasing habits.
- Permissions are enforced by the harness and by the systems it calls, never by asking a model to refrain from an action.
- Memory scopes are identical for every route, so a cheaper route cannot read records that the stronger route is not allowed to see.
- Every output is checked against a schema before any action runs, whichever model produced it.
- Prompts are stored as versioned files with a changelog, and a prompt change goes through the same review as a model change.
- Any tool that writes to a business system accepts an idempotency key, so a retry on a different model does not send the same message or create the same record twice. Stripe documents this pattern for its API, and it is a good model for internal tools too.
Retrieved documents, inbound emails and web pages can contain text written to manipulate the model. The OWASP guidance on prompt injection prevention is a useful baseline here. Treat model output and external content as untrusted input to the harness, not as instructions with authority over permissions or routing.
Write every routing decision to a log a person can review
If routing happens out of sight, nobody can tell whether the rules work. Each request should produce a record that answers six questions: which task class was detected, which rule selected the model, which model and version ran, which tools it called, whether the output passed validation, and whether a person approved, changed or overrode the result. OpenTelemetry's semantic conventions for generative AI define standard attribute names for models, operations and token usage, which makes these records easier to compare across tools. The OpenTelemetry observability primer explains how traces and spans tie one request to all of its steps.
Setting up the log:
1. Assign a request identifier when the task enters the agent, and carry it through every model call and tool call. 2. Record the routing rule identifier, not only the model name, so a reviewer can see why the route was chosen. 3. Store token counts and an estimated cost per task, so cost can be reviewed per task class. The FinOps Foundation's material on unit economics is a sound frame for this review. 4. Record escalations and human overrides as separate events. A high override rate on one route is a signal about the route, not a failure of the reviewer. 5. Limit log access the same way you limit the underlying data, because logs often contain customer text. 6. Set a review date for the log. A log nobody reads is storage, not governance.
Test each route before it carries real work
A model that looks strong in a demo can still fail on your inputs. For each task class, build a fixed set of realistic examples, including messy inputs, incomplete records and a few deliberately hostile ones. Score each candidate model on the criteria that matter for that class: correctness against a reference answer, schema validity, escalation where it should happen, and response time. OpenAI's evals guide and Anthropic's guidance on evals for AI agents both describe the same practice: define the grader, keep the test set stable, and rerun it whenever the model, prompt or tools change.
Illustrative scenario, not a real client: a distributor of clinic supplies wants one agent to answer order status questions and draft replies to wholesale price requests. Status questions are read-only, so they route to a small, fast model and are checked against the fields the order system returns. Price requests involve discount decisions, so they route to a more capable model and always stop at a staff approval step. When a newer small model appears, the team reruns the status evaluation set. It improves on accuracy but handles partially shipped orders poorly, so the route stays where it is until that case is addressed in the prompt. The harness, the tools and the log format do not change. Only the routing table changes, and only for the class that passed.
How Be Human handles routing in managed operations
Be Human designs and operates the harness as its own layer, so a model can change without changing the agent's identity, memory scopes, permissions or logs. In a first engagement, the routing table is written against the one role the agent is meant to perform, and each route is tied to an observable success criterion and a defined approval boundary. In managed operations, recurring evaluations rerun the test sets when a model or prompt changes, permission reviews confirm that each route reads only what it should, and controlled improvements move a task to a different model only after the evidence is recorded. The company keeps the records, so it can see why each route exists and what would make it change.
Review routing rules every quarter and remove what no longer earns its place
Routing rules age. Models are retired, prices change and the task mix shifts as the team learns what the agent does well. Schedule a review of the routing table alongside the permission review, and use the log as the evidence. Look for the routes that escalate most often, the ones whose outputs are overridden most often, and tasks that have drifted into a class the table does not cover. Remove routes that no longer justify themselves, and do not move a task to a new model without a passing evaluation run recorded in the same place.
A routing change is a change to the agent, so it should follow the same request, review and approval path as any other change. The steps are described in how to request and approve changes to a managed AI agent, and the permission side is covered in how to run a permission review for your AI agents, step by step. Start with one or two routes, widen only when the logs show the rules hold, and keep the harness exactly as it is while you do.
Take the free Be Human AI diagnostic to find the first task worth handing to an agent.
Frequently asked questions
Do we need more than one model to start? No. Many useful first agents run on a single model with a clear harness, a small permission set and a log. Add a second route only when a specific task class shows a measurable need, such as cost, speed or a quality gap in your own evaluation set.
Can a cheaper model handle customer-facing replies? It can, for simple and well-documented cases such as order status or opening hours. Replies that involve money, exceptions, complaints or commitments should go to a stronger model with an approval step, and every reply should be checked against the same validation rules before it is sent.
How do we keep routing from sending data to a model we did not approve? Make the data policy the first rule in the routing table, so a route can only be chosen from the list of models and hosting arrangements that your policy allows. Enforce that list in the harness, not in the prompt, and record each route in the log so the check can be audited.
What is the minimum a reviewer needs to see for each routing decision? The reviewer needs the task class, the rule that chose the model, the model and version, the tools called, the validation result, and any human approval or override. With those six fields, a reviewer can tell whether the rule was applied as written and whether it still matches the business's risk appetite.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


