Model Routing for Company AI Agents: When to Escalate
A guide to routing routine agent work to a smaller model, escalating hard cases, and keeping logs that let your team stay in control as models change.

Model routing for a company AI agent means deciding, for each task, which model should do the work and when that work must move to a stronger model or to a person. Routine steps such as classifying a message or filling a known form can usually go to a smaller model, while ambiguous, high-stakes or failing cases escalate under written rules. The routing rules and the logs that record each decision are what keep the company in control when the underlying models change.
Key takeaways
- Route work by task type and by the cost of a wrong answer, not by how difficult a request feels to the model.
- Give every route a written purpose, an owner, a set of allowed tools and a fallback path.
- Escalate on evidence the harness can check: failed validation, missing fields, repeated retries, sensitive categories and requests for decisions that belong to a person.
- Log the route, the model version, the escalation reason and the final outcome for every routed step.
- Re-run the evaluations for each route whenever a model or prompt changes, not only when the code changes.
Why one model for every task is the wrong default
A single model handling every request is simple to build and simple to explain, which is why many first agents work that way. The problem appears once the agent handles a mix of work: confirming an appointment, summarizing a supplier email, drafting a reply about a late payment and reading a contract clause. These tasks differ in how much a mistake costs, how quickly an answer is needed and how much context they require. Routing turns those differences into rules the company can read, test and change.
Routing is not the same as choosing the most capable model and hoping for the best. A large model can still fail on a poorly specified task, and a small model can handle a narrow, well-defined step reliably. The goal is to send each task to the least capable route that meets its success criterion, and to move work up only when evidence shows the smaller route is not enough.
How do you split the work into routes before choosing models?
Start from your list of tasks, not from a model catalog. The broader logic of matching tasks to models is covered in Model Routing for AI Agents: Match Each Task to the Right Model, but the first step is always internal to your operation. Work through the list in this order:
- List every task the agent performs, and write one sentence describing a correct result for each.
- Tag each task by the cost of a wrong answer: low, moderate or high.
- Mark any task that touches money, customer records, legal wording, access permissions or physical safety.
- Assign each tag a default route: a small model for narrow, checkable steps, a standard model for drafting and reasoning, or a person for decisions your policy reserves.
- Write the escalation trigger for each route in terms that someone other than the builder could check.
- Review the table with the person who owns the outcome, not only with the engineer who wires it.
Two routes are usually enough to start. Each additional route adds rules, logs and maintenance, so add one only when the evidence shows a real class of cases that the existing routes handle badly.
Which triggers should move work up a route?
Escalation should respond to things the harness can observe. A rule such as the model seems unsure is weak, because self-reported confidence can sound certain when the answer is wrong. Stronger triggers are checkable, and most of them can be written as a sentence a reviewer could verify from the log:
- The output fails its schema or leaves out a required field.
- A check against source data fails, such as a date that is not on the approved calendar or a value that is not in the approved table.
- The same step is retried more often than the limit you set.
- The request contains a category your rules mark as sensitive, such as a complaint that mentions a legal dispute or a health concern.
- The agent is about to use a tool that is not allowed for that route.
- The customer or employee asks for a decision that policy keeps with a person.
Limits on retries and steps belong in the harness, not in the prompt. The practical details are in Runaway AI Agent Loops: Token and Step Budgets With Alerts, and what a clean stop looks like is described in Stop Conditions for AI Agent Loops: When to Halt and Escalate.
What does a routing decision look like in practice?
The following scenario is illustrative and does not describe a real client. A property management team uses an agent to answer tenant messages, and the team has defined three routes:
- A message asking when trash collection happens goes to the small model, which retrieves the schedule from the approved document and cites it. The harness confirms that the date appears in that document before the reply is sent.
- A message asking whether a late fee can be waived goes to the standard model, which drafts a reply from the written policy but may not promise a waiver. A staff member reviews the draft before it leaves the system.
- A message reporting that a unit has no heat goes directly to a person with a priority flag. The agent sends only an acknowledgment and the on-call contact, and does not attempt a fuller answer.
The useful part of this example is not the three labels. Each route states what it may do, what it must hand back, and what the log will show afterward. A reviewer can read the log and see why each message went where it went.
What should the routing log record?
If you cannot reconstruct why a request went to a particular route, you cannot defend the routing rules to a customer, an auditor or your own team. Each routed step needs a record that a person can read without opening the code. OpenTelemetry's semantic conventions for generative AI offer a useful starting vocabulary for model and token details, even if your own schema differs. At minimum, the record should contain:
- A request identifier and the time of the routing decision.
- The task type, the route chosen and the rule that selected it.
- The model name and version as the provider reports them.
- Whether each check passed, failed or was skipped, and the reason for any skip.
- The escalation reason, the receiving route or person, and the final outcome.
- Any fallback used because a provider or tool was unavailable.
Keep the log focused on decisions. Where possible, store references to the source content rather than copying personal details into every record, and apply the same retention rules you use for the underlying conversation.
How do you keep control when models change?
A routing table that worked last month can fail after a provider replaces a model, after you upgrade to a new version, or after a fallback takes over the traffic. Treat each of these changes as a release. Version the routing table, the prompts and the tool permissions together, then run the affected route's test set against the candidate before it handles live work. A canary release, where a small share of live traffic goes to the new configuration first, lets you compare outcomes before a full switch.
The test set should come from real requests your team has already reviewed, including the ones that escalated and the ones that caused trouble. Write the expected result for each case in advance. The general method of testing agents against expected behavior is described in the OpenAI evals guide and in Anthropic's writing on evaluations for agents. When a route fails, the fix may be a better rule, a narrower tool permission or a move to a stronger route, and the change should be logged with its reason. When a provider is unavailable, the harness should follow a documented path, as described in Model Fallback Routing for AI Agents: Keep Work Running Safely.
Be Human designs and operates this layer as part of managed operations. In a first engagement, we define the routes for one useful role, the memory and tools each route may use, the approval boundaries and the observable success criterion that shows whether routing is working. In ongoing operations, we keep the routing table under version control, run recurring evaluations when a model or provider changes, review tool permissions, and handle incidents when an escalation does not reach a person. The company decides what the agent may do, and we keep the evidence that those decisions are still being followed.
Where should a company start without overbuilding?
Start with one role and two routes. Choose a task that is frequent, checkable and low in risk for the small route, and a task with a high cost of error for the person route. Add a standard route only when the logs show that the small route keeps failing on a real class of cases.
- Pick one role and write its success criterion in one sentence.
- Write the two routes, their allowed tools and their escalation triggers.
- Run the logs for a short period and review every escalation by hand.
- Change a rule only when the logs show a repeated pattern, and record the reason for the change.
If you are not sure which of your tasks deserves a smaller model and which should stay with a person, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.
Frequently asked questions
Can a smaller model answer customer messages safely? It can, for narrow replies built from approved text that the harness checks before anything is sent. Replies that promise a refund, change a contract or interpret a policy exception should go to a stronger route or to a person. Start with the replies your team already sends without judgment, and widen only when the logs support it.
How do we know a route is good enough? Build a test set from real past requests, including the hard ones, and write the expected result for each. Run the route against that set before it handles live work, and again whenever the model or prompt changes. A route is good enough when it meets the success criterion you wrote at the start, not when its answers sound fluent.
What should happen when a provider is down? The harness should move the step to a documented fallback route or pause it with a clear status, and it should record that this happened. A silent failure is worse than a delay, because nobody learns that work was lost. Review fallback events in the same way you review escalations.
Who should own the routing rules? The manager who owns the outcome should own the rules, because that person decides what a wrong answer costs. The engineer implements the rules and keeps the logs working. Every change should be written down with a reason, so the team can see why a route moved and who approved the move.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


