Model Routing for AI Agents: Match Each Task to the Right Model

Model routing for AI agents means choosing, for each type of task, the model that is capable enough, fast enough and permitted to handle it, instead of sending everything to one model. You define the task types, write a quality bar for each, assign a model to each type, and add a fallback for cases the first choice cannot handle. Then you test the routing itself, not only the models, so the decisions keep holding when inputs, providers or versions change.

Key takeaways

• Route by task type and risk, not by habit. A short, well-defined task rarely needs the same model as an open-ended, multi-step one.
• Write the quality bar first. A model is suitable only relative to a test the team can run and a result a person can accept.
• Treat permission and data rules as hard filters. Capability decides among the models that are allowed, never the other way around.
• Test the router as well as the models: misclassified tasks, ambiguous inputs, timeouts and fallbacks all need cases of their own.
• Measure cost per accepted outcome, and re-run your tests whenever a model, prompt or tool changes.

What does model routing mean for a business agent?

A business agent does many different kinds of work in a single day. It may read an incoming email, pull three fields from an attachment, summarize a long thread, draft a reply, check a policy, and decide whether to ask a person for approval. These tasks differ in difficulty, in the cost of a mistake and in how quickly someone expects an answer.

Using one model for all of it is simple, and it is a reasonable starting point. The trouble is that it hides a trade-off. If you pick a strong model for everything, simple tasks wait longer and cost more than they need to. If you pick a light model for everything, the hard tasks fail quietly, and a quiet failure in a business workflow is the expensive kind.

Routing resolves this by making the choice explicit. A routing layer looks at the task, decides which path it belongs to, and sends it there. Anthropic describes routing as a common workflow pattern in its guidance on [building effective agents](https://www.anthropic.com/engineering/building-effective-agents): classify the input, then direct it to a specialized follow-up. In a business setting the router is part of the harness, the layer around the model that holds instructions, tools, permissions and checks. If you want the vocabulary straight first, AI model, agent and harness: what is the difference? covers it.

Routing is not always needed. Some tasks do not need a model at all, and for those a fixed rule is safer. AI agent or deterministic workflow? A decision guide helps you separate the two before you start choosing models.

Which task attributes should decide the route?

Teams often route on a vague feeling that a task is easy or hard. A better approach is to describe each task type with a small set of attributes and let those attributes drive the choice. These are the ones that matter most in practice:

• Reasoning depth: does the task need several dependent steps, or one direct transformation?
• Cost of an error: what happens if the output is wrong, and who would catch it?
• Input size: how much context must the model read at once, and does it need to stay faithful to it?
• Output shape: must the result match a strict structure that another system will consume?
• Tool use: does the task call tools, and how many calls in sequence?
• Latency: is a person waiting, or does the work run in the background?
• Data sensitivity: which endpoints and regions are the company allowed to use for this data?
• Volume: how often does this task occur, and does repetition make small differences add up?

Most of these can be written into the task definition. A task contract your team can test already holds the inputs, outputs, limits and acceptance rules, so routing becomes a field in a document you already maintain, not a separate piece of tribal knowledge.

Notice that data sensitivity is a filter and not a preference. If a client's records may only be processed by approved infrastructure, then the set of eligible models is already smaller before you compare quality. Apply the filter first, then choose within what remains.

How do you match task types to models?

The list below is a decision guide, not a ranking of products. It describes the kind of model tier that usually fits each task type, and what must sit beside it. Adjust it to your own tests.

• Extracting fixed fields from clean, consistent documents: a smaller, faster model, with schema validation and a rule that rejects incomplete output.
• Labeling or classifying requests into a short list of categories: a smaller model with a confidence signal, escalating ambiguous cases to a stronger model or a person.
• Summarizing a long thread or document: a model that handles the full input reliably, with a check that every claim points back to the source.
• Drafting a message that a customer or partner will read: a stronger model for tone and judgment, with human approval before anything is sent.
• Planning across several tools and steps: the strongest reasoning tier you can justify, with step limits and checkpoints so work can resume if it stops.
• Arithmetic, totals and reconciliation: ordinary code for the calculation, and a model only to interpret messy input or explain the result.
• Anything touching restricted data: only the endpoints approved for that data, whatever the capability comparison says.
• A task type nobody has classified yet: default to the stronger model, log it, and review it so a proper route can be defined.

The last item matters more than it looks. A router needs a safe default for the unknown. If unfamiliar tasks fall to the cheapest path, you will learn about the gap from a mistake. If they fall to the stronger path, you learn about it from a log line.

### An illustrative scenario

Imagine a small distribution company whose operations inbox receives supplier emails (this is an illustrative scenario, not a real case). The agent first labels each email: delivery notice, invoice, complaint, or other. A smaller model handles this, because the categories are few and a wrong label is caught by the next step. For delivery notices, a smaller model extracts the date and reference into a fixed structure, and code validates the format. For complaints, a stronger model reads the full thread, drafts a reply and attaches the source passages, and a person approves it. For anything labeled other, the stronger model takes over and the case is added to a review list. Three routes, one default, and a person at the point where the cost of error is highest.

How do you test that routing decisions hold up?

Routing is a hypothesis: this task type is handled well enough by this model. Like any hypothesis it needs evidence, and the evidence comes from running the same cases through each candidate under the same scoring. This is the plan we recommend.

• Step one: list your task types and write an acceptance rule for each. If you cannot say what a good result looks like, you cannot choose a model for it.
• Step two: build a test set for each type from real, anonymized work, including the awkward cases: missing fields, mixed languages, contradictory attachments, requests that sit between two categories. Build a golden test set for your AI workflow walks through how.
• Step three: run every candidate model on every set with the same instructions and the same scoring. Use a rubric a reviewer could apply by hand, and spot-check automated scoring against human judgment. OpenAI's [evals guide](https://platform.openai.com/docs/guides/evals) is a useful reference for structuring this.
• Step four: choose the least expensive model that clears the bar with some margin, not the one that barely passes. A margin absorbs ordinary variation between runs.
• Step five: test the router on its own. Give it mislabeled, ambiguous and out-of-scope inputs, and check where each one lands. A perfect model behind a router that sends tasks to the wrong place still produces wrong work.
• Step six: test the failure paths. Simulate a timeout, a provider outage, a malformed response and a validation failure, and confirm that the fallback fires and that nothing is sent or changed by mistake.
• Step seven: run in shadow mode, where the agent produces results that people compare against their own work without anything going live. Run an AI agent in shadow mode before giving it authority describes the pattern.
• Step eight: repeat the whole sequence whenever a model version, prompt or tool changes. Pin the versions you tested so a silent upgrade does not change behavior unseen.

Two points deserve emphasis. First, the test set should include cases where the cheaper model is expected to fail, because that is what proves the escalation rule works. Second, a routing decision is not finished when the average looks good. Look at the worst cases within each task type, since a business pays for the failures, not the average.

What should you log and watch once routing is live?

The tests tell you the router worked on the cases you imagined. Production tells you about the rest. To learn from it, each completed task should leave a record that shows which task type it was assigned, which model handled it, whether a fallback or escalation occurred, how long it took, what validation said, and whether a person accepted or changed the result.

That is an observability problem, and the general ideas in OpenTelemetry's [observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) apply well: traces that follow one task through its steps, and consistent fields so records can be compared. For model calls specifically, the project is developing [semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/), which give a common vocabulary for recording them.

Once you have these records, a few questions become answerable:

• Which task types are escalating often, and should their default route change?
• Which routes produce results that people frequently edit?
• Are there task types appearing that no route was designed for?
• Has the mix of work changed since the last review?

Cost should be read the same way. The cost of a call is a poor guide, because a cheap call that needs rework costs more than a better one that gets accepted. The FinOps Foundation's material on [unit economics](https://www.finops.org/framework/capabilities/unit-economics/) points toward the right habit: tie spend to a meaningful unit of output. For an agent, that unit is an accepted outcome, such as a classified request that nobody had to correct.

Where does routing usually go wrong?

A few failure patterns repeat across business deployments.

The first is routing on the model's own confidence alone. Models can sound sure while being wrong, so confidence signals work best alongside external checks: schema validation, source matching, business rules and sampling by a reviewer.

The second is a router that is more complicated than the work. If you have two task types and one is trivial, a rule is enough. Add a learned classifier only when the categories are numerous or the inputs are too varied for rules. Every routing layer is one more thing to test and maintain.

The third is splitting the work across agents when routing would do. If one agent can do the job and only the model choice differs per step, routing inside one agent is simpler than coordinating several. Single-agent vs multi-agent systems: when to split the work explains when separation earns its overhead.

The fourth is forgetting that routing hides drift. When a provider updates a model, a route that passed last quarter may fail today without any change on your side. Scheduled re-evaluation, not memory, is the protection.

The fifth is ignoring permissions. A strong model should not receive broader access merely because it is more capable. Tool permissions and approval boundaries belong to the role, and they stay the same whichever model happens to be answering.

How does Be Human handle model routing in managed operations?

Be Human treats routing as part of the harness for a defined agent role. During a first engagement we describe the role's task types, the approval boundaries and the success criterion, then map each task type to an eligible model within the client's data and permission rules. Managed operations include recurring evaluations against the role's test sets, monitoring of escalations and failures, and controlled changes when a model, prompt or integration needs updating, with the client's team approving anything that alters what the agent may do.

Frequently asked questions

Do we need more than one model to start? Not necessarily. Begin with one capable model, write the task types and tests, and look at the records. If a clear group of simple tasks is slow or costly, or a group of hard ones is failing, that is the moment to add a second route.

Should the model decide its own route? It can help classify, but the decision should be checked by something outside it. Use rules for clear cases, a smaller classifier for varied ones, and a safe default to the stronger path when the signal is weak. Keep the logic visible so a reviewer can read why a task went where it did.

How often should we re-test routing? Re-test whenever a model version, instruction, tool or data source changes, and on a regular schedule even when nothing seems to have changed. Providers can alter behavior behind the same name, so scheduled runs on a stable test set are the earliest warning you have. Scheduled AI Agents: How to Run Recurring Tasks Safely covers running such jobs reliably.

What if the cheaper model passes our tests but people still correct its output? Then the tests are missing something. Add the corrected cases to the test set, tighten the acceptance rule, and re-run the comparison. The edits people make are some of the best evidence you will ever get about what good looks like.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.