Internal Benchmarks for AI Agents: How to Build Your Own Test Set

An internal benchmark for AI agents is a fixed set of real tasks from your own business, each with a written definition of a good result, that you run again whenever the model, prompts, tools or permissions change. It lets you judge a newer model against the work you actually need done, rather than against public leaderboards built from tasks you never chose. You build it from work your team has already done and knows how to judge, then keep it under review.

Key takeaways

• Public benchmarks measure general ability. Your benchmark measures whether an agent completes your work correctly and within your rules.
• Start from tasks your team has already completed, using records that show what a good outcome looked like.
• Write pass rules before the agent runs, and separate hard failures from differences in quality.
• Include cases where the right behavior is to stop, ask or hand over to a person.
• Rerun the same set on every model, prompt, tool or permission change, and keep a log of each run.

Why public scores do not answer your question

Model releases arrive with tables of results on general reasoning, coding and knowledge tests. Those results help compare vendors at a high level, but they do not tell you whether an agent will read your supplier emails correctly, respect your refund limits or notice when a customer record conflicts with the contract. A higher score on someone else's test can coexist with a regression in the one workflow your operation depends on.

The NIST AI Risk Management Framework treats measurement as an ongoing activity tied to the context where a system is used, not as a one-time certificate. For an operations leader, that becomes a practical question: what would we notice on a Monday morning if this agent got worse? Your internal benchmark is the instrument that answers it.

Choose tasks that carry real weight

Start from the list of tasks the agent is meant to handle, then test each candidate against four questions. How often does it happen? What does a wrong result cost? Can two reviewers agree on the right answer quickly? Does the answer depend on your records, documents or rules rather than general knowledge? Tasks that score high on frequency and cost belong in the set first, even when they feel routine.

• Frequency: the task happens every week, not once in a long while.
• Cost of error: a mistake creates a complaint, a payment problem, a compliance issue or a rework cycle.
• Verifiable answer: trained people would agree on whether the output passed.
• Context dependence: the right answer depends on your systems, policies or channels.
• Coverage: the set includes ordinary cases, messy inputs and a few known edge cases.

The best source is a folder of past work where the outcome is already known, such as tickets resolved well, orders shipped correctly or quotes that were accepted. Old work gives you ground truth without asking anyone to invent examples.

Turn each task into a case you can rerun

A case is a single test with everything needed to run it again. Store the input exactly as the agent would receive it, the context it is allowed to see, the tools it may call, the expected result and the rule for passing. Where the source contains personal data, anonymize it or replace it with synthetic records that keep the same structure and edge conditions. Each case record should include:

• A case identifier and task family, so results can be grouped and compared.
• The input as received, including attachments and imperfect formatting.
• The context the agent may use, such as the policy version or a snapshot of the customer record.
• The tools it may call and the approval a human must give before any action is taken.
• The expected outcome, written in plain words, with any exact values recorded.
• The pass rule and the named person who owns the case.

For most teams, twenty to forty cases per task family is a reasonable starting point. That is enough to reveal patterns without turning the benchmark into a project of its own. Evaluation guides from model providers, such as the OpenAI evals guide, describe the same basic structure: a dataset of inputs, a method for grading outputs, and a run that can be repeated under the same conditions. For a step-by-step approach to the dataset itself, see how to build a golden test set for your AI workflow.

Write pass criteria another person can apply

A criterion such as the answer should be good produces arguments, not measurements. Split your criteria into two layers. The first layer is hard gates: outcomes that always fail the case, such as sending a message to the wrong recipient, taking an action without the required approval, inventing a figure that is not in the source, or ignoring an instruction to stop. The second layer is graded quality: completeness, tone, use of the correct template and whether a summary is useful to the person who reads it.

Keep the graded layer simple. A three-level scale for each dimension is usually enough for an operations team to agree on and reuse. Anthropic's writing on evaluating agents points toward grading what the agent did in the environment, not only what it said. In practice, that means checking whether the order exists in the system with the right lines, not whether the agent reported that it had been created.

Record who graded each case and why. When two reviewers disagree on a case, the rule was probably unclear, so rewrite the rule before you rerun anything.

Include the cases where the right answer is to stop

Many agent failures are not wrong answers but wrong persistence. The agent keeps trying, guesses a missing value or follows a conflicting instruction without flagging it. Add cases where the correct behavior is to ask, pause or hand the matter to a person. Examples include a refund request above the approved limit, a contract with a missing signature block, or two records that disagree about a delivery address.

Score these cases as passes only when the agent stops at the right point and gives the reviewer enough context to decide without reading the whole history. Escalation quality is part of capability, and a benchmark that rewards confident completion of everything will reward the wrong behavior. For the handover format, see human escalation for AI agents, which sends the decision rather than the mess.

Run the baseline, then compare the candidate

The benchmark only means something when you compare runs under the same conditions. Follow this sequence each time you evaluate a change:

• Freeze the current setup: model version, prompt, tool list, permissions and memory sources.
• Run the full set on that setup and record every result, including hard gate outcomes.
• Change one thing at a time, such as a new model or a revised tool description.
• Run the same cases with the same grading rules and, where possible, the same reviewers.
• Compare results case by case, not only by average, and read regressions before improvements.
• Decide whether to adopt the change, adjust it, or keep the current setup.

Illustrative scenario: a small distribution company tests a newer model for its order confirmation agent. The overall results look better, but one task family, orders with split shipments, now fails a hard gate because the agent sends confirmations before stock is allocated. The team keeps the current configuration for that family, rewrites the tool instruction, and reruns only that family after the fix. The average hid the regression, which is why case-level results matter. For the adopt, adjust or roll back decision itself, see AI model update regression testing.

Keep the set current and decide who owns it

A benchmark decays when the business changes and the cases do not. When a policy is revised, a supplier changes its format or a team adds a new product line, review the affected cases and retire the ones that no longer reflect real work. Keep a short note on each change so future reviewers understand why the expected outcome changed. Between formal runs, sample live outputs each week so that quality problems appear before they reach a benchmark. A practical method for that sampling is described in how to sample AI agent outputs for quality each week.

Be Human's managed approach treats the benchmark as part of operations rather than a one-off project. During the first engagement, the useful role, the memory and tools it needs, the approval boundaries and the success criterion are defined with the business owner, and real tasks from that role become the initial cases. During managed operations, the same set is rerun when models, prompts, integrations or permissions change, failures are logged with enough detail to trace them, and cases are reviewed as processes evolve. The business owner keeps the decision to adopt, adjust or roll back a change.

If you are not sure which of your tasks should anchor the first benchmark, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

How many tasks should an internal benchmark include? Start with one or two task families that matter most, with twenty to forty cases each, rather than a broad set that nobody has time to review. Add families only after the first set is stable and the team trusts the grading. Larger collections become useful later, but they are harder to keep accurate.

Who should grade the results? The person who owns the process should define the pass rules, and at least one other trained reviewer should check a sample of grades to keep them consistent. Where the check is objective, such as whether a field was filled with the correct value, a script can perform it. Judgment about tone and usefulness still needs a person.

Can we use real customer data in the test set? Only where your data rules allow it, and with the least personal information needed to run the case. Anonymize names, contact details and identifiers, or create synthetic records that keep the same structure and edge conditions. Store the set with the same access controls that apply to the source systems.

How often should we run the benchmark? Run the full set before any change that could alter behavior, including a model update, a new tool, a revised prompt or a permission change, and run a smaller sample on a regular schedule to catch drift. The right rhythm depends on how often your systems change. Keep a short log of every run so you can explain later why a decision was made.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.