Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Build an AI Agent Eval Set From Your Real Business Cases

Learn how to build a small AI agent evaluation dataset from past business cases, with expected outcomes, so model upgrades are tested on your own work.

Be Human typographic cover with the title: Build an AI Agent Eval Set From Your Real Business Cases
Conceptual illustration created with AI · Be Human

An AI agent evaluation dataset is a small, fixed set of real past tasks from your own business, each paired with the outcome a competent person would accept. You build it by pulling closed cases from your records, writing down the expected result and why it is correct, then running every new model or prompt against the same set before it touches live work. The goal is not a leaderboard score. It is a plain answer to one question: does the new version handle our work at least as well as the current one?

Key takeaways

  • Start from cases your team already closed, not from prompts invented for a demo.
  • Record the expected outcome and the reason it is acceptable, not only the final text.
  • Keep the set small enough to review by hand, and include the rare, costly cases on purpose.
  • Run the same set on the current agent and on every candidate before an upgrade reaches live work.
  • Retire stale cases on a schedule so the set keeps measuring the business as it is now.

Why public benchmarks miss your business

Public benchmarks measure general capability. Your agent works with your product names, your approval thresholds, your customer history and the exceptions your staff learned over years. A model can score well in general and still misread a supplier's invoice layout, or apply a refund rule that changed last month. Anthropic's engineering guidance on evaluating agents makes the same practical point: a useful eval starts from realistic tasks with clear success criteria, not from abstract quality impressions.

This is why an internal set matters most at upgrade time. When a provider releases a new version, the question is not whether it is better in general. The question is whether it still does your work correctly, and which of your tasks changed.

Which past cases should go into the set?

Do not sample at random from the archive. Choose across four groups so the set reflects where the business can actually be hurt:

  • Frequent tasks, the ones the agent will repeat dozens of times a week, where small quality shifts add up.
  • Expensive tasks, where a wrong answer means a refund, a double booking or a wrong invoice.
  • Risky tasks, where the agent must stop and ask a person, such as a request outside the approval limit.
  • Tricky tasks, the cases where a previous employee had to make a judgment call or where the source document was messy.
A good set contains the cases that would have made a new employee ask for help.

Illustrative scenario: a small equipment distributor wants an agent to draft replies to refund requests. The operations lead pulls a few dozen closed tickets from the past months. She includes routine refunds, two disputes that went to the account manager, one request from a customer whose order was partly shipped, and a case where the original invoice was a scanned image. The set is not a statistical sample. It is a deliberate map of the work.

How do you write an expected outcome?

Each case needs more than an input and a model answer. Write the record in a form that a reviewer, or a judge, can check without reconstructing the situation. A workable case file includes:

  • The input exactly as it arrived, including attachments and any message thread.
  • The context the agent was allowed to see at that moment, such as order status or the policy version in force.
  • The expected action, for example "draft a refund confirmation" or "escalate to the account manager".
  • Acceptable variants, such as two wordings that both respect the policy.
  • Forbidden outcomes, such as promising a delivery date the warehouse did not confirm.
  • The reason, written in one or two sentences, so the case stays understandable after the person who wrote it has moved on.

The reason field is the part teams skip, and it is the part that makes the set useful later. When a case fails after an upgrade, the reason tells you whether the agent broke a rule or whether the rule itself has changed.

A step-by-step plan for the first set

The first version should be small enough to finish in a focused week. A practical sequence:

  • Step 1: Name the single task the agent will own, with its approval boundary written beside it.
  • Step 2: Pull closed cases from the last several months, and ask the people who handled them which ones were hard.
  • Step 3: Remove or redact personal data, and keep only what the task needs. Store the set with the same access rules as the source records.
  • Step 4: Write the expected action, acceptable variants, forbidden outcomes and reason for each case.
  • Step 5: Have a second person from the team review a sample of the expectations. Disagreements usually reveal an unwritten rule.
  • Step 6: Run the current agent on the full set and record which cases pass, which fail and why.
  • Step 7: Freeze the set under a version number, so any future comparison uses the same cases.

The NIST AI Risk Management Framework treats measurement and documentation as part of managing AI risk, which is why the version number and the reasons matter as much as the scores.

How should results be scored?

Separate hard constraints from quality. A reply that breaks a refund limit or sends data to the wrong customer is a failure, full stop, regardless of how well it is written. Quality dimensions such as clarity, completeness and tone can be graded on a simple scale, and they should be reviewed by people for the first few rounds.

Automated judges, including language models used as graders, can speed this up, but they need their own calibration. Check a sample of their verdicts against human decisions before trusting them on a case set. The OpenAI evals guide describes this general pattern of defining tests, running them and reviewing results, and it is a useful reference for structuring the process. For a deeper treatment of judge reliability, see LLM judges for agent evals.

Also track cost and time for each case. A candidate that passes every case but takes four times as long, or needs many more calls per task, may still be the wrong upgrade for your operation.

How to run an upgrade decision

When a new model or prompt arrives, run the baseline and the candidate on the same frozen set, then compare case by case rather than comparing averages. Read regressions first. A candidate that improves twelve cases but breaks one approval rule should not ship until someone understands that break. Then look at new passes, because they can show where the candidate is genuinely better and where a new capability is worth using.

Regressions also tell you what to fix in the agent's rules or tools before the next attempt. Keep a short written decision for each upgrade: what changed, what the cases showed, what was accepted and who approved it. Before a candidate with real authority goes live, it is usually wise to observe it in shadow mode first. See running an AI agent in shadow mode before giving it authority.

The set itself also needs upkeep. Policies change, products are renamed and old cases describe processes nobody follows anymore. Schedule a review, remove cases that no longer reflect the work, and add new cases from recent escalations. The mechanics of retiring outdated cases are covered in eval sets for AI agents and how to retire stale cases.

How Be Human approaches this in a managed engagement

In a Be Human engagement, the eval set starts from the first role's defined task and approval boundaries, not from a generic template. The team works with the people who handled the real cases to select them, write expected outcomes and mark the forbidden actions. Candidate models and prompt changes are run against the frozen set before they reach production, and the results are recorded in a readable change log. During managed operations, the set is kept current through recurring evaluations, permission reviews and retirement of stale cases, so the business keeps one reliable answer to the upgrade question over time.

Where to start

If you are not sure which of your tasks would make the best first case set, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

How many cases do I need in the first evaluation set? Start with enough cases to cover the frequent, expensive, risky and tricky groups for one task, which is often a small set that one person can review in a day or two. A larger set is useful later, but a small set that is accurate and well documented is worth more than a big one full of unclear expectations. Add cases as you learn where the agent fails.

Can I use real customer records in the evaluation set? You can, but only after removing or redacting personal data that the task does not need, and only with the same access controls you apply to the source system. Many teams keep a separate version of the set built from synthetic records that mirror the structure of real cases. Check your own legal and contractual obligations before copying records into any new location.

Who should write the expected outcomes? The people who did the work should be involved, because they know the unwritten rules and the exceptions that a manager may have forgotten. A second reviewer should check a sample of the expectations to catch disagreements early. Automated judges can help with grading later, but they should not write the ground truth themselves.

What if a new model passes more cases overall but fails one important case? Do not ship on the average. Treat any failure on a hard constraint, such as an approval limit or a data rule, as a blocker until someone explains it and decides whether the rule, the tool or the agent needs changing. Record the decision and the reason so the next upgrade starts from a clear history.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →