Superintelligence Evals: Build Fifty Real Cases for Business Agents
How to build an AI agent evaluation set of fifty real business tasks, with expected outcomes, so each model or prompt change faces the same bar.

An AI agent evaluation dataset is a fixed set of real tasks from your own operation, each with an input, an expected outcome and a rule for judging the result. For a business agent, fifty well-chosen cases drawn from actual work are a practical starting point for showing whether a new model, prompt or tool change made the agent better, worse or only different. The goal is not a perfect score but a shared bar that every change is measured against.
Key takeaways
- Build the set from tasks your staff already completed, not from invented examples.
- Write the expected outcome and the judge rule for each case before you run any new version.
- Include routine work, hard edge cases, missing information and requests the agent must refuse or escalate.
- Freeze the set, version it, and rerun it on every model, prompt, tool or retrieval change.
- Group failures by cause so a drop in results points to something you can fix.
Why a fixed set of real cases beats a good demo
A demo shows that an agent can do something once, under favorable conditions. An evaluation set shows whether it still does the work your business needs after a change. Anthropic's engineering guidance on evaluating agents treats evals as repeatable tests with defined tasks and graded outcomes, and OpenAI's evals guide makes a similar case for writing down expected behavior before tuning. Both point toward the same discipline: decide what counts as success before you look at the new output.
Fifty is a useful size for a first set. It is large enough that one lucky answer cannot hide a regression, and small enough that someone in operations can read each case and argue about whether it is fair. Larger sets can come later, once the first one has proven that it reflects the work.
How do you gather candidate tasks without inventing them?
The raw material should be last month's work, not what the team imagines the agent will face. Ask the people who do the task which requests they repeat, which ones generate follow-up questions, and which ones they would hand to a new colleague with a warning. Then pull real examples from the ticket queue, the shared inbox, the order log or the call notes.
Build the candidate list in this order:
- Write the single role the agent will perform and the tools it may touch, in one short paragraph.
- List the recurring tasks that person completed during a typical week, and mark the ones that take the most judgment.
- For each task, copy a real request and the final outcome the team accepted, keeping the original wording where possible.
- Remove or mask personal data, account credentials and anything a customer would not expect to see in a test environment.
- Tag each example as routine, difficult, incomplete, out of scope or previously mishandled.
If you are unsure how to structure this, the approach in how to evaluate AI agents on real business tasks follows the same sequence from role to examples.
What mix of fifty cases reflects the real workload?
A set made only of easy requests will report success while hiding the problems that matter. Make the mix proportional to real volume for routine work, but make sure the rare, high-consequence cases are present even when they are uncommon. A useful starting shape has five groups:
- Routine tasks the agent should complete without help.
- Requests with missing or conflicting input, where the correct behavior is a clarifying question.
- Actions that cross an approval boundary and must stop for a person.
- Out-of-scope requests the agent should decline and route to the right owner.
- Past mistakes taken from the incident log or from staff corrections.
Illustrative scenario: a wholesale distributor's order desk uses an agent to draft replies to trade customers and propose order changes. Most of the candidate cases are standard reorders. The team then adds a request for a discount the agent is not allowed to grant, an order where two delivery addresses conflict, and a case where an order was already refunded once. Those last three cases test whether the agent respects its boundaries, which is the question the routine cases cannot answer.
Write the expected outcome and the judge rule before you run anything
Each case needs four things: the input, the context the agent is allowed to see, the expected outcome and a pass rule. The expected outcome should be concrete. For an action, it is the tool call that should happen and the key arguments it must contain. For a reply, it is the facts that must appear and the commitments that must not be made. Write these before you see the new model's output, because otherwise the bar tends to move toward whatever the new version produces.
Some checks can be deterministic: the correct record was updated, the approval request was created, no second write occurred. Others require judgment, such as whether a reply is complete or clear. For judgment-based checks, a model judge can help with scale, but its grades must be compared with a human reviewer on a sample before anyone relies on them. The method is covered in LLM judges for AI agent evals.
Where an action has side effects, test the decision without executing it. Recorded tool responses, sometimes called fixtures, let you check what the agent tried to do against a known state, without touching live systems. The OWASP guidance on authorization is useful here as a reminder that the agent's permissions should match the decision being tested, not the broadest access it could have.
Run the set on every change, and read the failures, not just the score
Before testing a candidate change, run the set against the current production configuration and store the results. Then run the candidate and compare case by case. An overall average can conceal damage: one lost approval boundary matters more than several slightly weaker phrasings. Group the results by category, so you can see whether the drop is in refunds, in escalations or in ambiguous requests.
Outputs can vary between runs, so sensitive cases deserve more than one run, and the spread between runs is worth recording. For each failure, write one line stating the likely cause: retrieval returned the wrong record, an instruction was ignored, a tool argument was wrong, or the judgment about escalation was off. A shared vocabulary for these causes is described in the failure taxonomy for business agents.
The decision after each run is one of three: adopt the change, adjust it and test again, or roll it back. The process for that decision is set out in regression testing for AI model updates.
A pass rate tells you that something changed. The failure notes tell you whether the change is acceptable.
Figure placement note for layout: the image sits after this section's predecessor.
How Be Human handles the set in managed operations
In a managed engagement, Be Human starts from the same role definition used in the first engagement, including the memory and tools the agent needs and the approval boundaries it must respect, and builds the case set from that definition. The cases are stored with versions, the judge rules are reviewed with the operations owner, and each model, prompt or tool change runs against the set before it reaches production. Recurring evaluations, permission reviews and incident response can keep the set aligned as the business and its tools change, so the bar stays the same even when the software underneath it does not.
Keep the set honest as the business changes
Evaluation sets go stale. Products change, policies are revised and staff find new edge cases that the original fifty did not anticipate. Add a case every time a real incident or staff correction reveals a gap, and retire a case only when the work it represents no longer exists, recording why it was removed. The NIST AI Risk Management Framework describes measurement and monitoring as continuing activities rather than a one-time launch step, which fits how a case set should be treated.
Two habits protect the set from quietly losing its value. First, do not edit the expected outcome of a failing case to match the agent's new behavior unless the original expectation was wrong, and record that decision. Second, keep the cases separate from the examples used to write prompts, because an agent tuned directly on its test answers will appear to improve without improving on the work. Corrections from staff are a good source for new cases, and the method for turning them into rules is described in turning staff corrections into agent rules.
Turn the first fifty into a decision
Once the set runs, you have a baseline, a list of failures and a clearer picture of which tasks the agent handles reliably. The most useful next question is which task should be handed to the agent first, and what it must prove before it receives any write access. A shadow run, where the agent proposes actions that a person reviews before anything happens, is often the natural bridge, and the steps are described in running an AI agent in shadow mode before giving it authority. If you are not yet sure which of your tasks would make the best first case, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.
Frequently asked questions
How many cases do I need for a first evaluation set? Fifty is a practical starting point for one agent role, because it is large enough to expose regressions and small enough for a person to review case by case. If the role covers very different task types, give each type a few cases rather than adding many to the largest group. Expand the set only after the first version has proven that it reflects the real work.
Can an LLM judge grade the cases for me? A model judge can grade judgment-based qualities such as completeness, tone or whether a reply followed a rubric. Deterministic checks, such as whether the right record changed or a single approval was requested, should not depend on a model's opinion. Before relying on any judge, compare its grades with a human reviewer on a sample of cases.
Should the evaluation cases contain real customer data? Use real work to shape the cases, but store them with personal data removed or masked, under the same access controls as other business records. Test actions against recorded tool responses rather than live systems, so a failing case cannot write to a real account. Review the set with whoever owns privacy obligations in your organization before it is shared outside operations.
How often should the set be run? Run it before every model, prompt, tool or retrieval change is deployed, and again on a regular schedule so that slow drift appears even when nothing was changed on purpose. Keep each run's results so you can see the trend over time, not only whether the latest run passed or failed.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


