Evaluation Sets for Superintelligent Agents Built From Real Work
Build an AI agent evaluation set from real past tickets and tasks: redact examples, write pass criteria, and keep the set current as the work changes.

An evaluation set for an AI agent is a fixed collection of realistic tasks, each paired with the input the agent receives, the context it is allowed to use, and a written rule for what a correct result looks like. Build it from real past tickets, requests and handoffs with personal details removed, then score every change to the agent against the same set before that change touches live work.
Key takeaways
- Start from work that already happened. Past tickets show the requests, edge cases and wording your team actually sees.
- Redact before you copy. Keep the structure of the task and the decision point, and remove names, account numbers and anything the agent does not need.
- Write pass criteria that a second person could check. If two reviewers would disagree about a result, the rule is not specific enough yet.
- Version the set, keep it small enough to run after every change, and add new cases from real failures.
Why real past work beats invented test prompts
Many teams begin by writing a list of questions they imagine customers asking. Those questions are useful for a first smoke test, but they rarely match the messy version of the work: the order number pasted into the middle of a complaint, the attachment that arrived as a scanned image, the request that depends on something agreed in a phone call last month. Real past work contains that mess. It also contains outcomes. Someone decided, someone issued a refund, someone escalated, someone wrote a reply the customer accepted. Those decisions are your reference points, and no invented prompt can supply them.
Past work also shows how often each kind of task appears. A rare but costly case deserves a place in the set even if it appears only occasionally, because a failure there can do real damage. A common, cheap case deserves a place too, because a regression there will be noticed quickly by the people who depend on the agent every day. The goal is not to replay history exactly. The goal is to give the agent the same kinds of inputs your business really produces, in roughly the same proportions.
This matters most for roles where the agent reads or writes records. A test prompt that says "check the status of my order" tells you very little about whether the agent can handle a record that was updated twice, cancelled in one system and still open in another. Past tickets contain those states because they happened.
Step by step: build the first version
Use this sequence for a first version. For one agent role, it should take a focused working session or two, not a quarter of project work.
- Pick one agent role and one decision it makes, such as answering delivery status questions or drafting a first reply to a quote request.
- Export a sample of past tasks of that type from the system where the work happened. Include ordinary cases, difficult cases and cases that were escalated.
- Group the sample by outcome: resolved with the action the agent would propose, resolved by a human with a different action, escalated, or closed without action. The groups show which decisions carry the most weight.
- Redact each case, then freeze three things: the input, the context the agent would have had at that moment, and the reference outcome.
- Write one pass criterion per case. State what the output must contain, what it must not do, and which tool actions are allowed.
- Record why each case is in the set. A case without a stated reason becomes noise, and nobody can tell later whether it still matters.
- Run the current agent on the whole set and save the run with its model version, prompt or configuration version and the set version.
- Review the disagreements with the people who handled the original work, then tighten the criteria or remove cases that were genuinely ambiguous.
The saved run is as important as the set. Without it, you cannot compare the next version against this one or explain why a score moved.
How to redact examples without losing the signal
Redaction should remove what identifies a person or a company, and what the agent does not need to make the decision. It should keep the shape of the task: the request type, the state of the record, the policy that applies, the timing and the wording problem. A ticket that reads "Maria Souza from Acme asked about order 88213, which was shipped to the wrong branch" can become "Customer asked where an order went after it was sent to the wrong location. Order is in shipped state. Policy allows rerouting within the same region without approval." The second version keeps everything a reviewer needs to judge the answer.
Replace identifiers with stable placeholders such as CUSTOMER_A, so the same person appears consistently across related cases and the agent can be tested on follow-up messages. Keep timing relative when it matters, for example "the parcel was collected before the request was made", rather than storing exact dates that could identify someone. Keep the tool outputs the agent would have seen, but strip account numbers, payment details and free text containing personal data that the decision does not depend on.
Check each redacted case by asking whether a reviewer who never saw the original could still say what a good answer looks like. If not, the redaction removed too much. If the case still contains sensitive details, it removed too little. Confirm the handling rules for personal data with your legal or privacy contact before the set leaves the system of record, and limit access to the people who need it.
Writing pass criteria a business owner can check
A pass criterion is a short, testable statement attached to each case. Good criteria name something observable: "The reply states the current shipping status from the record." "The agent does not promise a refund date." "The case is routed to a human because the amount is above the approval limit." Weak criteria say things like "the answer is helpful" or "the tone is professional". Those qualities matter, but they need to be turned into checks that someone else can apply the same way.
Separate three kinds of checks. Content checks confirm that the right facts appear and that no wrong facts appear. Action checks confirm which tools the agent called, with which arguments, and whether it stayed inside its permissions. Boundary checks confirm that the agent stopped where it should, asked for approval or escalated. Many failures that look like poor writing are really action or boundary failures. Scoring the three kinds separately makes the report useful to the person who has to decide whether the agent can be trusted with more work.
Anthropic's guidance on evals for agents describes grading agents on outcomes and transcripts rather than on a single answer, which fits this structure well (see the source list below). When a criterion depends on judgment about free text, some teams use a second model as a grader. That can work, but the grader needs its own calibration against human decisions before its verdicts count. The LLM Judges for AI Agent Evals: Where They Mislead and How to Check Them article covers the usual failure patterns.
Illustrative scenario: a delivery question that needs a person
Illustrative: a small furniture retailer wants an agent to answer delivery questions from customers. In past work, most messages asked for tracking status. A smaller group involved a damaged item, a request to change the delivery address after dispatch, or a complaint about a missed appointment. The team built a set of a few dozen cases from the previous months, redacted them, and wrote criteria for each one.
The address-change cases carried a boundary criterion: the agent must not confirm the change, must send the order state and the customer's request to the staff queue, and must tell the customer that a person will reply. On the first run, the agent answered the status questions correctly and handled the damage reports well, but it confirmed the address change in two of the cases. The team did not start by rewriting the prompt. They first removed the agent's permission to edit the delivery address, added the boundary check to the set so the same error would be caught automatically, and reran the set. The next run passed the boundary checks, and the team had a record of what changed and why.
In Be Human's managed model, the first engagement defines the role, the memory and tools it needs, the approval boundaries and the success criterion, and the evaluation set is built from that same material rather than from a separate wish list. Operations then keep the set current through recurring evaluations, permission reviews and incident response, so changes to models, prompts or connected systems are checked against the same cases before they reach customers. The business keeps the decisions about what the agent may do.
Keeping the set current
An evaluation set goes stale. Policies change, products change, the underlying model is updated, and new kinds of requests appear. The risk management guidance from NIST treats measurement of AI systems as an ongoing activity rather than a one-time check, and an evaluation set is one of the practical tools for that. Three habits keep it useful.
First, add every real failure that reaches a human as a new case within the same week, with the corrected outcome written down. Second, retire cases whose underlying policy no longer exists, and record why they were removed so the history stays readable. Third, version the set the same way you version the agent. Each run should reference the set version, the prompt or configuration version and the model. When a score drops, that record tells you whether the agent changed, the tools changed, the policy changed or the test itself changed.
Before an agent gets write access to any system, run it in shadow mode, where it proposes actions that a person reviews without anything being executed. The shadow mode pilot guide explains how to compare those proposals against the same reference outcomes. If you are not sure which of your current tasks would make a good first case, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.
Frequently asked questions
How many cases does a first evaluation set need? Start with a set small enough that a person can review every case and the agent can run it after each change. Cover the common request types first, then add the failures and edge cases your team already remembers. The size matters less than whether each case has a stated reason and a criterion someone else can check.
Can I use last month's agent answers as the reference? Not directly. Past agent output shows what the agent did, not what was correct. Use the outcome the human team reached, or a corrected version that a reviewer approved, as the reference for each case.
Is it safe to put real tickets into an evaluation environment? It depends on the data and on the systems involved. Redact personal data before the set leaves the system of record, limit access to people who need it, and use test fixtures or read-only copies whenever the agent would otherwise write to a live system. Your legal or privacy contact should confirm the rules that apply to your business and your customers.
How often should the set be run? Run it before any change to the model, prompt, tools or connected systems, and on a regular schedule for the production agent. Add a run after any incident. The schedule should match how quickly your business changes, not an arbitrary calendar.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


