From Anonymized Tickets to Agent Tests: Labels That Stay Current
Learn how to build an AI agent evaluation dataset from anonymized support tickets, label expected actions, and refresh the set as your business changes.

An AI agent evaluation dataset is a fixed set of real cases from your own work. Each case records what the agent could see, the action a competent person would expect, and the boundaries it must respect. Build it from anonymized support or operations tickets, label the expected action rather than the exact wording, and review the set on a schedule so retired policies stop counting as passes.
Key takeaways
- Start from closed tickets that already show what the business decided, not from prompts someone invented.
- Label the expected action, the allowed tools and the forbidden actions, so graders check outcomes instead of phrasing.
- Remove personal data before anyone outside the approved group reads a case.
- Version the set and retire cases when a policy, a tool or a product rule changes.
- Run the set before every change to prompts, tools or models, and keep each result in a log someone can read.
Why closed tickets make better test cases than invented prompts
An invented prompt shows what its writer imagined a customer might ask. A closed ticket shows what customers actually asked, which information was missing, which exception staff granted, and where the work went wrong. A ticket usually holds the request, the context, the steps a person took and the outcome. That is the raw material a test needs.
Closed tickets also contain the hard cases. Unusual requests, conflicting instructions and policy exceptions are rare, and they are exactly the cases that a prompt writer forgets. Pulling a normal period of tickets plus a handful that staff remember as difficult gives you coverage of both the common path and the failures that cost money or trust.
Anonymize tickets before anyone labels them
The labeling group should see only what it needs to judge the correct action. Anonymization is therefore part of the design, not a cleanup step at the end.
- Remove names, contact details, postal addresses, account and order identifiers, and free-text notes that contain personal information.
- Replace identifiers with consistent placeholders, so the same person keeps the same placeholder across cases and each case still reads as one story.
- Keep the business facts the agent needs, such as plan type, dates relative to the policy, product category and prior escalations.
- Exclude attachments you cannot clean, such as scanned documents or screenshots with visible personal details.
- Record who approved the export, where the cleaned file is stored and who can open it.
If a case cannot be cleaned without losing the decision that matters, leave it out of the shared set and keep it in a restricted file for the people who handled it.
Label the expected action, not the wording
Each case needs a label that describes what a correct run does. A useful label has five fields:
- Input snapshot: the request and the context the agent could see, plus the policy version in force on that date.
- Expected action: reply with specific information, create a record, route to a named role, or decline with a stated reason.
- Allowed tools: which tools may be called and under which limits.
- Forbidden actions: for example, issuing a payment above the approval threshold or sending a message that requires sign-off first.
- Pass criteria: an outcome a grader can check, such as exactly one record exists, the amount field matches the policy and the approval marker is present.
Graders should check outcomes and boundaries, not whether the reply matches a reference sentence. Two correct replies can differ in wording. A reply that promises an action the agent never took is wrong even when it reads well. Anthropic's guidance on evaluating agents makes the same distinction between judging the final state of a task and judging the text around it.
A step-by-step plan for the first version
Keep the first version small enough that the people who know the work can review every case and every disagreement.
- Choose one workflow with a single clear outcome, such as refund requests or appointment changes.
- Export closed tickets from a typical period, then add the difficult cases that staff still discuss.
- Anonymize the export and remove attachments you cannot clean.
- Have two people who run the work draft labels independently, without discussing them first.
- Compare the disagreements and write the rule that settles each one into a short label guide.
- Split the cases into a development set that the team can study and a held-out set that you run only before a release, so changes are not tuned to the test.
- Write the pass criteria and the procedure for running the set, whether manual or automated.
- Run the current agent once, record the results and keep that run as the baseline for every later comparison.
Illustrative scenario: refund requests in a subscription business
The following scenario is illustrative and does not describe a real company. A subscription business receives refund requests by email. Its policy allows a refund within a set window when proof of purchase is attached, sends other requests to a finance reviewer, and never refunds the same charge twice. Four cases from the export show what the labels must capture.
- A request inside the window with proof attached: the expected action is one refund issued through the billing tool, followed by a confirmation to the customer.
- A request just outside the window that describes a documented service failure: the expected action is a routed case for the finance reviewer, with a summary and no promise of money.
- A second request for a charge already refunded: the expected action is to report the existing refund and create no new record.
- A request on an account flagged for a chargeback review: the expected action is to stop, say that a person will follow up and send nothing about the dispute.
The second request is where a duplicate refund can appear if the refund tool is not designed for safe retries. That is why the pass criteria should count the refund records in the system, not only read the reply. For the design side of that check, see idempotent actions for refunds.
Keep the set current and decide who owns it
A test set goes stale quietly. A policy changes, a tool returns different fields or a product line is retired, and an action that was correct last quarter becomes wrong. Give every case a policy version and a named owner. When a policy changes, mark the affected cases for review instead of deleting them silently, and retire a case only with a written reason. Add new cases from each escalation the agent mishandled, so the set grows where the business actually fails. For a method on removing outdated cases, see retiring stale eval cases.
Be Human designs and operates this kind of set as part of managed operations. The engagement starts by agreeing with the business owner which workflow the set measures, what success looks like and which actions need human approval. Be Human then prepares the anonymized export with the business's approval, drafts labels with the people who run the work and keeps the set under version control. Recurring runs follow each change to prompts, tools or models, and results are reviewed alongside permission reviews, so the set covers what the agent is allowed to do as well as what it says. Where a model grades free-text answers, the grader is checked against human labels before anyone relies on it, as described in LLM judges for agent evaluations.
If you are unsure which queue would make the best first test, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.
Frequently asked questions
How many cases does a useful evaluation set need? Start with a set small enough that your team can review every case and every label disagreement, because a careful set of a few dozen cases is more useful than a large set nobody trusts. Add cases whenever a real failure appears that the set did not catch. Grow the set first in the workflows where a mistake is costly, not evenly across every queue.
Can we use customer tickets directly if the data stays inside the company? Internal use does not remove obligations for personal data. Remove personal details before labeling, limit who can open the file and confirm with your legal or privacy function how the export is treated where your business operates. Brazilian organizations can consult the guidance published by the ANPD, the national data protection authority.
Should an AI model grade the agent's answers? A model can help with free-text answers, but it must be checked against human labels before you rely on its scores. For actions such as refunds or record changes, check the system state directly instead of asking a model to read the transcript. Model graders are most useful as a second opinion on wording and tone.
How is an evaluation set different from monitoring live runs? An evaluation set is a fixed test you run before a change goes live, so the same cases give comparable results over time. Monitoring watches real runs as they happen and flags unusual behavior. You need both, and the cases from monitoring alerts are often the best candidates to add to the set next.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


