Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Golden Test Sets for Superintelligent Agents: Build One This Week

Build a golden dataset for AI agent evaluation from real tickets. Write expected outcomes and score every model or prompt change the same way.

Be Human typographic cover with the title: Golden Test Sets for Superintelligent Agents: Build One This Week
Conceptual illustration created with AI · Be Human

A golden dataset for AI agent evaluation is a small, fixed collection of real cases, each paired with the outcome a correct agent must produce and a rule for judging it. You run the same collection after every model change, prompt edit or tool update, so results are compared with each other rather than remembered from a demo. A useful first version can be assembled in a week from resolved tickets, operations requests or customer questions your team already handled.

Key takeaways

  • A golden set is a fixed, versioned list of real cases with expected outcomes, not a list of impressive prompts.
  • Start from resolved tickets, because they show the questions people actually ask and the answers your business already accepted.
  • Write each expected outcome as something you can observe: the action taken, the fields extracted, the escalation made, or the answer withheld.
  • Score every run with the same rules, and rerun the full set before any model, prompt or tool change goes live.
  • Keep the set small enough that a person can review it, and change it only through a logged edit.

Why one test set beats a pile of demos

Most teams test an agent by typing a few messages they can think of and reading the replies. That works until someone switches the model, edits the system prompt to fix one complaint, or adds a new tool, and the agent quietly gets worse at something nobody tried that week. A demo measures what the tester remembered to try. A golden set measures what the business has already decided is correct.

The value is consistency. When the same cases run before and after a change, differences in results come from the change rather than from the tester's mood, the phrasing they happened to use or the order in which they tried things. That consistency is what lets an operations leader ask a precise question: which cases got better, which got worse, and is the trade acceptable?

What goes into a golden case

Each case needs four parts: the input exactly as it arrived, the context the agent would have had (the state of the customer record, the order status, the policy version in force), the expected outcome, and the reason that outcome is correct. Keep inputs unedited, including typos, mixed languages and forwarded threads, because real traffic contains them. Remove personal data or replace it with fictional identifiers before the set leaves the systems where it was created.

Expected outcomes should describe behavior, not wording. "Declines a refund above the approval limit and routes the case to the finance queue" can be checked. "Responds politely and helpfully" cannot. Include negative cases as well: requests the agent must decline, questions it must hand to a person, and inputs ambiguous enough that the right move is a clarifying question. Agents tend to fail most visibly when they act on something they should have left alone, so those cases deserve as much attention as the routine ones.

A step-by-step plan for the first set

Work through these steps in order. A small team can finish them in a few focused sessions.

  • Pick one workflow with a named owner and a steady flow of resolved work. Do not start with the whole company.
  • Export the recent resolved cases for that workflow and sort them into four groups: handled correctly, handled with rework, escalated, and mishandled.
  • Select cases from every group, including several where a person had to correct the outcome. Those corrections show where the rules are unclear.
  • Write the expected outcome for each selected case in one or two sentences, using observable actions and fields.
  • Attach the policy or document that makes each outcome correct, so a reviewer can check it later without asking around.
  • Have a second person write or review the expected outcomes independently. Disagreements usually reveal a policy gap, not a scoring problem.
  • Store the set in a versioned location with a date, an owner and a short change note.
  • Run the current agent against the set once and record the result as the baseline before changing anything.

A shorter format guide for this step is available as how to build a golden test set for your AI workflow. The point of the baseline is that every later change is compared against something fixed, so do not skip it even when the first results look disappointing.

How to score each case without a panel of experts

Scoring needs three outcomes: pass, fail with a known reason, and needs review. A pass means every required element of the expected outcome appears and no forbidden action occurs. A fail records which element was missing or which forbidden action happened. Needs review covers cases where the agent's answer is defensible but differs from the expected outcome, and those are often the cases where the expected outcome itself needs updating.

Automated checks handle structured outcomes well: the right record was touched, the right field was set, the escalation queue received the case. Free-text answers need either a short rubric a reviewer can apply in under a minute or a model-based grader. If you use a model to judge answers, check the grader against cases a person has already scored, because a judge can share the blind spots of the agent it grades. The guide on LLM judges for agent evaluations, calibrated before you trust them explains how to run that check.

Keep the report simple: how many cases ran, how many passed, which failures are new compared with the last run, and failures grouped by reason. A change becomes a candidate for release only when the new failures are understood and the owner has accepted them in writing.

Illustrative scenario: a refund-question agent

This scenario is illustrative and does not describe a real client. Suppose a support agent answers questions about delayed orders and can issue store credit up to a fixed limit. The team builds a golden set of about twenty cases. It includes a delay still inside the promised window, which needs the standard answer; a delay past the window, where store credit within the limit is correct; a request above the limit, which must route to finance; a Portuguese message about a cancelled order, which must be answered in Portuguese; and a claim of a missing delivery, where the agent must ask for the tracking reference before any credit is considered.

The baseline run passes most cases but fails the Portuguese cancellation case, because the agent replies in English. The team adds a language-matching instruction to the prompt and reruns the set. That case now passes, but one store-credit case routes to finance by mistake, because the new instruction made the agent more cautious about the limit. Without the golden set, the team would have shipped the change and learned about the second failure from a customer complaint. With it, the regression is visible before release, and the owner decides whether to adjust the instruction or the expected outcome, with a note explaining which.

Keeping the set current, and how Be Human handles it

A golden set drifts. Policies change, products change, and new kinds of requests appear. Review the set on a regular schedule, add cases from new failures and escalations, and retire cases whose underlying policy no longer applies. Every edit should carry a note saying why, because an unexplained change to expected outcomes makes past results impossible to compare. The same set supports decisions about model updates, and the guidance on regression testing for model updates shows how to turn results into adopt, adjust or roll back decisions. Versioning the set itself is covered in golden task sets as a versioned test bed.

Be Human treats the golden set as part of each agent's operating record rather than a one-time test. In managed operations, the cases are built from the client's resolved work, kept under version control with named owners, run before each approved change, and reviewed alongside monitoring and permission reports. Access to the cases is limited to what each agent needs, and any change to expected outcomes goes through the same approval path as a change to the agent itself.

Where to start

Choose the workflow where a wrong answer costs your team the most time, and build its first set this week, using resolved work you already have. Keep the set small, write outcomes that someone can check, and record the baseline before you change anything else. If you are not sure which workflow deserves the first set, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.

Frequently asked questions

How many cases does a golden dataset need? There is no fixed number that works for every workflow, but the set should be small enough for one person to review in a sitting and broad enough to cover routine work, edge cases and cases the agent must refuse or escalate. Coverage matters more than volume. A modest set with clear expected outcomes is more useful than a large one that nobody reads.

Can we build a golden set without engineers? Yes, for most business workflows. The owner of the process can export resolved work, select representative cases and write expected outcomes. Engineers are most useful for automating structured checks, such as confirming that the right record changed or the right queue received a case.

How often should we run the set? Run it before every model change, prompt edit or tool update, and on a regular schedule even when nothing has changed, because policies and customer behavior shift over time. Scheduled runs are also how gradual quality decline becomes visible before it reaches customers.

Is it safe to use real tickets as test cases? It can be, provided you remove or replace personal data, limit access to people who need it, and store the set under the same controls as the source records. Treat the set as sensitive business material, not as a disposable spreadsheet, because it describes how your company decides things.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →