Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Golden Datasets for AI Agents: Build a Real-Work Test Set

A golden dataset is a small set of labeled real company cases that every model or prompt change must pass before production. Here is how to build one.

Be Human typographic cover with the title: Golden Datasets for AI Agents: Build a Real-Work Test Set
Conceptual illustration created with AI · Be Human

A golden dataset for an AI agent is a small, labeled set of real cases from your own operations. Each case records the input the agent received, the outcome an experienced person would accept, and the limits that make that outcome acceptable. Every model change, prompt edit or tool update must pass this set before it reaches customers, invoices or records, so evaluation becomes a repeatable check instead of an impression.

Key takeaways

  • Build the set from work your team has already completed, not from invented prompts.
  • Label the accepted outcome and the boundary, so a reviewer can judge the result without guessing.
  • Fix the pass rule before running a change, including which failures block release.
  • Keep the set small, versioned and readable by the people who own the work.
  • Re-run the full set on every model, prompt, tool or policy change, and sample live output as well.

What a golden dataset is, and what it is not

A golden dataset is not a leaderboard and it is not a demo script. It is the minimum standard your business agrees on, written as cases the agent must handle correctly. Public benchmarks measure general ability. Your set measures whether the agent reads your quote format, applies your refund rule, uses the right customer record and stops when a decision belongs to a person.

Teams often start with a generic test suite because it is easy to find. The risk is that a generic suite can pass while the agent still fails on the exception that costs you a customer. A useful set is small enough that a manager can read every case in less than an hour, and specific enough that a failure is obvious when it appears. It does not replace monitoring after release. It is the gate that stops a known failure from returning. For a broader view of judging agents on operational work, see how to evaluate AI agents on real business tasks.

Start from cases your business already handled

The best source is work your team has already finished. Pull from closed tickets, approved quotes, resolved exceptions and the handoffs that went badly. Include ordinary cases, because most daily volume is ordinary, and add the awkward ones: an address written in two formats, a customer who switches between Portuguese and English in one message, or a supplier invoice that arrives as a scanned image with a stamp across the total.

Include cases where the correct answer is to stop and ask a person. Those cases test the boundary, and boundaries fail quietly when nobody tests them. Remove personal data the test does not need, and keep only what your organization is allowed to use for this purpose. The NIST AI Risk Management Framework treats measurement and documentation as part of managing AI risk, which is a good reason to write down where each case came from.

A step-by-step plan for the first set

Build the first set for one task with a clear owner and a clear finish line. Drafting a reply to a standard quote request is a better first target than an entire operations desk.

  • Pick one task, one owner and one finished output that someone can judge.
  • Export twenty to thirty completed cases from recent work, and remove personal data the test does not need.
  • Write each input exactly as the agent receives it, including attachments, timestamps and earlier messages in the thread.
  • Write the accepted outcome in plain words: the facts that must appear, the actions that must not happen, and the handoff that is expected.
  • Tag each case as ordinary, edge or stop-and-ask, so coverage gaps are visible at a glance.
  • Ask a second person to label a sample independently, and rewrite any label that two reviewers would read differently.
  • Store the set with a version number and a short note on what it covers and what it leaves out.

Keep the first version plain. A spreadsheet or a folder of text files is enough at the start. What matters is that each case can be read by the person who answers for the work, without a technical explanation.

Set the pass rule before you run anything

Decide what passing means before the first run. If you choose the rule after seeing the results, the rule tends to fit the result. Separate hard failures from soft issues. A hard failure is a wrong customer, a duplicated action, an invented price, a missing approval or a message sent without the required handoff. A soft issue is tone, length or formatting. An average score can hide the failure that matters, so any hard failure should block the release regardless of the overall result. Write the rule as one sentence per failure type and keep it next to the set, so a reviewer can see it without asking.

If you use a model to grade outputs, check the grader against human labels first. The article on LLM judges for business agent evaluations explains where automated graders mislead. A practical decision guide for each run looks like this:

  • A hard failure on any case stops the change, and the failing case stays in the set permanently.
  • If only soft issues appear, the process owner decides whether the wording is acceptable, and the decision is recorded with the version.
  • A case that passed before and fails now is a regression, even if the overall result improved.
  • A new failure found in production is added to the set before the fix is merged.

Run it on every change, and keep the set alive

Run the full set whenever the model version, system prompt, tool definitions, retrieval sources or business rules change. Tool changes deserve particular care, because a small edit to a tool description can alter which tool an agent chooses. The guide on regression checks for model updates describes how to decide whether to adopt, adjust or roll back a change.

Illustrative scenario: a distributor uses an agent to draft replies to standing-order changes. A team edits the prompt to make replies shorter. The new version passes the tone checks, but in two cases it drops the cut-off time for same-day delivery. The hard rule on cut-off times blocks the release. The prompt is corrected, the set is run again, and the change log records which case caught the problem.

Keep the set alive after release. When staff correct the agent, turn that correction into a case, as described in turning staff corrections into agent rules. When a policy changes, review every case that depends on it before the next run. Retire a case only with a written reason, so the set does not quietly become easier. Log each run with the version tested, the result, and the name of the person who approved any exception.

How Be Human handles the set in managed operations

Be Human designs the first agent role with its test set in place, starting from cases the business already handled and writing the approval boundary next to each one. In managed operations, the set stays current: it is re-run when models, prompts or tools change, reviewed during permission reviews, and extended after incidents. The work is remote, in English or Portuguese, and the people who know the cases best take part in labeling them. If you are not sure which task should go first, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

How many cases does a golden dataset need? Start with twenty to thirty cases for one task, chosen to cover ordinary work, edge cases and stop-and-ask moments. Coverage and label quality matter more than the count. Grow the set when a new failure appears, not on a fixed calendar.

Can an AI model grade the golden dataset? It can help with scale, but it should not be the only judge at the start. Compare its grades with human labels on a sample, and adjust the grading instructions until both agree on hard failures. Keep human review for any case where a wrong result would be costly.

How often should the set run? Run it on every change that can alter behavior, including model versions, prompts, tool definitions and policies. Run it on a schedule as well, because some drift comes from external systems the agent reads rather than from changes you make. A failed run should block release until someone records a decision.

Who should own the golden dataset? The owner should be the person who answers for the work, such as an operations lead or finance manager, not only the engineer who built the agent. Technical staff maintain the harness and the run process, while the business owner approves additions, retirements and pass rules. A named approver keeps the set from becoming something nobody reads.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →