Golden Sets for Business Agent Evaluations: Build One in a Week
A golden set is a small, fixed list of real business cases with agreed outcomes, the baseline every new agent version must beat before it ships.

A golden set for business agent evaluation is a small, fixed collection of real cases, each with the input the agent receives, the outcome a competent person would accept, and a short note on why that outcome is right. Build one in a week from work your team already does, score every new model, prompt or tool change against it, and ship a version only when it matches or beats the current baseline on the cases that matter most.
Key takeaways
- A golden set is a baseline for your work, written with your rules, not a public benchmark that says little about your customers.
- Draw cases from real requests and past decisions, then keep the edge cases that have caused trouble before.
- Write each expected outcome so a second reviewer can check it without asking the author what they meant.
- Score every candidate change against the same frozen set, and record which cases changed and why.
- Retire stale cases on a schedule so the baseline keeps describing the business as it works today.
What a golden set is, and what it is not
A golden set is the reference a change must not break. It is not a leaderboard, a demo script or a pile of chat logs. A public benchmark tells you how a model performs on someone else's tasks. A golden set tells you whether the agent still handles your refund requests, your supplier emails and your account changes the way your operations team expects, and it does so with the same cases every time.
The difference matters because agents change in quiet ways. A new model version, a revised prompt, a different tool schema or a changed memory policy can each improve one behavior while damaging another. Without a fixed reference, the team judges each change by the example that happened to come up in review that day. With a golden set, the question becomes concrete: which cases got better, which got worse, and is any worsening acceptable for this role?
If you already keep a broader evaluation collection, the golden set is the smaller, stricter core of it. For a wider discussion of building internal tests from operations data, see Build an AI Agent Eval Set From Your Real Business Cases.
A one-week plan to build the first set
The first version does not need to be complete. It needs to be specific enough that someone other than the builder can run it and reach the same verdict. A practical sequence for one role looks like this:
- Day one: name the single role, its inputs, the tools it may call and the actions that require a human approval. Write the success criterion in one sentence.
- Day two: pull recent real requests for that role and group them by type, value and risk. Note which types the team handles by hand today.
- Day three: select the cases. Include the common path, the expensive mistakes, the known edge cases and at least one request the agent must refuse or escalate.
- Day four: write the expected outcome for each case, including the exact fields, the forbidden actions and the pass rule.
- Day five: run the current agent against the set, have a second person review the disagreements, fix any case that was unclear, and freeze the version as the baseline.
Remove personal data before the cases leave the systems where they were created. Replace names, account numbers and contact details with consistent placeholders so that the logic of each case survives. The set should be readable by the people who will approve changes, which usually means it lives in a shared document or repository with an owner, not inside one engineer's notebook.
How to choose cases that represent real work
The most common failure in a first golden set is that it looks like the examples the builder remembers rather than the work the role actually receives. Pull from the record. Ticket queues, order histories, approval logs and the notes staff leave when they correct something are all better sources than what a manager imagines the job involves.
A balanced set usually mixes four kinds of case. Each kind answers a different question about the agent.
- Routine cases that represent most of the volume, to confirm the basic path works.
- Costly cases where a wrong answer creates money, legal or reputational exposure, to confirm the agent stops or escalates in time.
- Ambiguous cases where the correct answer depends on context, to confirm the agent asks for what it lacks instead of guessing.
- Adversarial or out-of-scope cases, such as a request to bypass a policy or a message that tries to override instructions, to confirm the boundaries hold.
Avoid overweighting the cases that were easy to write down. If the set contains only tidy requests, a passing score tells you very little. It is also worth keeping at least one case per past incident, anonymized, so that a failure you have already paid for cannot quietly return. The incident itself may become a case through a short review, as described in AI Agent Incident Postmortem: A Blameless Template for Failed Runs.
Writing expected outcomes a reviewer can check
An expected outcome is the part most teams underwrite. A case that says the agent should answer helpfully is not testable. A case that states the required fields, the permitted action, the forbidden action and the reason is. Each entry in the set should carry the same small structure: the input and any context the agent would have, the expected outcome, the pass rule, and a note explaining the judgment behind it.
For tasks with free text, the pass rule is often a set of checks rather than a single match. The agent must mention the correct policy, must not promise a date it cannot confirm, must not reveal another customer's details, and must hand off when a threshold is crossed. Writing these checks in plain language keeps them reviewable. When a check needs a model to judge the text, treat that judge as another component to calibrate. The trade-offs are covered in LLM Judges for Agent Evals: Calibrate Before You Trust Them.
Illustrative scenario: a customer operations team wants an agent to draft replies to delivery delay messages and to propose a credit when the policy allows it. Its golden case for a delayed order that is inside the policy window specifies that the reply must state the new delivery estimate from the carrier record, must propose the standard credit, must not promise a faster delivery, and must route to a person if the customer mentions legal action. A second case for an order outside the window expects no credit, a polite explanation and no commitment beyond the policy text. Each case names the reviewer who signed off on it, so a later dispute can be traced to a decision, not a memory.
Running the set and reading the result
A golden set produces a verdict only when the rules for reading it are agreed before the candidate runs. Otherwise the team will argue about the result after seeing it. Decide the rules once, write them into the set's header, and apply them to every candidate.
Use a simple decision guide:
- If any must-pass case fails, the candidate does not ship, regardless of its overall performance.
- If every must-pass case passes and some non-critical cases regress, review each regression and decide whether the trade-off is acceptable for the role.
- If the candidate improves some cases and leaves the rest unchanged, check that the improvements are not caused by a shortcut, such as a longer answer that happens to contain the right keywords.
- If the results are mixed in a way nobody can explain, add cases that isolate the difference before deciding anything.
- If the candidate passes everything, still run it in a limited setting before it receives real authority, for example through a controlled pilot like Run an AI agent in shadow mode before giving it authority.
Record every run with the model or prompt version, the date, the verdict and the names of the people who reviewed disagreements. The record is what lets you explain six months later why the current agent behaves the way it does. Official guidance on evaluations, such as the [OpenAI evals guide](https://platform.openai.com/docs/guides/evals), describes this same loop of fixed test sets, graded outputs and repeated comparison, and the [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) places measurement and documentation inside a wider governance practice.
Keeping the set honest over time
A golden set decays. Policies change, products change, tools are renamed and customers ask new questions. A baseline that describes last year's business will reward agents for behaviors that no longer serve anyone. Put the set under the same change control as the agent. Any case added, edited or retired gets a reason, an author and a reviewer. Retire cases that describe a process that no longer exists, and keep a short history so the baseline can be compared fairly across time. For guidance on pruning, see Eval Sets for AI Agents: Retire Stale Cases Before They Mislead.
Be Human designs and operates personalized agents whose identity, memory, channels, tools, permissions and operating context stay under human control. In a first engagement, the golden set is written alongside the one useful role it measures, its memory and tools, its approval boundaries and its observable success criterion, so the baseline and the thing it judges are defined together. In managed operations, the set is rerun when an integration changes, reviewed during recurring evaluations and updated through the same process as the agent itself, so that the team always knows what a passing score means.
If you are not yet sure which of your tasks should get a golden set first, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.
Frequently asked questions
How many cases should a first golden set contain? Start with a number small enough that one person can review it in a sitting, and make sure it covers the common path, the costly mistakes and the known edge cases. Size matters less than clarity, because a few dozen well-written cases will guide decisions better than a large set whose expected outcomes nobody can explain. Grow the set when a new failure appears or when a new part of the role goes live.
Can an AI model grade the golden set automatically? It can grade some checks, such as whether a reply mentions a required policy or avoids a forbidden promise, but only after you have tested the grader against human judgments on a sample. Deterministic checks, such as confirming that a field has the correct value or that a refund was not issued twice, should stay as plain code wherever possible. Keep a human review step for cases where the expected outcome involves judgment.
Who should write the expected outcomes? The people who do the work should draft them, because they know what a good result looks like in practice, and a second person should review them, because they are the ones who will be asked to accept a change based on the verdict. Legal, finance or compliance staff should sign off on cases that touch regulated commitments. Record the names so that every expected outcome has an owner who can explain it later.
How often should the golden set change? Keep it frozen while you compare a candidate, so that the comparison is fair, and change it deliberately between comparisons when the business, the policy or the tools change. A monthly or quarterly review is a reasonable starting rhythm for most teams, with an immediate review after any incident. Every change should be logged with a reason, so the baseline remains a reliable reference rather than a moving target.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


