Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Eval Sets for AI Agents: Retire Stale Cases Before They Mislead

A routine for reviewing AI agent eval cases, retiring outdated ones and adding new cases from real work, so evaluations keep measuring what matters.

Be Human typographic cover with the title: Eval Sets for AI Agents: Retire Stale Cases Before They Mislead
Conceptual illustration created with AI · Be Human

To keep an AI agent evaluation set useful, review every case on a fixed cadence, retire the cases that no longer describe the current work, and add cases drawn from real tasks that went wrong or were corrected by staff. A test set that matched the work at one point can keep passing an agent that no longer serves the process, or fail a correct agent for following a rule the business has dropped. This article gives a routine for each cycle, criteria for retiring a case, and a record that owners can read.

Key takeaways

  • A passing score only means something if the cases still match the current work.
  • Give every eval case a named owner, a source, a list of dependencies and a date of last review.
  • Retire a case when the rule or process behind it has changed, and record the reason.
  • Add new cases from escalations, staff corrections and sampled outputs, not from imagination alone.
  • Archive retired cases so you can explain old scores and detect a past failure that returns.

Why stale eval cases mislead

An eval case is a claim about the work: given this input, a good result looks like that. When the work changes, the claim can quietly stop being true. A refund process gets a new approval step, a product is renamed, a form is replaced, or a customer segment moves to a different channel. The old case still runs, still scores and still appears in the dashboard, even though it no longer tests anything the business cares about.

Stale cases mislead in two directions. Some inflate confidence: the agent passes easy cases that no longer represent the hard parts of the job, so the average stays high while real failures happen in areas the set does not cover. Others punish correct behavior: the agent follows the current rule, the case expects the retired rule, and the score drops. The team then tunes the agent backward to satisfy an outdated expectation. That second failure is the more expensive one, because it creates a regression that looks like progress.

Published guidance on agent evals, such as Anthropic's notes on demystifying evals, treats the suite as something that needs ongoing upkeep as the agent and its environment change. Good maintenance follows the same logic. The eval set is a living record of the business process, not a fixed exam.

Start with a case register

Before reviewing anything, put the cases in a register that a business owner can read. A spreadsheet or a versioned file is enough. For each case, record:

  • An identifier and a one-sentence description of the task in plain words.
  • The input, with personal data removed or replaced by realistic placeholders.
  • The acceptance rule: what a good result must contain, avoid or do, including any required tool action or escalation.
  • The source: where the case came from, such as a real escalation, a staff correction, a sampled output or a policy document.
  • The owner who answers whether the expectation is still right.
  • The dependencies: the policy version, tool schema, template or data source the expectation relies on.
  • The date of last review and the decision taken at that review.

The dependency field matters most. It lets you find every affected case when a policy or tool changes, instead of rereading the whole set. If you have no register yet, a golden test set is a good starting point, because it forces each case to name its expected behavior.

A review routine for each cycle

Run the review on a fixed cadence that matches how quickly your process changes. A monthly rhythm suits most operational workflows. Add an extra pass whenever a policy, tool, template or catalog entry changes.

  • Collect the changes. List every policy edit, tool schema change, template update and catalog change since the last review, and match each one against the dependency field in the register.
  • Read failures and passes side by side. For each failure, decide whether the agent erred or the expectation is stale. Check the source of truth the case cites before deciding.
  • Classify each affected case as keep, revise, split or retire, and write one sentence explaining the decision.
  • Review a sample of cases that passed. A case that passes every time may be too easy to tell you anything.
  • Add new cases from the period's escalations, staff corrections and sampled outputs, following the criteria in the section below.
  • Run the updated set against the current agent version. Report scores by category, not only as one average, so a drop in one area does not hide behind gains elsewhere.
  • Record the new set version and the reasons for every change in the change log, so the next reader can compare scores across versions.

When should you retire a case?

Retirement is a decision, not a deletion. Use the rules below and record the reason each time.

  • Retire when the business rule behind the case no longer applies and no one can name a replacement rule.
  • Revise when the intent still holds but the wording, example, output format or source document is outdated.
  • Split when one case checks several behaviors and can fail for different reasons. A mixed case cannot tell you which behavior broke.
  • Retire a case when the tool or data it needs was removed and the workflow itself was retired too. If the workflow continues through another path, revise the case instead.
  • Keep a case, even one the agent passes easily, when it guards a costly failure such as a wrong commitment to a customer. These cases act as regression guards, and you want them to keep passing.
  • Archive each retired case with its reason and the last version it ran in. You will need it to explain old scores and to check whether a past fix has come back.

Illustrative scenario: a return policy changes

Illustrative scenario: A support team runs an agent that drafts replies to customers asking about returns. The eval set holds a dozen cases built on the earlier workflow, where customers were asked to wait for a manual approval before shipping. The company then moved most products to an instant return label. The agent began offering the new label, which was correct, but several cases failed because they expected the approval step. The score dropped, and an engineer started editing the prompt to bring back the old wording.

The monthly review caught the problem because the register linked those cases to the returns policy. The team revised the expectations for the affected cases, retired two cases that described an approval step no product used any longer, and archived them with the policy version. They then added a few cases from recent escalations where customers asked about the label on excluded products. The change log recorded each decision, so the next reviewer could see why the scores moved.

Add new cases without flooding the set

The best new cases come from work the business already did. Three sources are reliable: escalations that a person had to resolve, corrections staff made to agent drafts, and outputs pulled from a sample of recent production work. Each one shows a real input and a real judgment about the right outcome. A staff correction is especially useful, because the reviewer has already written down what good looked like.

Add a case only when you can state its acceptance rule in a sentence. If you cannot, the case is not ready, and the underlying process probably needs clarifying first. Keep the mix balanced across categories so the set does not fill with routine requests while rare, high-cost situations go untested. Remove personal data before a case enters the set, and limit who can read the register when inputs contain customer information.

Ownership, cadence and how Be Human handles it

Each eval set needs a named owner from the business side, not only the technical team. The owner answers whether an expectation is still right, approves retirements and signs off on changes to high-cost cases. Engineers run the tooling and record results. The owner decides what the agent should be measured on. The NIST AI Risk Management Framework treats measurement and ongoing monitoring as part of managing AI risk, and this division of responsibility is one practical way to meet that expectation. Write the cadence into the runbook so the review does not depend on memory. Automated LLM judges can speed up scoring, but they still need calibration against a human reviewer and cannot decide whether an expectation is still correct.

Be Human's managed operations include recurring evaluations, permission review and controlled improvements. In practice, the case register sits alongside the agent's memory, tools and change log, each case is tied to the policy or tool version it depends on, and changes to the test set are versioned so scores stay comparable over time. Retirements are proposed and approved by the business owner rather than applied silently.

If you are unsure which workflow deserves a maintained eval set first, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

How often should an agent eval set be reviewed? A monthly review suits most operational workflows, with an extra pass whenever a policy, tool, template or catalog entry changes. The right rhythm depends on how fast your process moves. If the rules change every week, tie reviews to those changes rather than only to the calendar.

Should we delete old eval cases or keep them? Keep them in an archive with the reason for retirement and the version they last ran in. Deleting them removes your ability to explain past scores or to check whether a fixed failure has come back. The active set should contain only cases that match current work.

Who should own the eval set? A business owner should own the expectations, because that person decides what good looks like for the work. The technical team runs the tooling and records results. Splitting responsibility this way prevents anyone from quietly changing a case to match what the agent happens to produce.

Can an LLM judge replace human review of eval cases? An automated judge can score many outputs quickly, but it still needs calibration against human decisions, and it cannot decide whether the expectation itself is still right. Use it to speed up checks, then sample its verdicts and compare them with a reviewer's judgment before trusting the scores.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →