How to Sample AI Agent Outputs for Quality Each Week

Sample AI agent outputs by sorting the agent's work into risk tiers, then reviewing a fixed share of each tier every week. Review every output that can cause harm or cannot be undone, draw a random sample from routine work, and add a targeted sample of unusual cases. When a reviewer finds an error, enlarge the sample for that category until the agent proves stable again.

Key takeaways

• Scale review effort to risk first and volume second. A small number of high-stakes outputs deserves more attention than a large pile of low-stakes ones.
• Combine a random sample, which shows the typical quality, with a targeted sample, which shows the edge cases where agents fail.
• Set a minimum sample per category, so a quiet week never means no review.
• Treat the sample size as adjustable: grow it after any change or error, shrink it only after a run of clean weeks.
• Record every review result in one place, so you can see trends instead of isolated opinions.

Why not read everything, or nothing?

Two habits tend to appear when a company first deploys an agent. The first is reading every output. It feels safe, but it does not last. Reviewers tire, attention drops, and the review quietly turns into a rubber stamp. The second is reading nothing once the first few days look fine. That feels efficient, but quality changes over time as inputs, documents, tools and models change.

Sampling sits between the two. You accept that you will not see every output, and in exchange you design the sample so that it catches the problems that matter. The question is no longer "how many outputs should we read?" but "which outputs, how often, and what happens when we find something?"

A useful sample does three jobs. It estimates typical quality, it surfaces rare but costly failures, and it produces a record you can compare from week to week. If your sample does only the first job, you will miss the failures that hurt. If it does only the second, you will not know whether the everyday work is slowly getting worse.

How do you scale the sample to risk and volume?

Start with risk, because volume alone is a poor guide. An agent that drafts internal meeting summaries can run at high volume with light review. An agent that proposes a refund, a contract clause or a customer commitment needs close review even when it produces only a few outputs a week.

Place each type of output into one of three tiers, based on what happens if it is wrong:

• High risk: the output reaches a customer, moves money, changes a record of truth, or triggers an action that is hard to reverse. Review all of them, or require approval before they leave the agent.
• Medium risk: the output is used by an employee who will probably notice a problem, or it can be corrected cheaply. Review a fixed number per category each week, with a minimum so that small categories are never skipped.
• Low risk: the output is internal, reversible and easy to ignore if wrong. Review a small random draw, enough to notice a pattern.

Volume then adjusts the middle and bottom tiers. If a medium risk category produces only a handful of outputs in a week, review nearly all of them, because a small sample from a small population tells you little. If it produces a very large number, a fixed count per week is usually enough, since the confidence you gain from each extra item shrinks quickly. What matters more is that the draw is genuinely random and covers the different types of request, not that the count is large.

When you are unsure where something belongs, put it in the higher tier. You can move it down later with evidence.

What belongs in the targeted sample?

A random sample tells you about the average output. Most serious failures are not average. They cluster around particular conditions, so add a targeted sample on top of the random one.

Good candidates for targeted review:

• Outputs where the agent escalated, hesitated, retried or used an unusual number of steps.
• Outputs that a user edited heavily, rejected or complained about.
• Requests that look new, such as a document type, customer segment or language the agent has not handled before.
• The first outputs after any change to the model, instructions, tools or connected documents.
• Outputs that cite no source, or cite a source that is old or from an unexpected folder.
• Cases where the agent was uncertain, keeping in mind that stated confidence is only a rough signal. The article on what confidence scores really mean explains why it should prompt a look, not replace one.

The targeted sample is usually small, but it is where reviewers learn the most per item. Log enough about each run to find these cases without searching by hand. The guide on what to log for AI agent observability lists the fields that make this practical, and the OpenTelemetry observability primer is a good general reference for the idea of traces and signals.

A weekly sampling plan, step by step

This plan fits in a small team without special tooling. A shared spreadsheet is enough to start.

• Step 1: List the output types. Write down every kind of thing the agent produces or does, such as a draft reply, a classification, a summary, a record update or a scheduled action.
• Step 2: Assign a tier to each type, using the consequence of an error. Ask one person with authority to confirm the tiers.
• Step 3: Set a weekly count per type. High risk gets full review or pre-approval. Medium and low get a fixed count with a floor, so a quiet week still receives review.
• Step 4: Draw the random sample on a fixed day. Choose items by a method that the agent and the reviewers cannot influence, such as every Nth item or a random pick from the week's log, not whatever happens to be on screen.
• Step 5: Add the targeted sample from the list above.
• Step 6: Review against a short checklist (next section), and record a result for each item: acceptable, minor issue, or serious issue.
• Step 7: Apply the expansion rule. Any serious issue in a category doubles that category's sample the following week and triggers a look at the cause. Several clean weeks in a row allow a modest reduction, never below the floor.
• Step 8: Hold a short weekly review. Look at the trend, decide what changes, and assign an owner for each fix.

The expansion rule in step 7 is the part most teams skip. Without it, a discovered error is an anecdote. With it, an error automatically buys more scrutiny where it is needed.

What should a reviewer check in each output?

A short, consistent checklist beats a long, subjective one. Reviewers who judge on feel will disagree with each other, and you will not be able to compare weeks. Keep it to the points that matter for the task:

• Was the answer correct against the source of truth, not merely plausible?
• Did it stay within the task and the permissions the agent was given?
• If it relied on documents, did it use the right ones and say so? See how to make an agent cite its sources.
• Was the tone and format right for the audience?
• Did it escalate when it should have, and not when it should not?
• If it took an action, was the action correct and the record accurate afterwards?

Give each item a pass or fail, plus a free text note when something fails. Free text notes are where patterns first appear. After a few weeks, you will see the same phrases returning, and those become candidates for new test cases.

Failures found in sampling should feed your test set, not just your fixes. If a reviewer catches a wrong answer, add that case to a golden test set so the same mistake is checked automatically after every future change. Sampling finds new problems in live work, while the test set stops old problems from returning. For a wider view of testing against real tasks, see how to evaluate AI agents on real business tasks. The NIST AI Risk Management Framework and the evals guides from OpenAI and Anthropic offer more background on structured measurement.

An illustrative scenario

This is an illustrative scenario, not a real case. A small services firm runs an agent that handles incoming customer emails. It produces three kinds of output: a drafted reply that a staff member sends, an automatic category label on each email, and a proposed change to a customer record that waits for approval.

The owner tiers them like this. The record changes are high risk, so every one is approved by a person before it is applied, and the approval itself counts as the review. The drafted replies are medium risk, because a staff member reads them before sending, but busy staff can approve too quickly. The category labels are low risk, since a wrong label only misroutes an email.

For the week, the owner sets: all record changes reviewed, twelve replies drawn at random across the different email types, and eight labels drawn at random. The targeted sample adds any reply that staff edited heavily and any email that arrived in a language the agent had not seen before.

In the second week, the reviewer finds one reply that quoted an outdated policy. That is a serious issue, so the reply sample doubles the next week, and the team checks which document the agent used. They find that an old policy file was still in the connected folder. They remove it, add the case to the test set, and the following week's larger sample comes back clean. After three clean weeks, the reply sample returns to twelve. At no point did anyone read every email, yet the one costly error was found and traced to its cause.

How does the sample change over time?

Treat the first weeks of any agent as a period of heavy sampling. If you can run the agent in parallel with human work before giving it authority, do so. The guide on running an agent in shadow mode describes how to compare its output with human decisions before anything is sent.

After launch, raise the sample whenever something changes: a new model version, edited instructions, a new tool, a new document source or a new type of customer request. Lower it slowly and only when the record supports it. Keep a permanent floor, because quality can decline gradually without any single event to blame. The article on spotting AI agent drift covers the slow changes that a steady weekly sample is well placed to detect.

Also watch the reviewers. If every item passes for many weeks, ask whether the checklist is too lenient or the sample too easy, and occasionally insert a known bad example to confirm that reviewers still catch it.

How Be Human handles sampling

In Be Human's managed operations, the sampling plan is defined during the first engagement, alongside the role, the tools and the approval boundaries. Output types are assigned to risk tiers, high risk actions go through human approval, and recurring evaluations draw random and targeted samples from the agent's logs. Results are recorded, reviewed with the client's owner, and used to adjust instructions, permissions or test cases through a controlled change process.

Frequently asked questions

How many outputs should I spot-check each week? There is no single right number, because it depends on risk and volume. Review all high risk outputs, set a fixed count with a minimum floor for each medium and low risk category, and add a small targeted sample. Increase the count after any change or error, and reduce it only after several clean weeks.

Should the sample be random or chosen by a person? Use both. A random draw gives an honest picture of typical quality and prevents reviewers from picking only easy or interesting items. A targeted draw catches the unusual cases, such as heavy edits, escalations and new request types, where failures are most likely.

Who should review the sampled outputs? Ideally someone who knows the task well enough to judge correctness, and who is not the person who built or tuned the agent. Rotating reviewers helps reveal disagreement in the checklist. For high risk work, the person accountable for the outcome should see the results directly.

What do I do when a spot check finds a problem? Record it, classify how serious it is, and enlarge the sample for that category. Then find the cause, whether it is an instruction, a document, a tool or a model change, and fix it through a documented change. Finally, add the case to your test set so the same mistake is caught automatically in future.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.