Weekly Review Sample Size for AI Agents: Start Small, Then Adjust
Set a weekly review sample for AI agent outputs using risk tiers, a starting count, and rules for raising or lowering it as evidence grows.

For most teams running a few AI agents, a workable starting point is a fixed weekly sample split by risk: every high-impact action checked, and a bounded random sample of routine work checked on top. Begin with about 10 reviewed outputs per risk tier, then raise or lower each tier based on the severity of misses, the amount of change in the agent's scope, and the number of clean weeks you have recorded. The sections below show how to define those tiers, how to pick the sample, and how to adjust it without guessing.
Key takeaways
- Review by risk tier, not with one flat number for every agent and every task.
- Start with a small fixed sample per tier and one named reviewer, then hold the setting for four weeks before changing it.
- Check every action that writes to a customer, moves money, or changes a permission, regardless of sample size.
- Raise the sample after a serious miss, a new tool, or a model change. Lower it only after a clean run with a written reason.
- Sampling only works when the reviewer can see the input, the tool calls, and the decision the agent made.
Why one number for all agent outputs fails
The question "how many outputs should we review?" sounds like it has a single answer, but a flat number fails in two directions. If the number is high enough to matter for a risky refund workflow, it wastes the time of your team on low-stakes drafts. If it is low enough to be sustainable for drafts, it can miss the one wrong refund that costs real money. The count only makes sense once you know which population you are sampling from and how much a miss in that population costs.
A better question is: for each kind of agent output, how much harm does an undetected error cause, and how many outputs do we need to see to notice a pattern before it spreads? That question produces different answers for different tasks, and it gives you a reason to defend each number when someone asks why the team reviewed 10 summaries but 100 percent of payment changes. Keep the numbers in plain language in your operating notes, so the reasoning survives staff changes.
The NIST AI Risk Management Framework describes managing AI risk as an ongoing activity of mapping, measuring, and monitoring context-specific risks, not a one-time approval. Sampling is one of the most practical ways to keep that monitoring active without reviewing everything.
A step-by-step plan for the weekly sample
Use this sequence the first time you set up review for an agent, and repeat the population and tier steps whenever its tools or permissions change.
- Step 1: List every output type the agent produces in a week, such as draft replies, internal summaries, record updates, and proposed refunds.
- Step 2: Assign each output type to a risk tier. A simple scale with three tiers works for most teams: high (changes external state or money), medium (customer-facing text or decisions that can be reversed), and low (internal drafts and summaries).
- Step 3: Set the weekly sample for each tier. For high tier, review every output and do not sample. For medium and low tiers, start with about 10 outputs each.
- Step 4: Select the sample at random from the week's outputs within each tier. Do not pick the ones that look interesting or the ones a staff member happened to open.
- Step 5: Record the verdict for each reviewed output using the same short fields every time: pass, minor issue, or failure, plus the failure category.
- Step 6: Compare the week's results against the previous four weeks and apply the adjustment rules in the section below.
- Step 7: Write one line in the operating log saying what changed and why, or that nothing changed.
The randomness in step 4 matters more than the exact count. If reviewers choose the outputs themselves, the sample will drift toward the easy-to-read cases, and your sense of quality will be too optimistic.
How many outputs to review at the start
A starting count of about 10 per medium or low tier is a practical default, not a statistical guarantee. With 10 reviewed outputs, you can see whether a problem exists and whether it looks systematic, such as the same wrong field being filled in several times. You cannot use 10 clean outputs to prove that the failure rate is below a particular level. If you need that kind of claim, for example before expanding an agent to a new customer segment, you need a larger sample and a clearly stated tolerance.
If your weekly volume is small, say a handful of outputs per tier, review all of them for the first month. Sampling only saves time when the population is large enough that reviewing everything is unnecessary.
A sample tells you where to look. It does not tell you that everything else is fine.
Illustrative scenario: a small operations team runs one agent that drafts customer replies, summarizes support tickets, and proposes refund requests for a human to approve. The team classifies refund proposals as high tier and reviews all of them, classifies replies as medium tier with a sample of 10 per week, and classifies internal summaries as low tier with a sample of 10 per week. In the third week, the reply sample shows two drafts that quote an outdated return window. The team does not yet know whether this is a one-off or a retrieval problem, so it raises the medium-tier sample to 20 for the following two weeks and adds a note to check the policy document source.
How to adjust the sample as evidence grows
Adjustment should follow written rules rather than mood. A reasonable starting policy is to hold each tier's sample for four weeks, then decide using the record. The decision guide below covers most cases.
- Raise the sample for a tier when any failure in the high impact category appears, even once.
- Raise the sample for a tier for two weeks after any model, prompt, or tool change, then reassess.
- Raise the sample when the same failure category appears in two consecutive weeks, even if each occurrence looks minor.
- Lower the sample only after four consecutive weeks with no failures and no minor issues in that tier, and only by a modest step, such as from 20 to 10.
- Never lower a high tier sample below full review while the agent has write access to external systems.
- Hold the setting when the results are mixed, and write down what you are watching for in the next period.
The lowering rule is the one teams most often skip, because it feels like admitting success is slow to arrive. Lowering the sample is a decision to accept less direct visibility, so it should be recorded and reversible. If a failure appears after you lower a tier, go back to the higher sample immediately and record the gap.
Drift can also make a clean sample misleading. An agent can remain accurate on the tasks you check while slowly changing on tasks that are rarer in your sample. Read the guidance in AI agent drift: how to spot quality decline over time when your sample results look stable but users report problems.
What a reviewer should record for each output
The review is only useful if the record can be compared week to week. Keep the fields short and fixed. A reviewer should be able to complete the record in a few minutes for a single output.
- Output identifier and date, so the reviewer and auditor can find the exact item.
- Risk tier and the reason the output was in that tier.
- Verdict: pass, minor issue, or failure.
- Failure category from a short list you agree on in advance, such as wrong facts, wrong tool, wrong recipient, missing approval, or unclear wording.
- Whether the stated reason in the agent's output matched the evidence it had, which catches confident answers built on missing sources.
- A one-line note on what a correct output would have looked like.
The last field is often the most valuable for improving the agent. Staff corrections written this way can become rules the agent follows next time. The method for turning corrections into rules is covered in turn staff corrections into agent rules.
How Be Human handles sampling in managed operations
In Be Human's managed operations, the review plan is defined in the first engagement, alongside the role, the memory and tools the agent needs, the approval boundaries, and the success criterion. Each risk tier is tied to specific tools and permissions, so the high tier is defined by what the agent can change rather than by a general feeling about the task. Monitoring records the verdicts and failure categories in a form the owner can read, and permission review runs on a schedule, so a tier does not quietly change when a new tool is added. When evidence supports a change, the adjustment is recorded as a controlled change with a reason, and the sample can be raised again without a new project. Be Human designs and operates this as a remote service in English and Portuguese, and the sampling method itself stays under your control.
If you are not yet sure which agent task should carry the first review tier, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.
Frequently asked questions
How many AI agent outputs should a new agent have reviewed each week? Start with every output for any action that changes external state or money, and about 10 random outputs per week for each medium or low risk tier. Hold that setting for four weeks so you have enough record to see patterns. After that, adjust using written rules rather than a fixed guess.
Does a clean weekly sample mean the agent is reliable? It means the outputs you checked were acceptable, which is useful but narrower than reliability. A small sample can miss rare failures and cannot prove a failure rate. Pair sampling with a stable set of test cases and watch for changes in the types of errors you record.
Should the business owner or a staff member do the review? The reviewer should know the work well enough to judge whether an output is correct, which is often a staff member who does the task by hand. The owner should set the tiers and read the weekly summary, but does not need to review every item. Rotate reviewers occasionally so one person's habits do not define the standard.
How often should the sample size change? Review the settings at the end of each four week period, and also after any incident, model update, or new tool. Changes should be small and recorded, with a clear reason. Frequent large changes usually mean the tiers were drawn poorly, so revisit the risk classification before adjusting the count.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


