Superintelligence Sampling Audits: How Often a Person Checks
Learn how to draw a sample of AI agent actions for human audit, set the starting rate by risk, and change that rate based on what reviewers find.

An AI agent audit sample is a small, deliberately chosen set of the agent's actions that a person reviews against its instructions, permissions and expected results. Draw it from a complete action log, sort it by risk so the most consequential actions get more review, and change the rate after each review round instead of on a fixed calendar. The method below works for a single support agent or a team of several agents.
Key takeaways
- Audit a sample of actions drawn from a complete action log, not a handful of conversations someone happened to read.
- Stratify the sample by risk so that actions which move money, change access or contact customers get more review than internal drafts.
- Start with heavy review for any new capability, and lower the rate only after reviewed actions show a stable pattern.
- Raise the rate immediately after a material error or a policy breach, and record the reason for every rejected action.
- Write down the draw method, the reviewer checklist and the rate rules so a second person can reproduce the sample.
Why sample at all instead of reading everything
An agent that answers customers, updates records or prepares invoices can produce far more actions in a week than one person can read. Reading none of them leaves you guessing, and reading a few random conversations tells you little about the actions that actually changed something in a system. A well drawn sample gives you evidence about the whole population within the time a reviewer really has.
Sampling has one real limit: a rare failure can sit outside the sample. For that reason the sample is never the only control. Actions that must be approved before they happen belong behind an approval gate, not in a post-hoc audit, and rare, high-impact actions should be reviewed in full. Sampling covers the large middle of routine work, where the question is whether quality drifts over time.
Build the population before you draw anything
A sample is only as honest as the list it comes from. Define the population as every action the agent took during the review window that wrote, sent, deleted, moved money, changed a permission or read personal data. Trace-based monitoring helps here. The OpenTelemetry primer describes traces, logs and metrics as the signals that let you reconstruct what a system did, and an agent action record should carry the same kind of detail: which agent acted, through which tool, against which record, and with what outcome.
- Export the action log for the review window, including agent identity, tool name, target record, timestamp, a summary of inputs and the outcome.
- Reconcile the count against the tool's own records for a few known actions, so you know the log is complete before you trust it.
- Collapse retries and duplicates so that one business action is counted once. Idempotent tool calls make this easier, which is why many API providers offer idempotency keys.
- Tag each action with a risk class using the rules in the next sections.
- Freeze the population for the window. If the log changes after you draw the sample, the draw can no longer be reproduced.
Include attempted actions that were blocked, because the blocks are evidence in their own right. A practical method for reviewing them is covered in how to review blocked agent actions.
How to draw the sample
Use a stratified sample. Split the population into strata by risk class and by tool, then draw a simple random sample within each stratum. Random draws stop the reviewer from picking the most interesting-looking cases, which tends to overrepresent unusual activity and underrepresent the ordinary work that quietly goes wrong.
Add targeted samples on top of the random draw, and label them as targeted in the record. Useful triggers include the first use of any new tool, every action taken in the day after a permission change, actions touching a record that a customer has complained about, and actions where the agent's own log shows a low-confidence note or a conflict between its instructions and the data it found.
Record the random seed or the exact method, so another reviewer could reproduce the draw. Without that, the sample becomes a story you tell about the agent rather than evidence about it.
Set the starting rate by risk tier
The rate should follow consequence, not habit. Name three tiers in your charter before the first review window opens, and attach each action type to one of them. Reviewer capacity sets the absolute count, while the tiers decide how much of that attention each action type receives.
- Tier one: actions that move money, change access, delete data or send messages to customers under the company's name. Review every action of this type until its approval gate has held for a full review cycle, then sample at a high rate that the reviewer can finish within the cycle.
- Tier two: actions that create or update business records, or read personal data for a stated purpose. Sample at a moderate rate, and include every first-time action type in the window.
- Tier three: internal drafts, summaries and searches with no external effect. Sample lightly, mainly to detect quality drift and to confirm the agent is not reaching data it should not touch.
Write the starting rate beside each tier, together with who reviews, how long a review takes and what counts as a pass. A rate without those details erodes quickly, because nobody can tell whether it is being met.
Let the results move the rate, up and down
Define pass criteria before anyone reads an action. A useful scale has four outcomes: pass; minor error that needs no change; material error that changes an outcome for a customer, an employee or the company; and policy breach, where the agent acted outside its permissions. Reviewers should record the outcome and the reason in the same entry, because reasons are what let you fix causes instead of symptoms.
The rule for raising the rate is immediate. A material error or a policy breach moves that action type to full review, opens an incident review, and holds the rate there until the cause is fixed and the following review cycles come back clean. The rule for lowering the rate is slow. Step down one level after a stable run of clean windows, never straight to zero, and never because the calendar says so. A single clean week after a material error is not evidence that the cause has gone.
Illustrative scenario: a company uses an agent to draft follow-up emails to existing customers. In the first review window, the reviewer finds one draft that quotes a product detail removed from the catalogue. The team moves email sends to full review, traces the error to a stale catalogue connection, repairs the connection, and keeps full review for two cycles before stepping the rate down one level at a time. For how a company can earn less supervision in stages, see staged gates for company agents.
What the reviewer checks and what you keep
A reviewer needs a short, fixed checklist so that two people looking at the same action reach the same answer. For each sampled action, the reviewer should confirm the following points.
- Was the action within the permissions granted to the agent for that tool and that record at the time it ran?
- Did it serve the goal in the agent's instructions, and nothing beyond that goal?
- Was the data it used permitted for this purpose?
- Is the outcome correct when checked against the source system?
- Can a second person reconstruct what happened from the log alone?
Keep the evidence together: the population snapshot, the draw method, the list of sampled action identifiers, the reviewer notes, and every decision that changed a rate. This bundle answers later questions from customers, auditors, insurers or your own leadership far better than memory does. The NIST AI Risk Management Framework treats measurement and monitoring as continuing work rather than a single event, which is the posture a sampling program needs. For how to explain agent decisions after the fact, see decision records for superintelligent decisions.
How Be Human runs sampling audits in managed operations
In Be Human's managed operations, audit sampling is one part of monitoring, recurring evaluations and permission review. For each agent, we define the action population, the risk tiers, the starting rates, the reviewer roles and the criteria that move a rate, and we keep each review decision alongside the agent's operating context so the evidence stays under the client's control. When a sample reveals a material error, the same process carries through incident response and the permission change that follows, and rate changes are made as controlled, recorded improvements rather than silent edits.
If you want to find the first task worth handing to an agent, and the sampling rules it would need, take the free Be Human AI diagnostic.
Frequently asked questions
How many actions should a sample contain? The count should be what a qualified reviewer can finish inside one review cycle without rushing, because a sample that goes unread is worse than a smaller one that is read carefully. Set the count per tier in your charter and revisit it when reviewer time or agent volume changes. Copying another company's count rarely fits your risk profile.
Can an audit sample replace an approval step? No. A sample tells you after the fact whether the agent's actions were sound, while an approval gate stops a consequential action before it happens. Use gates for actions that must not occur without a person, and use sampling for the routine work behind them.
What should trigger a full review instead of a sample? A material error, a policy breach, a newly connected tool, a permission change or a complaint tied to a specific action should each trigger full review of the affected action type for a set period. Full review should end on written criteria, not on a feeling that things look fine.
Who should perform the review? Choose someone who understands the business purpose of the action and has the authority to stop it, and keep the reviewer separate from the person who configured the agent wherever the risk justifies it. For tier one actions, consider a second reviewer as well, as described in two-person approval for AI agents.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


