User Acceptance Testing for AI Agents: Let Staff Sign Off

User acceptance testing for an AI agent means the people who will work alongside it run it on their own real tasks, record what happens, and decide whether it is fit for use. The outcome is a written go or no-go signed by named staff and an accountable owner, not an impression from a demo. Done well, it needs a defined set of tasks, a scoring rule agreed in advance, and a short record that anyone can read months later.

Key takeaways

• Acceptance belongs to the people who do the work, not to the builders. Builders prepare the test, staff judge the result.
• Build the test set from real, recent tasks, including awkward cases, out-of-scope requests and situations where the agent should stop and ask.
• Agree the pass rule and the severity levels before testing starts, so the decision is not negotiated after the fact.
• End with a written record: go, conditional go or no-go, with the evidence, open issues and named signatories.
• Treat acceptance as repeatable. Rerun a smaller version of it after every meaningful change to the agent.

What is user acceptance testing for an AI agent, and how is it different from other testing?

Most testing of an AI agent is technical. Engineers check that the agent connects to its tools, that permissions are set as designed, and that automated evaluations pass. This work matters, and automated evaluations are especially good at catching regressions. Anthropic's engineering article on evaluations for AI agents explains why a repeatable set of graded tasks is the backbone of improving an agent safely.

User acceptance testing asks a different question: would the person who owns this work put their name on the result? A technically correct answer can still be unusable. It may be too long, in the wrong tone, missing the detail a colleague always needs, or confident where a careful person would hesitate. Only the people who do the job can judge that, because they carry the consequences when the output goes to a customer, a supplier or a manager.

There is also a governance reason. The NIST AI Risk Management Framework treats measurement and human oversight as activities tied to the real context of use. A signed acceptance record is a practical way to show that a named human reviewed the agent in its actual context before it was relied on.

Who should test the agent, and who decides?

Keep three roles separate.

• Testers: the people who perform the work today. Include an experienced person who knows every exception, a newer person who will rely on the agent more, and one natural skeptic. A skeptic finds problems that enthusiasts forgive.
• Facilitator: someone who runs the sessions, keeps time, collects findings and answers procedural questions. This should not be the person who configured the agent, because builders unconsciously steer around known weaknesses.
• Approver: the manager accountable for the outcome of the work. This person reads the evidence and signs. In a larger initiative the executive sponsor is informed of the decision, and our article on the executive sponsor role describes what that involves.

Choosing testers is the same problem as choosing pilot users. If you have not done that yet, how to choose the first users for an AI agent pilot covers the criteria. One difference: testers for acceptance should be asked for a specific, bounded time commitment, with the time protected from normal workload. Testing squeezed between other tasks produces shallow feedback.

How do you build a test set from real work?

The test set is the most important artifact. Weak sets produce false confidence, so build yours from the work itself rather than from tasks invented for the occasion.

Start by asking each tester for recent, completed examples of the work the agent will handle. Where the examples contain personal or sensitive data, apply the same access rules the agent will operate under, or use redacted copies. Then sort the examples into groups:

• Routine tasks: the everyday requests that make up most of the volume. The agent must handle these well or it has no reason to exist.
• Awkward tasks: incomplete information, conflicting instructions, unusual formats, a customer who writes in a different language than expected.
• Boundary tasks: requests that sit just inside or just outside the agent's role. These show whether the role description is clear to the agent as well as to people.
• Stop-and-ask tasks: situations where the correct behavior is to escalate to a human or decline. Testing only what the agent should do misses half the picture.
• Adversarial tasks: a document or message that contains instructions aimed at the agent, or a request to use a tool or data it should not touch. The OWASP guidance on prompt injection prevention and on authorization (deny by default, least privilege) gives a useful vocabulary for designing these cases.

Keep some tasks hidden from the people who configured the agent. If the builders have seen every test in advance, you are measuring their preparation, not the agent.

What does a step-by-step acceptance plan look like?

The plan below fits a single agent role and can run over a few working sessions. Adjust the length to the risk of the work.

• Step one, fix the scope: write one paragraph on what the agent is accepted for and what it is explicitly not accepted for. Everything outside it is out of scope for the decision.
• Step two, agree the pass rule: for each task type, define what a good result looks like in plain words, and decide how many tasks must pass for a go. Write this down before anyone runs a test.
• Step three, define severity: settle the levels described in the next section so that findings are classified consistently.
• Step four, assemble the test set: gather the real tasks, sort them into the groups above, and have the approver confirm the set is representative.
• Step five, brief the testers: explain the scope, how to record a finding, and that reporting a failure is the purpose of the exercise. Do not coach them on how to phrase requests; they should write as they normally would.
• Step six, run the sessions: each tester works through their tasks, records the agent's output, and marks it against the agreed definition. The facilitator logs timing, questions and anything surprising.
• Step seven, review findings together: group duplicate findings, classify severity, and assign each to a fix, a documented limitation or a decision to accept the risk.
• Step eight, fix and retest: the builders correct what can be corrected, and the testers rerun the failed tasks plus a sample of those that passed, to confirm nothing else moved.
• Step nine, decide and sign: the approver reads the record and records go, conditional go or no-go, with the testers' individual views attached.

If a task needs access to a customer record or a payment system, confirm that the permissions in the test environment match what production will grant. A test that runs with broader access than production will flatter the result. A permission review before testing prevents that gap.

How should testers score and classify findings?

Testers need a short, shared language. Complicated rubrics are abandoned by the second session. Score each task on a few questions, answered yes or no with a comment:

• Was the result correct?
• Was it complete enough to use without rework?
• Was the format and tone acceptable for its audience?
• Did the agent stay within its role and permissions?
• Did it escalate or decline when it should have?

Then classify every finding by severity:

• Blocker: the agent did something it must never do, such as acting outside its permissions, exposing information to the wrong person, or presenting an invented fact as true in a way that could cause harm. A single blocker in scope means no-go for that scope until fixed and retested.
• Major: the result is wrong or unusable in a way that costs real rework, or the failure repeats across several tasks. Several majors in one task group usually mean that group is not accepted.
• Minor: usable with a small edit, such as phrasing or layout.
• Note: a suggestion or preference that does not affect acceptance but is worth keeping for later improvement.

It is tempting to turn this into one score, but a score hides the difference between many small annoyances and one serious failure. Keep the severity counts visible. For help defining the measurable outcome that the acceptance rule should connect to, see how to define success criteria for an AI agent pilot.

How do you write the go or no-go record?

The record is short and plain. It should let someone who was not in the room understand what was tested, what was found and why the decision was reasonable. Include:

• The agent role and the scope accepted, and the scope excluded.
• The test set description, including how tasks were chosen and which were hidden from builders.
• The pass rule agreed beforehand.
• Findings by severity, with the status of each: fixed and retested, accepted as a known limitation, or open.
• The decision: go, conditional go with stated conditions and a date for review, or no-go with the reasons.
• Names and roles of testers, facilitator and approver, with each tester's own view recorded, including dissent.

Use the decision options consistently. A go means the agent is accepted for the stated scope. A conditional go means it is accepted with limits, such as restricted task types, human review of every output for a period, or a narrower group of users. A no-go means acceptance is withheld, and the record lists exactly what must change for a retest.

Here is an illustrative scenario. A finance operations team tests an agent that drafts replies to supplier questions about payment status. Testers run routine status questions, a few with missing invoice references, and several requests asking for an exception to payment terms. The agent handles the status questions well. But in the exception requests it twice implies that an exception is possible, which is a decision only a manager can make. The testers classify this as a blocker. The approver records a conditional go limited to status questions, with exception requests routed to a person, and schedules a retest of the exception tasks after the fix. The record shows what was accepted, what was not and why, so nobody later assumes the agent was approved for more than it was.

A go is not permanent. The stages of maturity are covered in what each AI stage must prove, and changes after launch should follow a safe update process, with a reduced acceptance run attached to any change that touches behavior, tools or permissions.

How does Be Human handle acceptance testing?

In a Be Human engagement, the first step defines one role, the memory and tools it needs, the approval boundaries and an observable success criterion. Acceptance testing is built from those same definitions. We help the client's staff assemble the test set from real work, prepare the environment with production-equivalent permissions, facilitate the sessions, and keep the findings and decision record. Fixes are made under human control, retests are run before the go is confirmed, and a smaller acceptance run is repeated after controlled changes during managed operations.

Frequently asked questions

Who should sign off on an AI agent, the IT team or the business? The business owner of the work should sign off, because that person is accountable for the results. IT and security review the technical and access side, and their findings feed into the record, but they should not substitute for the judgment of the people who do the job.

How many tasks should the test set include? Enough to cover each task group several times, and few enough that testers can finish without fatigue. The right size depends on the risk of the work: a role that sends messages to customers deserves a larger and more adversarial set than one that drafts internal summaries. Quality and coverage of the tasks matter more than volume.

What if testers disagree about whether the agent is good enough? Record the disagreement rather than averaging it away. Look at which tasks produced it, since it often reveals an unclear standard in the work itself. The approver then decides, and the dissenting view stays attached to the record so it can be revisited at the next review.

Do we need to repeat acceptance testing after the agent goes live? Yes, in a smaller form. Whenever the agent's instructions, tools, memory sources, permissions or underlying model change, rerun the tasks most likely to be affected, plus the stop-and-ask and boundary tasks. Keep the earlier test set and add new real cases that surfaced in daily use.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.