How to Define Success Criteria for an AI Agent Pilot

AI agent pilot success criteria are the measurable conditions, written before the pilot begins, that decide whether you expand the agent, adjust it or end it. The most useful form is three thresholds on a small set of measures: a go line, a fix band and a stop line. When these are agreed in advance, the review meeting compares evidence to a page everyone signed, instead of arguing about how the agent felt.

Key takeaways

• Write the criteria before the pilot starts, and have the sponsor and the people doing the work agree to the same page.
• Use three thresholds, not one: a go line, a fix band and a stop line, each tied to a measure and to an action already agreed.
• Measure outcomes on real work: quality of finished tasks, respect for boundaries, effort saved, reliability and cost per completed task.
• Record a baseline of how the work is done today, otherwise no improvement can be shown or denied.
• Time box the pilot and cap the number of fix cycles, so the fix band does not become an endless extension.

Why do impressions make a poor verdict?

Most pilots end in a meeting where people describe their experience. The enthusiast remembers the afternoon the agent drafted a hard reply in seconds. The skeptic remembers the one wrong answer that reached a customer. Both memories are real, and neither tells you whether the agent is good enough to run part of the business.

Without criteria, three things tend to happen. The goal drifts, because the pilot quietly absorbs new requests and the original job is forgotten. The bar moves, because a result that looked acceptable in week two looks disappointing once people get used to it. And the decision becomes political, because the person who proposed the pilot wants it to succeed and the person who inherits the risk wants it to be safe. This is one of the patterns behind pilots that stall before production: nothing was wrong enough to stop and nothing was clear enough to expand.

Written criteria remove most of that. They turn a feeling into a comparison, and they let a disagreement be about the evidence or the threshold, which are things you can inspect, rather than about people.

What do go, fix and stop actually mean?

A single pass or fail line is too blunt for an agent. Real results usually land in between, and you need to know in advance what happens there. Define three outcomes, and attach an action to each.

• Go: the measures meet or beat the agreed line, and no safety condition was breached. The action is to move to the next phase, with the scope, permissions and monitoring that were described at the start.
• Fix: the measures fall short of go but stay above the stop line, and the shortfall has a plausible, nameable cause such as missing documents, a weak tool or an unclear instruction. The action is a time boxed round of changes, followed by a re-test on the same material.
• Stop: a measure falls below the stop line, or a safety condition is breached, or the fix budget is used up without reaching go. The action is to end the pilot, switch off the agent's access and write down what was learned.

Two details matter. First, the fix band needs a limit, such as one or two improvement rounds, otherwise it becomes a polite way of never deciding. Second, some conditions should be stop triggers regardless of the averages. An agent that sends an external message it was only allowed to draft, or reads data outside its scope, has failed a boundary, and no good average offsets that.

Which measures belong in an AI agent pilot?

Choose few measures, and choose ones that describe finished work rather than activity. Message counts and hours of use say the agent was busy, not that it was useful. A workable set for most business roles looks like this:

• Outcome quality: of the tasks the agent completed, how many a qualified person accepted as correct with no edits, with light edits, or not at all. Define each category with examples before the pilot.
• Boundary behavior: how often the agent acted inside its permissions, stopped to ask when it should have, and left the actions reserved for humans to humans. The idea behind where an agent should stop and ask gives you the events to count.
• Effort: the time a person spends on the same task with the agent, including review and correction, compared with the baseline. Review time is part of the cost.
• Reliability: how often a task failed to finish, stalled, or needed someone to restart it, and how quickly problems were noticed.
• Adoption: whether the people the agent is meant to help actually route work to it, and what they do when they do not.
• Cost per completed task: the full running cost divided by tasks finished and accepted, not by messages sent. The article on cost per completed task explains why the denominator matters.

You do not need all six for every pilot. Pick the three or four that match the reason the agent exists, and keep the rest as observations. If you cannot say why a measure would change your decision, drop it.

How do you set the thresholds, step by step?

Do this as a working session with the sponsor, the person who owns the process and someone who will review the agent's output daily. It usually fits in one sitting if the role is narrow.

• Step one: write the job in a few sentences. State the task, the inputs, the systems the agent may touch, and what it must never do. A job description for the first agent role is a good starting document.
• Step two: capture the baseline. For a representative sample of recent tasks, record how long they take, how often they are redone, and how long customers or colleagues wait. Use real history, not memory.
• Step three: pick the measures. Choose three or four from the list above and define each one in a sentence that two reviewers would score the same way.
• Step four: set the go line. Ask what result would make you comfortable giving the agent more responsibility, then write it as a number or count against the baseline.
• Step five: set the stop line. Ask what result would make you uncomfortable continuing, even with improvements. Work backward from the cost of an error, not from optimism.
• Step six: describe the fix band. Name the causes you would accept as fixable, the changes you would make, and the maximum number of rounds.
• Step seven: list the hard stop triggers. These are the boundary breaches that end the pilot regardless of other results.
• Step eight: fix the pilot window, the review date and the decision makers. Include who can veto and what happens if they disagree.
• Step nine: sign. Everyone present agrees that the page is the standard, and that changing it mid-pilot requires a written note with a reason.

The most common mistake is setting the go line from the best demonstration anyone has seen. Set it from what the business needs. An agent that is slightly worse than a person but takes the dull, repeatable part of the work may be a good decision. An agent that is better on average but occasionally acts outside its limits may not be.

What does this look like in practice?

This is an illustrative scenario, not a report of a real engagement.

A distribution company wants an agent to draft replies to order status emails. A person reads each draft, then sends it. Before the pilot, the operations lead pulls forty past requests, records the time each took, and notes which ones needed a lookup in more than one system. That is the baseline and also the first test set.

The team agrees on three measures: draft quality, boundary behavior and handling time including review. They write the thresholds like this:

• Go: nearly all drafts on the test set and on live requests are sent with no edits or light edits, the agent never sends anything itself, and handling time including review is clearly below the baseline.
• Fix: more drafts need heavy edits than the go line allows, but the errors cluster around a nameable cause, such as one carrier's tracking format. One improvement round is allowed, then a re-test on the same forty requests plus a fresh set.
• Stop: any message is sent without approval, any customer record outside the order system is read, or the draft quality remains below the stop line after the fix round.

At the review, drafts for split shipments are the weak spot. The result sits in the fix band, the cause is clear, and the team has already agreed what to do. They improve how the agent reads shipment records, re-run the same material and meet again on the date written on the page. Nobody has to decide whether the agent seemed good.

How do you keep the evidence honest?

Thresholds are only as reliable as the measurements behind them. A few habits protect them.

• Use a fixed test set alongside live work. A golden test set lets you compare the agent before and after each change on identical material. Evaluation guidance from model providers, such as Anthropic's material on evals for agents, makes the same point: test on realistic tasks and grade outcomes, not just wording.
• Score with a rubric, and have a second person score a sample. If two reviewers disagree often, the measure is not defined well enough.
• Keep the agent's logs. You need to see which task, which tools, which approvals and which result, so that a surprising score can be traced to its cause. Include the cost of the run in the same record.
• Freeze changes during measurement. If you edit instructions or tools every few days, you are measuring a moving target. Batch changes into the planned fix round.
• Report the awkward cases. Include tasks the agent refused, escalated or abandoned. They are often where the real design questions are.

It also helps to separate who builds from who scores. The people who configured the agent tend to read ambiguous results generously. A reviewer from the process owner's team does not.

How does Be Human handle pilot criteria?

Be Human treats the success criterion as part of the first engagement, not as something added at the end. The work starts by defining one useful role, the memory and tools it needs, its approval boundaries and an observable measure of success. Those agreements become the go, fix and stop page for the pilot. During the pilot, the operating record shows tasks, approvals and outcomes, and managed operations can handle monitoring, recurring evaluations and controlled improvements during the fix rounds. The decision stays with your team, made against criteria you helped write.

What should you do before the pilot starts?

Use this short checklist the week before launch.

• The role is described in a few sentences, with a clear list of what the agent must not do.
• A baseline exists for the same tasks, drawn from real history.
• Three or four measures are defined well enough for two people to score the same way.
• Go, fix and stop thresholds are written, with an action attached to each.
• Hard stop triggers are listed, and the agent's permissions match them.
• The pilot window, review date and decision makers are named.
• Everyone involved has read the page and agreed that changes need a written reason.

If the pilot passes, the same page becomes the starting point for the next phase, and your team can use it when it comes to getting people to use the agent in daily work. If it does not, you will know why, which is the other reason to run a pilot.

Frequently asked questions

How long should an AI agent pilot run? Long enough to include the normal variety of the work, including at least one busy period if you can, and short enough that the team stays focused. Set the window and the review date in the criteria, and treat an extension as a decision that needs a written reason.

What if we have no baseline for the current process? Build a small one now. Take a sample of recent tasks and record time, rework and waiting time, even if the sampling is rough. A rough baseline written before the pilot is far more useful than a confident memory after it.

Who should make the go, fix or stop decision? The business owner of the process should decide, with input from the person who reviews the agent's output and whoever is responsible for risk. The people who built or configured the agent should present the evidence, but should not be the only voice in the decision.

Can we change the thresholds during the pilot? Only with a written note that states what changed and why, agreed by the same people who signed the original. Changing a threshold because the results look bad is a decision to stop being honest about the pilot. Changing it because you learned the original measure was ill-defined is legitimate, as long as it is recorded and the affected results are re-scored.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.