How to Choose the First Users for an AI Agent Pilot

Choose pilot users by the work they do, not by their enthusiasm for AI. A good first group covers the real mix of tasks, includes at least one sceptic, has managers who protect their time, and gives specific written feedback on a regular rhythm. Enthusiasts are useful, but they should be a minority of the group.

Key takeaways

• Select users from the workflow the agent will own, so their feedback describes real work and not experiments.
• Mix enthusiasts, pragmatists and sceptics. A pilot that only pleases volunteers proves very little.
• Check three things per candidate: representative tasks, available time and willingness to write down what went wrong.
• Agree in advance what users will do, what the agent may access and how feedback is recorded.
• Treat the group as a measurement instrument. Its job is to produce evidence for a decision, not to generate applause.

Why do enthusiasts alone give a misleading pilot?

The people who volunteer first for an AI pilot are usually the ones who already use AI tools, enjoy tinkering and forgive rough edges. They are valuable, and you should include some of them. But they behave differently from the colleague who handles the same queue every day under deadline pressure and has no appetite for learning a new tool.

Enthusiasts tend to phrase requests well, so the agent looks better than it will for everyone else. They also work around problems quietly instead of reporting them, and they rarely test the dull, repetitive cases that make up most of the real workload. The result is a pilot that feels successful and then stalls when a wider group arrives with ordinary habits and ordinary patience.

The opposite mistake also happens. A manager picks the people with the lightest workload because they have time. Those users often do not carry the hard cases, the exceptions or the approvals, so the agent is never tested where it matters.

The goal is a group whose combined work looks like the role you are testing. If you have not yet written that role down, start with a job description for your first AI agent role, because you cannot pick representative users for a role that is still vague.

What makes someone a good pilot user?

Score each candidate against a short list of criteria. You do not need a formal matrix, but you should be able to explain why each person is in the group.

• Does real work in the target workflow: they handle the tasks the agent will touch, at least several times a week.
• Sees the messy cases: they deal with exceptions, incomplete inputs and awkward customers or colleagues, not only the clean path.
• Has time protected by a manager: the pilot adds a small amount of work, such as reviewing outputs and writing notes, and someone has agreed to that.
• Writes specific feedback: they can say what the agent did, what they expected and what it cost them to fix.
• Will say no: they are comfortable telling you the agent is not useful for a task, without worrying about seeming negative.
• Has enough standing to be believed: when they report a problem or a success, their peers take it seriously.
• Fits the permission boundary: their access and responsibilities match what the agent will be allowed to do on their behalf, so you are not testing with someone who holds unusual rights.

Two disqualifiers are worth stating plainly. Do not choose someone who has been told to participate and has no say in it, because resentment turns into thin feedback. And do not choose someone whose manager has not agreed to the time, because the pilot will be the first thing dropped when the week gets busy.

How many people and which roles should the group include?

Keep the group small enough to support personally and large enough that one person's quirks do not dominate. For a single agent role, a handful of people up to around ten is a workable range for most mid-sized teams. Smaller companies can run a meaningful pilot with three or four, as long as they cover different parts of the work.

Use this decision guide to balance the group:

• If most candidates are already heavy AI users, add at least one person who rarely uses AI tools.
• If the group has no one who handles exceptions or escalations, add the person the team goes to when something unusual happens.
• If every candidate sits in one team but the workflow crosses two, add one user from the downstream team who receives the agent's output.
• If no candidate is a sceptic, invite one deliberately and tell them their job is to find weak points.
• If the group is all senior, add someone newer who follows the written process more literally and notices where documentation is missing.
• If a candidate holds approval authority, decide whether they act as a user, a reviewer or both, and keep those roles distinct.

This balance matters more than headcount. A group of six well-chosen people will teach you more than fifteen volunteers who all work the same way.

A step by step plan for selecting the group

The selection itself can be done in a few working days. Follow these steps in order.

• Step 1: Write the agent role in one paragraph, including the tasks it will handle, the systems it touches and what it must never do on its own.
• Step 2: List the actual tasks in that workflow, with a rough sense of which are frequent, which are rare and which are risky.
• Step 3: Name the people who perform those tasks today, by task, not by job title. Titles hide who really does the work.
• Step 4: Ask each candidate's manager whether the time is available and what would have to give. Record the answer.
• Step 5: Talk to each candidate for a short conversation. Explain what the pilot involves, what you will ask of them and that critical feedback is wanted. Let them decline.
• Step 6: Check the mix against the decision guide above and adjust. Replace or add people until each major task type has at least one user who performs it regularly.
• Step 7: Define a person who owns the group day to day, someone who answers questions, collects notes and escalates problems.
• Step 8: Write down the selection rationale and the success criteria the group will help test, so the choice can be reviewed later.

Step 8 connects to the evidence you will eventually need. If the pilot is judged against targets that were agreed after the fact, nobody will trust the result. How to Define Success Criteria for an AI Agent Pilot covers how to set those targets before users start.

Illustrative scenario: choosing users for an operations agent

This is an illustrative scenario, not a description of a real engagement. A distribution company wants an agent to draft replies to order status questions and prepare a daily summary of delayed shipments for the operations team. Four candidates come forward or are suggested.

• An analyst who loves new tools and has already tried several AI assistants on her own.
• A senior coordinator who handles most of the difficult customer escalations and has little patience for tools that add steps.
• A junior coordinator who follows the written process closely and often asks questions about unclear rules.
• A warehouse liaison who receives the delayed shipment summary and acts on it, but does not write customer replies.

A manager might be tempted to pick the analyst and the junior coordinator because they are the most willing. The better group includes all four. The analyst provides fast, detailed prompts and finds edge cases. The senior coordinator tests whether drafts are actually usable on hard cases and whether review time is lower than writing from scratch. The junior coordinator exposes gaps in the documentation the agent relies on. The warehouse liaison judges the part of the output that leaves the team, which the other three never see.

The senior coordinator may be the least enthusiastic, and that is a reason to include her. If she finds the agent useful for her kind of work, the result carries weight with the rest of the team. If she does not, you learn why before a wider rollout.

How do you keep feedback useful once the pilot starts?

Good selection can still be wasted if feedback is casual. Set up a simple routine on the first day.

Ask users to record a small number of facts for each noteworthy case: the task, what the agent produced, whether they used it as is, edited it or discarded it, and roughly how long the correction took. A shared form or a plain document is enough. The point is that observations are comparable across users, so you can see patterns instead of anecdotes.

Hold a short, regular check-in with the whole group, and speak individually to anyone who goes quiet. Silence is rarely a good sign. It usually means the person stopped using the agent, or stopped believing feedback would lead to anything. Close the loop visibly: when a reported problem is fixed, tell the person who reported it. How to Collect Employee Feedback on an AI Agent and Act on It goes deeper on routines that keep people engaged.

Also prepare users before the agent starts. They should know what it can and cannot do, what data it sees and what to do when its output looks wrong. AI Training for Employees: What to Teach Before the Agent Starts outlines that groundwork. Wider adoption questions belong to a later stage, and How to Get Your Team to Actually Use an AI Agent picks up from there.

Finally, keep permissions deliberate. The agent should act within a defined boundary, and each pilot user's rights should be checked against it. Principles such as least privilege, described in the [OWASP authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html), apply to the agent's access as much as to human accounts.

How Be Human handles pilot user selection

Be Human treats the choice of pilot users as part of defining the first role, not as an afterthought. In a first engagement we work with the company to identify the workflow, map the tasks to the people who perform them, agree the approval boundaries and set an observable success criterion. We then help shape the user group, set up a feedback routine and review the notes with the company's owner. Under managed operations, that feedback feeds recurring evaluations and controlled improvements, and permissions for the pilot group are reviewed as part of regular operations. The company keeps the decisions about who participates and what the agent may do.

What should you decide before the pilot moves on?

Near the end of the pilot, use the group's evidence to answer three questions: did the agent help with the real task mix, where did it fail, and what would have to change before more people use it. Compare what the enthusiasts reported with what the sceptics reported. Where they agree, you have a strong signal. Where they differ, find out whether the difference comes from skill, role or a genuine weakness in the agent.

Then decide whether to extend, adjust or stop. Proof of Concept, Pilot or Production: What Each AI Stage Must Prove helps frame what this stage should have shown before you commit further. A pilot that ends with a clear no and a clear reason is still a useful result, because it was run by people who do the work.

Frequently asked questions

Should pilot users volunteer or be assigned? Use a mix, but make participation a real choice. Ask managers to nominate people based on the work they do, then have a conversation with each nominee and allow them to decline. Pure volunteers skew toward enthusiasts, while pure assignment often produces resentful and thin feedback.

How long should users stay in the pilot group? Keep the same people for the whole pilot so their feedback is comparable over time. Changing the group midway makes it hard to tell whether the agent improved or the users simply differed. If someone must leave, replace them with a person who does similar work and note the change.

What if the best candidates have no time? Reduce the ask before you lower the standard. A pilot user may only need to review outputs and write short notes, and a manager can often shift a small amount of other work. If a key task type has nobody available, it is better to narrow the scope of the pilot than to test that task with users who do not perform it.

Should executives be pilot users? Include them only if they perform the work the agent will handle. Leaders who do not touch the workflow can follow the results as sponsors and reviewers, but their feedback on daily tasks will be second hand. Their role is to remove obstacles and make the final decision based on the evidence.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.