Capture a Baseline Before Your AI Pilot Starts to Prove Value

To run a baseline measurement before an AI pilot, choose the one workflow the agent will touch and record how it performs today: how long items take, how many arrive, how often work is redone, and what the people doing it say. Take these from real work samples over a normal stretch of time, write down how each was measured, and store the record where it cannot be quietly edited. Without that record, any improvement you report later is an opinion rather than evidence.

Key takeaways

• Record the baseline before the agent touches the workflow, because the old way of working is hard to reconstruct once it has changed.
• Measure a small set: time, volume, rework, waiting and human opinion. A few honest measures beat a dashboard nobody trusts.
• Write down how each measure was taken, by whom and from which sample, so the later comparison uses the same method.
• If the data does not exist, sample by hand over a short, normal stretch of work instead of estimating from memory.
• A baseline can be a range or a plain description. It does not need false precision to be useful.

Why does a pilot need a baseline before it starts?

Most pilots end with a conversation that goes the same way. Someone says the agent is helping. Someone else asks, compared with what? If nobody wrote down what the work looked like before, the answer comes from memory, and memory flatters whichever story the speaker prefers. Enthusiasts remember the old process as slower than it was. Skeptics remember it as smoother than it was.

A baseline removes that argument. It is a dated, boring description of the current state, taken before anyone has a stake in the result. It also protects the project in the other direction: if the pilot does not help, the baseline shows that clearly and early, which makes it cheaper to adjust or stop.

There is a practical reason too. Once an agent is live, the workflow changes around it. People skip steps, rename categories, reroute exceptions. The original process quickly becomes impossible to observe. You get one chance to measure the before, and it closes the day the pilot begins.

A baseline is also the natural companion of your success criteria. Our guide on how to define success criteria for an AI agent pilot explains what a pilot should achieve. The baseline is what makes those criteria checkable, because a target such as faster or more consistent only means something relative to a starting point.

Which before-measures are worth recording?

You do not need a measurement program. You need a handful of measures that match the role the agent will play. For most business workflows, these six cover the ground:

• Time per item: how long one unit of work takes from start to finish when someone works on it. Record a typical case and a hard case separately.
• Waiting time: how long an item sits untouched between steps, such as waiting for an approval, a missing document or someone returning from leave. This is often larger than working time and often where an agent helps most.
• Volume and shape: how many items arrive, and whether they come steadily or in bursts. An agent that handles an average week may still struggle on a Monday morning.
• Rework: how often finished work comes back for correction, and what the corrections are about. This becomes your quality baseline.
• Handoffs and exceptions: how many people touch an item, and which kinds of items need a senior person to decide.
• Human view: a short written note from the people doing the work about what is tedious, what is risky and what they would stop doing first.

One more measure is worth adding if finance will ask about it: the people time spent per unit of work. The FinOps Foundation describes unit economics as relating cost to a business unit of output, and the same thinking applies here. You are not calculating a price. You are noting how much effort one completed item takes today, so a later comparison has something to stand against.

Notice what is missing from this list: anything about the AI itself. The baseline describes the work, not the tool. That is what keeps it fair.

How do you capture the baseline, step by step?

The plan below fits a single workflow and can be done by the person who owns that workflow, with a few hours of effort spread over the sampling period.

• Name the workflow in one sentence, including where it starts and where it ends. If you cannot draw the boundary, the pilot cannot be scoped either.
• Pick the measures from the list above that the agent could plausibly change. Three or four is enough. Leave out measures nobody would act on.
• Define each measure in plain words before collecting anything. For example, first reply means the first message that moves the request forward, not an automatic acknowledgment.
• Choose a sampling window that includes a normal stretch of work, ideally with one busy and one quiet period. Avoid holidays, launches and unusual weeks unless they are the point of the pilot.
• Pull real items rather than reconstructing them. Use timestamps, ticket histories, shared inbox threads or a simple tally sheet kept by the team for a short time.
• Record the method alongside the result: who measured, which items, how they were chosen and what was left out.
• Ask the people doing the work for their written view, and keep their words rather than summarizing them.
• Store the baseline as a dated document with the owner's name, and agree that it will not be edited after the pilot starts. Corrections go in as a new version with a note.
• Fix a small set of example items, a sample of representative requests, to reuse later. Evaluation guides, including the OpenAI evals guide and Anthropic's writing on agent evals, rest on the same idea: compare behavior on a stable set of examples, not on whatever happens to arrive.

Bring the finished baseline to your AI pilot kickoff meeting so everyone in the room starts from the same picture. Agreeing on it there is much easier than arguing about it after results come in.

What if the data does not exist yet?

This is the usual situation, and it is not a blocker. Many teams have no timestamps, no categories and no tracking beyond a shared inbox. There are three honest ways forward.

First, sample by hand. Ask the team to note a few facts about each item for a short, normal stretch of work: when it arrived, when it was finished, whether it was corrected. A simple sheet is enough. A modest sample taken carefully is worth more than a large estimate taken from memory.

Second, use ranges and descriptions. If an item takes somewhere between a short while and most of a day depending on its type, write exactly that. A range with a stated method is a legitimate baseline. A single invented figure is not.

Third, record what you cannot measure. If rework is not tracked anywhere, say so, and note how you will start tracking it. A later claim about quality then comes with a clear label: measured only after the pilot began. That is weaker evidence, and saying so is part of being credible.

If you do have systems that already log events, the observability concepts in the OpenTelemetry primer are a useful reference for what is worth capturing: traces of what happened, and when. You do not need that tooling to start, but the mindset helps. Record events as they occur rather than reconstructing them afterward.

What does a baseline look like in practice? An illustrative scenario

This is an illustrative scenario, not a real case.

A small professional services firm plans to pilot an agent that drafts first replies to incoming client requests. Before the pilot, the operations lead takes a sample of recent requests from the shared inbox, including a busy week and a quiet one. For each request she notes when it arrived, when the first useful reply went out, whether a colleague had to correct it, and which kind of request it was.

Her baseline document says that first replies typically take somewhere between a few hours and most of a working day. It says corrections cluster around requests that need a contract lookup. It records that two senior people handle almost every exception. It also quotes a short note from the team: the most draining part is searching old threads for context, not writing the reply.

After the pilot, she repeats the same method on a comparable sample. The result is nuanced. Replies to routine requests go out sooner, and the search-for-context complaint fades. Corrections on contract-related requests remain, but now they happen at a different step. Because the baseline recorded categories and not just averages, the firm can see precisely where the agent helped, where it did not and what to improve next. No one has to invent a number to say so.

How does Be Human handle baselines in managed work?

In a first engagement, Be Human defines one useful role for the agent, the memory and tools it needs, the approval boundaries and an observable success criterion. The baseline is part of that last item: before the agent goes live, we agree with the company which measures describe the current workflow, how they will be taken and who owns the record. During managed operations, recurring evaluations repeat the same method against the same example set, so changes are compared like with like. Changes to the agent are logged, so a shift in results can be traced to a specific change. For what the early period typically involves, see what the first months with a managed AI agent look like.

Which mistakes make a baseline useless?

Most failed baselines fail in the same few ways. Check your own plan against this list:

• Measuring after launch. Once people have adapted to the agent, the before can no longer be observed, only guessed.
• Changing definitions midstream. If first reply meant one thing in the baseline and another later, the comparison is invalid even if every figure is accurate.
• Sampling only good weeks. A baseline taken during a calm period makes the pilot look worse than it is, and one taken during a crisis makes it look better.
• Measuring everything. A long list of measures produces a long list of disagreements. Keep to what the agent could plausibly change.
• Letting the pilot team own the baseline alone. The person with the most to gain from a good result should not be the only one who wrote down the starting point.
• Treating the baseline as proof. It is a reference, not a verdict. Other things change during a pilot, such as staffing, seasonality and client mix, and you should note them.

The same discipline helps when you read other people's claims. Our piece on how to evaluate AI case studies asks the same questions of vendors: what was the starting point, how was it measured and what else changed. If you ask that of others, expect to be asked it yourself.

Frequently asked questions

How long should I spend capturing a baseline? Long enough to cover a normal stretch of work, including at least one busier and one quieter period, and no longer. For a single workflow this is usually a modest effort by one owner, not a project. If it starts to feel large, you have probably chosen too many measures.

What if my team already uses an AI tool informally? Then record the baseline as it is, and describe the informal use in it. Note which people use the tool, for which tasks and roughly how often. The pilot will be compared against the real current state, and that state includes the shadow tools.

Do I need special software to measure a baseline? No. A shared spreadsheet, exported timestamps from a ticketing system or inbox, and a short written note from the team are enough to start. Software helps later when you want recurring evaluations, but a baseline built by hand is perfectly valid if the method is written down.

Can I still prove value if I forgot to capture a baseline? You can build a partial one. Look for old records, such as archived threads or past reports, and sample those. If nothing exists, state plainly that results are measured only from the start of the pilot, and consider pausing the rollout to a second team so the first team can act as a comparison. Our guide to choosing the first users for a pilot covers how to set that up.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.