Scale, Extend or Stop: How to Decide After an AI Pilot

To decide after an AI pilot, compare the results with the success criteria you wrote before it started, then choose one of three outcomes. Scale if every gate is met, extend once if a single named gap remains, and stop if the evidence shows a structural problem or a risk you cannot bound. Demos and anecdotes do not count as evidence, while measured results, reviewed errors and a named owner do.

Key takeaways

• Write the decision rule before the results arrive, and tie it to the success criterion set at the start of the pilot.
• Scale only when every gate is met: result, reliability, adoption, control, cost and ownership.
• Extend once, for one named question, with an end date and a default outcome if the question stays open.
• Stop when the cause of failure is structural or the risk cannot be bounded, and treat the work already done as an asset to keep.
• Missing evidence is also an answer: if the pilot cannot show results, the measurement was the problem, and fixing it comes before any expansion.

Why is the end of a pilot the easiest place to decide badly?

A pilot ends with people attached to it. The team that built it wants it to work, the sponsor has talked about it internally, and the users who liked it are the loudest voices in the room. None of that is evidence, yet all of it shapes the meeting.

Three mistakes repeat. The first is scaling on momentum: the demo was good, so the agent goes to more teams before anyone has studied its errors. The second is extending out of discomfort: nobody wants to say yes or no, so the pilot gets another stretch with the same vague goals and arrives at the same uncertainty. The third is stopping because of one visible mistake, when the reviewed record shows that mistake was rare, caught by an approval step and fixable.

A decision rule protects against all three. It does not remove judgment. It moves judgment to the moment before the results exist, when you can still be honest about what would change your mind. If you have not written the criteria yet, how to define success criteria for an AI agent pilot walks through it. The stage distinctions in proof of concept, pilot or production explain what a pilot, specifically, is supposed to prove.

What evidence should be on the table?

Bring a short evidence packet, not a presentation. Everything in it should be something a skeptical colleague could check.

• The success criterion, the baseline it was measured against and the result, all in the same unit.
• A sample of reviewed outputs, including the failures, each classified by cause and by severity.
• Usage by the people the pilot was designed for, and the reasons given by those who did not use it.
• Approval boundaries: how often the agent reached a limit, and whether people stayed within the rules.
• A log of incidents and changes: what went wrong, how quickly it was handled, what was altered afterwards.
• Run cost per unit of completed work, not total spend during the pilot.
• The business owner by name, and how the agent would be operated if the original builder were unavailable.
• Permissions and data access as they stand today, compared with what was approved at the start.

Notice what is missing: testimonials and screenshots of impressive answers. Keep those for the announcement, not the decision.

Two habits make this packet reliable. Recurring evaluations on a fixed set of representative tasks show whether quality held steady or drifted, which a single good week cannot show. Guidance such as [Anthropic on evaluating AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) describes how to build them. Cost per unit of work, the core idea in [FinOps unit economics](https://www.finops.org/framework/capabilities/unit-economics/), tells you what scaling would cost, which a small pilot's total bill never will.

What decision rule separates scale, extend and stop?

Apply the rule in this order. First test whether every scale gate is met. If not, check the stop triggers. If none applies, test whether the extension conditions are satisfied. Each gate is marked met, not met or unknown, and unknown counts as not met.

Scale when all of these hold:

• Result: the success criterion was met on the agreed measure, against the baseline, over a period long enough to include ordinary and awkward days.
• Reliability: reviewed failures are mostly minor, no serious failure is unresolved, and the pattern of failures is steady or improving.
• Adoption: the intended users rely on the agent by choice, and you know why the others do not.
• Control: the agent stayed inside its approved permissions and approval boundaries, and every incident was handled and recorded.
• Cost: run cost per unit of work is known and sensible compared with the value of that work.
• Ownership: a named business owner exists, and there is a documented way to operate the agent without the original builder.

Scaling is a new piece of work, not a continuation. The next team needs a defined scope, a permission review and a view of capacity. How to scale AI agents across a company, one team at a time covers that sequence.

Extend when all of these hold:

• The result is promising, but one specific gap prevents a clear yes, such as a data source that arrived late or a user group that barely took part.
• The gap can be closed with a named change inside a short, fixed window.
• You can say in advance which result would turn the extension into a scale and which would turn it into a stop.
• No serious risk is open.

Stop when any of these is true:

• The success criterion was missed and the cause is structural: the process varies too much, the needed data cannot be reached, or the work is too infrequent to matter.
• Users who were given a fair chance did not adopt the agent, and the reasons are not fixable with training or small changes.
• The risk cannot be bounded: the agent would need access or autonomy you would not approve, or serious errors keep appearing despite changes.
• Run cost per unit exceeds the value of the work, with no realistic route to change that.

One caution about unknowns. If the pilot could not produce the evidence at all, you cannot honestly scale. Extend with one question, which is how to obtain the missing measurement, or stop if nobody can say how.

How should the decision meeting run?

The order of the meeting matters as much as the rule. A persuasive opening can anchor everyone before the data is seen.

• Restate the criteria exactly as written at the start of the pilot, without edits.
• Let the owner of the business process, not the builder, summarize the result in plain terms.
• Review the failures out loud, starting with the most serious.
• Walk the scale gates one by one and mark each as met, not met or unknown.
• If any gate is not met, check the stop triggers. If none applies, test the extension conditions.
• Record the decision, the owner, the date and what happens next, on one page.

The person accountable for the process makes the call, with input from the owner, the operator and whoever answers for risk. The builder presents evidence but does not decide.

How do you extend or stop without wasting the work?

An extension is a contract with yourself. Without terms it becomes a quiet way of avoiding the decision. Write down five things:

• The single question the extension exists to answer.
• The end date, fixed and short.
• The owner who will report the answer.
• The evidence that counts as an answer, defined before the window opens.
• The default outcome if the question is still open on the end date. Make it stop, or scale a narrower scope, never a second extension.

Allow one extension per pilot. A second one usually means the pilot was designed badly, and the better move is to redesign it with clearer criteria.

Stopping is a legitimate result. A pilot that shows cleanly that a process is not ready has done its job, and it cost far less than a failed rollout. What matters is how you close it:

• Remove access and credentials the pilot created.
• Decide what happens to the data and the stored memory: export, archive or delete, according to what was agreed.
• Keep the reusable parts: process maps, evaluation sets, written instructions and lessons about the data.
• Tell the users what is ending and why, so the stop does not read as a silent failure.

How to stop an AI pilot without wasting the work already done goes through these steps in detail.

An illustrative scenario

This is an illustrative scenario, not a real case. A regional distributor runs a pilot in which an agent drafts replies to routine order-status and delivery questions for its customer service team. The criterion set at the start is the time from request to a sent reply, with every draft reviewed by a person and every correction recorded.

At the end, reply time improved clearly for the main product line. Reviewer corrections were mostly wording. One serious error, a wrong delivery promise, was caught at the approval step and led to a new rule. However, the data for the second product line arrived late, so the agent handled very few of those requests.

The leadership walks the gates. Result: met for the main line, unknown for the second. Reliability: acceptable, with the serious error resolved. Adoption: good. Control: the boundaries held. Cost: known. Ownership: the service lead is named. No stop trigger applies, so they consider an extension, and it passes the test because the gap is specific and closable.

They extend once. The question is whether the agent performs equally well on the second product line. The window is fixed, the service lead owns the answer, and the default if the question stays open is to scale the main line only and remove the second from scope. Everyone leaves the room knowing what happens next.

How does a managed approach handle this decision?

Be Human designs and operates personalized AI agents. A first engagement defines one useful role, the memory and tools it needs, its approval boundaries and an observable success criterion, and that definition is what makes an end-of-pilot decision possible. During the pilot, managed operations can include monitoring, recurring evaluations, permission review and incident response, so the evidence packet is built as the work goes and not reconstructed afterwards. At the decision point, the packet is prepared against the written criterion, and the next step is agreed in writing: continued operation, a controlled extension, or a wind-down with access removed and the work documented. The decision itself stays with your leadership.

Frequently asked questions

How long should a pilot run before you decide? Long enough to include ordinary and awkward periods of the process, such as a month-end close or a seasonal peak if it matters to the work. Set the length in advance based on the process, not on convenience. Do not extend only to wait for a better-looking result.

Who should make the decision? The business leader accountable for the process, with input from the process owner, the operator and whoever answers for risk. The builder or provider should present evidence and recommendations but not decide. Having one named decision-maker also prevents the pilot from drifting between committees.

What if the results are good but the team does not use the agent? Treat adoption as a gate, not a detail. Find out why: unclear training, poor fit with the daily workflow or low trust in the output. If the cause is specific and fixable, extend once with that as the question. If people who tried it chose to stop, that is a stop trigger.

Can we scale part of the pilot and stop the rest? Yes, if every gate is met for a clearly defined part, such as one team or one product line. Write the narrower scope down and treat the remainder as stopped, or as a separate question with its own criteria. Avoid scaling a vague whole when the evidence supports only a part.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.