How to Stop an AI Pilot Without Wasting the Work Already Done

To stop an AI pilot well, agree in advance what evidence ends it, then run a short wind-down: freeze changes, record the results, export the configuration and data worth keeping, revoke access, and write down a decision to stop, pause or continue. The work is only wasted if nobody can find it afterwards. A pilot that ends with a clear record is a finished experiment, not a failure.

Key takeaways

• Write stop criteria before the pilot starts, so ending it is applying a rule rather than winning an argument.
• Pause and stop are different decisions. Pause when the conditions are temporary, stop when the case itself does not hold.
• Preserve five things: the role definition, the configuration, the knowledge sources, the evaluation cases and the lessons.
• Close access deliberately. Tokens, service accounts, channels and provider-held data stay open until someone closes them.
• End with a one-page decision record, a named owner and a review date, so the next attempt starts from evidence.

Why define stop criteria before the pilot starts?

Most pilots are launched with a clear idea of what success looks like and no idea of what ending looks like. When results are mixed, the team keeps going because stopping feels like admitting a mistake. The pilot drifts, access stays open, nobody owns it, and the lessons live in one person's head.

Stop criteria fix this by moving the hard conversation to the beginning, when everyone is calm. They belong in the same document as your success criteria. If you have not written those yet, How to Define Success Criteria for an AI Agent Pilot covers how to make them observable. For each success criterion, add its mirror: the result, by a named point in the pilot, that means you end it.

Good stop criteria share a few traits:

• They refer to something you can observe, such as reviewed outputs, escalations or time saved on a defined task.
• They name a review point, not an open-ended "we will see".
• They name who decides, and that person is a business owner, not only the technical lead.
• They cover risk as well as performance: a data boundary that cannot be met, or an action the agent took outside its approval limits.

The NIST AI Risk Management Framework treats decommissioning and the ability to disengage a system as part of managing risk across its life, not as an afterthought. A pilot is the cheapest place to practise that discipline.

Which signals say stop, and which say pause?

Pausing and stopping lead to different work. A pause keeps the option to resume, so you preserve state and keep a small amount of upkeep. A stop closes the pilot, so you preserve knowledge and remove everything else.

Use this decision guide at each review point. Read it top to bottom and act on the first line that applies.

• Stop if the success criterion cannot be reached within the current scope and you cannot name a specific change that would alter that.
• Stop if the agent needs data or access that your policy does not allow and no approved alternative exists.
• Stop if the pilot has no accountable business owner and none can be named.
• Stop if an incident showed the approval boundaries do not hold and the fix is larger than the pilot's remaining value.
• Pause if a system the agent depends on is being replaced or migrated, and the integration would need to be rebuilt anyway.
• Pause if the pilot users are unavailable for a stretch, for example during a seasonal peak, and results would be distorted.
• Pause if an incident needs investigation before anyone can judge the results.
• Continue with a narrower scope if results are mixed but you understand why, and the cause is something you can change.
• Move toward production if the criterion is met and the controls are in place. Proof of Concept, Pilot or Production: What Each AI Stage Must Prove explains what that step must demonstrate.

Two cautions. First, low usage is a signal, not a verdict. Before you stop, check whether people were trained, whether the agent sits where they work, and whether the first users were the right ones. Second, do not stop a pilot only because it surprised someone. Surprises are the reason to run a pilot.

What should you preserve when a pilot ends?

The asset of a pilot is not the agent running today. It is the knowledge of what the role should be, how it was configured, what it was tested against and what happened when real people used it. Save these items in a place the business owns.

• The role definition: what job the agent was meant to do, for whom, with what boundaries, in plain language.
• The configuration: instructions, tool definitions, settings, model choices and the version history showing what changed and why.
• The knowledge sources: which documents and systems the agent drew on, how they were prepared, and what was left out on purpose.
• The evaluation set: the test cases, the expected outcomes and the last scored results. This is often the most reusable item, because it lets a future attempt, with a different model or a different provider, be measured against the same bar. Anthropic's guidance on evaluating agents explains why a maintained set of test tasks is what turns an impression into evidence.
• Logs and the audit trail: sampled conversations, actions taken, approvals given and rejected, and any incidents.
• User feedback: what people liked, what they worked around, and what they asked for that the agent could not do.
• Integration notes: which accounts, scopes and endpoints were connected, so a later team does not rediscover them.
• The decision record: the one-page summary described below.

Store these in formats you can read without the original vendor's tools. Plain documents, exported text and structured files beat a screenshot of a dashboard. How to Avoid AI Vendor Lock-In With Business Agents goes deeper on portability, and AI Agent Backup and Restore: What to Save and How to Test It covers how to confirm that what you saved can actually be used. A backup you have never opened is a hope, not an asset.

Be selective about conversations and personal data. Keep what you need to learn from and what you are required to retain, and apply your normal retention rules to the rest. More is not safer here.

How do you wind down access, data and users?

This is the plan to follow once the decision is made. The order matters: you capture before you close, and you close before you announce that it is over.

• Announce the end date and the reason to pilot users, and tell them what happens to anything they stored in or produced with the agent.
• Freeze changes. No new instructions, tools or knowledge sources while you close down, so the final snapshot matches what was tested.
• Take a final snapshot of the configuration and record the version identifier.
• Run the evaluation set one last time against the final version and record the results, including the failures.
• Export the logs, conversations and outputs you decided to keep, and apply your retention rules to the rest.
• Review and revoke access. List every account, service identity, API key and authorization the agent held, then remove each one. Google's OAuth documentation describes how granted tokens can be revoked, and OWASP's authorization guidance supports the habit of granting least privilege and removing it when the need ends. How to Run a Permission Review for Your AI Agents, Step by Step is a good template for the checklist.
• Disable inbound channels, such as shared inboxes, chat integrations and webhooks, and leave a short redirect message so nobody writes to an agent that no longer answers.
• Ask any provider that held company data to confirm in writing what they will delete or return, and what, if anything, they retain.
• Switch off usage allowances, subscriptions and scheduled jobs, and give contractual notice where required.
• Hold a short review with the business owner and a few pilot users, and write down what you learned.
• Archive everything under a named owner with a review date, and record where it lives.

If the pilot is paused rather than stopped, do steps one through five and the channel handling, but keep the configuration in a restorable state and reduce access to the minimum rather than to nothing. Put a date on the calendar to decide again, because an undated pause is a quiet stop with unclosed doors.

Teams also need to hear it. People who used the agent should not discover it has gone from an error message. How to Announce an AI Agent to Your Team, With Message Samples can be adapted for the closing message too, since the same questions apply: what changed, why, and what to do instead.

What does a wind-down look like in practice?

Illustrative scenario: a distribution company pilots an agent that drafts replies to order-status questions for its customer service team. The success criterion is that reviewers accept most drafts with light edits. The stop criteria say the pilot ends if the order data cannot be read reliably, and pauses if the underlying order system changes.

Partway through, the company decides to replace its order system. The agent's integration would have to be rebuilt, so the review applies the decision guide and chooses a pause, not a stop. The team freezes changes, runs the evaluation set against the final version and records the results. It exports the instructions, the tool definitions and the list of knowledge sources, then removes the agent's write-capable access and keeps a read-only connection closed until the new system is ready. The shared inbox integration is switched off with a short redirect note for customers' internal contacts.

The decision record says what worked (draft quality on routine questions), what did not (unusual order changes needed a person), and what must be true to resume (a stable connection to the new system and a refreshed evaluation set). Months later, the team resumes from a written record rather than from memory. If it had instead decided to stop, the same artifacts would sit in the archive, ready for a later attempt with a different scope.

How does Be Human's managed approach handle a pilot ending?

Be Human's first engagement defines one useful role, the memory and tools it needs, the approval boundaries and an observable success criterion. Stop and pause conditions can be agreed in that same scoping step, so the end of a pilot is a planned event. During managed operations, configuration changes are versioned, permissions are reviewed on a schedule and evaluations are repeated, which means the materials a wind-down needs already exist. When a pilot ends, the work is mostly exporting what the business owns, revoking access and recording the decision, rather than reconstructing what was built.

How do you write the closing decision?

A one-page decision record is the artifact that makes the rest worth doing. It should be readable by someone who was not in the room. Include these parts:

• The decision: stop, pause until a stated date, or continue with a changed scope.
• The reason, tied to the stop criteria that were written in advance.
• The evidence: evaluation results, usage patterns and reviewed examples of good and poor outputs.
• What was learned about the process, the data and the people who used the agent.
• What was preserved and where it is stored.
• What access was closed, by whom and when.
• Conditions for trying again, and the owner who will judge them.

Keep it to plain statements. If the lifecycle after launch is unfamiliar territory, AI Agent Lifecycle Management: From Launch to Retirement shows where this record fits among the stages an agent passes through.

Frequently asked questions

Is stopping a pilot a sign that it failed? Not if you learned what you set out to learn. A pilot exists to find out, cheaply, whether a role suits an agent in your environment. A clear no, backed by evidence and a record, is a useful result. The failure is a pilot that lingers without a decision.

How long should a paused pilot stay paused? Only as long as the condition that justified the pause, and always with a review date. If the date arrives and the condition has not changed, decide again: extend with a new date or convert the pause into a stop. Open-ended pauses keep access, subscriptions and responsibility alive without anyone using them.

Who should decide to stop an AI pilot? The business owner who sponsored the pilot should decide, informed by the people who run it and the people who use it. Technical staff supply evidence, but the decision is about whether the role is worth continuing, which is a business judgment. If the stop criteria were written in advance, the decision is mostly confirming which line was crossed.

Should we delete the data when a pilot ends? Delete what you have no reason or obligation to keep, and keep what you need for learning, audit or legal retention. Apply your normal retention rules, and get written confirmation from any provider that held company data about what they will delete or return. Decide this before the wind-down, not after, so the export and the deletion do not conflict.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.