Quarterly AI Review: Decide What to Keep, Fix or Retire

A quarterly AI review for companies is a short, recurring leadership meeting where every agent workflow is examined against its original purpose and ends in one of three decisions: keep, fix or retire. It works when each workflow has a named owner, a small evidence pack and a written success criterion, so the decision rests on what the workflow did rather than on how it feels. The output is a decision log, not a slide deck.

Key takeaways

• Review workflows one by one, not the AI program as a whole. Every workflow leaves the room with a keep, fix or retire label and a named owner.
• Send a short evidence pack before the meeting: purpose, completed work, corrections, incidents, cost per completed task and permissions in use.
• Make "keep" earn its place. A workflow that cannot show it still serves its purpose should not continue by default.
• Give every fix an owner, a specific change, a date and a re-check. A workflow that is fixed twice and still fails becomes a retirement candidate.
• Treat retirement as a normal outcome. It includes revoking access, exporting what is worth keeping and telling the people who used the workflow.

Why does an AI setup need a recurring review?

Agent workflows drift. The model behind them changes, a connected tool updates its behavior, the documents they rely on go out of date, and the team's real process moves while the workflow stays still. Meanwhile permissions accumulate, because it is always easier to grant access than to remove it.

None of this announces itself. A workflow rarely breaks loudly. More often people quietly stop trusting it, start double-checking its output, or route around it. Without a scheduled moment to look, every workflow defaults to "keep" through inertia, and the cost of that choice is invisible.

A recurring review replaces inertia with a decision. The three-way format matters because it removes the comfortable fourth option, "leave it and see." Frameworks such as the NIST AI Risk Management Framework describe managing AI as a continuing cycle of mapping, measuring and managing, not a one-time sign-off. A quarterly cadence is a practical way to make that cycle real for a business that does not have a dedicated risk department.

This review is different from a pilot review. A pilot asks whether an idea deserves to go live. The quarterly review asks whether something already live still deserves its place, its budget and its access. If you have not yet written down what success means for a workflow, start with how to define success criteria for an AI agent pilot, because the review depends on that definition.

Who should be in the room, and how should the session run?

Keep the group small and give each person a clear job.

• The decision maker: an executive sponsor who can approve a retirement or a budget change on the spot.
• Workflow owners: the business person accountable for each workflow's outcome, not the person who configured it.
• The operator: whoever runs the technical side, whether an internal engineer or an external partner. They answer questions about incidents, integrations and changes.
• A risk and data voice: someone who can speak to permissions, sensitive data and any rules the company has agreed to follow.

If you are unsure who should hold these roles, AI operating model for companies: who owns, reviews and runs lays out a division of responsibilities that fits this meeting well.

The evidence pack should arrive before the session. If the first ten minutes of discussion are people reading numbers for the first time, the review will turn into a status update. The meeting itself is for judgment: what the evidence means and what to do about it.

What evidence should each workflow bring?

The goal is a one-page summary per workflow that anyone in the room can read quickly. It should answer the same questions every quarter so that trends become visible. Use this checklist.

• Purpose and success criterion: the sentence that says what the workflow is for and the observable result that shows it is working.
• Completed work: how much work finished correctly end to end, as opposed to how much was started or how many messages were exchanged.
• Human corrections and escalations: how often people had to fix output or take over, and what kinds of mistakes repeated.
• Incidents and near misses: anything that went wrong, almost went wrong or required a rollback, plus what changed afterward.
• Cost per completed task: the full running cost divided by finished work, as described in measuring AI agent cost per completed task. The FinOps Foundation's guidance on unit economics is a good reference for why a unit measure beats a total.
• Permissions in use: what the workflow is allowed to read, draft and send, compared with what it actually used during the quarter.
• Knowledge freshness: when the documents and memory it depends on were last reviewed, and by whom.
• User signal: a short note from the people who work with it, including whether they still use it and what they do instead when they do not.

Two supporting practices make this evidence trustworthy. First, keep a small set of representative test cases for each workflow and run them every quarter, so you can compare behavior over time. Anthropic's writing on evaluating agents and the OpenAI evals guide both describe this habit of testing against realistic tasks. Second, make sure logs and traces exist so the operator can answer "what happened" without guessing. The OpenTelemetry observability primer explains the basic vocabulary of logs, metrics and traces.

If some of this evidence does not exist yet, that is itself the first finding. Record it as a fix for the monitoring, and decide the workflow on whatever evidence you have.

How do you decide to keep, fix or retire?

Use a consistent rule for each label, written down, so decisions do not depend on who argues best.

• Keep when the workflow still meets its success criterion, corrections are stable or falling, permissions match actual use, and the owner confirms the process it supports has not changed. Keep still produces an action: a date for the next review and any small maintenance item.
• Fix when the purpose is still valid but evidence shows a specific, nameable problem: stale knowledge, a drifting integration, a permission that is too broad, an approval step that is missing, or a repeated class of errors. A fix must state the change, the owner, the date and how you will know it worked.
• Retire when the purpose no longer exists, the process moved elsewhere, people have stopped using the workflow, the cost per completed task no longer makes sense, or the risk cannot be brought within the boundaries you set. Also retire when a workflow has been fixed twice and the same problem returns.

A few rules keep the format honest.

• Only the decision maker can approve "keep" for a workflow without a current success criterion. If no one can say what success is, the default is fix, and the fix is to write it.
• A fix carries over at most one review. If it is not resolved by the next session, it must be converted to retire or escalated with a new plan.
• Any workflow that gained write or send authority during the quarter gets a second look at its approval boundaries. Where your AI agent should stop and ask covers how to set those.
• Retiring is a checklist, not a switch: revoke access and credentials, stop scheduled runs, export documentation and any memory worth preserving, archive the decision log entry, and tell the people who used it what replaces it.

A workflow you cannot retire cleanly is a workflow you do not fully control.

What does a step-by-step review look like in practice?

Here is a plan you can run as written for your first review, then adjust.

• Step one, two weeks before: the operator compiles the inventory of every live workflow, including informal ones that individuals set up. Anything not on the list is not reviewed, which is why the list must be complete.
• Step two, one week before: each owner receives their one-page evidence summary and adds a short note on whether the underlying process changed.
• Step three, at the start of the session: confirm the decision rules and read the incident summary for the whole quarter before looking at individual workflows.
• Step four, per workflow: the owner states the purpose in one sentence, the operator highlights the evidence that stands out, and the group discusses only what is surprising.
• Step five, per workflow: the decision maker names the label. If the group disagrees, ask which single piece of evidence would change the answer, and assign someone to get it.
• Step six, at the end: read back the decision log aloud, with owners and dates for every fix and retirement.
• Step seven, within a week: the operator carries out the changes, using a careful release routine. How to update an AI agent in production without breaking it is a useful companion here.
• Step eight, at the next review: open with last quarter's fixes and retirements. Did they happen, and did they work?

Step eight is the one most often skipped, and it is what turns a meeting into a management practice.

What does this look like for a real set of workflows?

Illustrative scenario: a professional services firm has three live workflows. The first drafts replies to inbound client requests for human approval. The second compiles a weekly status summary from project notes. The third checks incoming contracts against a list of standard clauses.

The request drafting workflow shows steady use, corrections that are stable and permissions limited to drafting. The owner confirms the process is unchanged. Decision: keep, with a small maintenance item to refresh the tone guidance that the team updated recently.

The weekly status summary still runs, but the project team moved to a different tracking tool during the quarter and the workflow now reads an old source. People have started writing the summary by hand. The purpose is still valid. Decision: fix, with the integration repointed to the new source, a named owner, a date and a re-check at the next review.

The contract clause checker was built when the firm used one contract template. Since then, most agreements arrive on client paper, the clause list has not been reviewed in a long time, and the legal lead reports that nobody opens its output. Decision: retire. Access is revoked, the clause list is exported to the legal team's own folder, and the owner tells the team what to use instead.

Notice that none of the three decisions needed a dramatic number. They needed an honest picture of use, accuracy and fit.

How does Be Human handle this as part of managed operations?

Be Human treats the review as part of the recurring operating work, not a separate consulting exercise. The operator prepares the inventory and evidence pack, keeps the representative test cases current, flags permissions that were granted but unused, and carries out the agreed fixes and retirements under the same change controls used for any production update. The decisions stay with your people. What managed AI services include describes the wider set of recurring tasks around this, and the approval boundaries and success criteria defined in the first engagement become the standard each workflow is reviewed against.

Frequently asked questions

How often should a company run an AI review? Quarterly suits most companies because it is frequent enough to catch drift and infrequent enough that there is real evidence to look at. If a workflow handles sensitive data or has authority to send messages externally, add a lighter monthly check on incidents and permissions, and keep the full decision for the quarterly session.

Who should make the final keep, fix or retire decision? The executive sponsor, with input from the workflow owner and the operator. The owner is accountable for the business outcome and the operator for technical facts, but someone with authority over budget and access needs to decide, otherwise fixes and retirements stall.

What if we have no metrics for a workflow yet? Start with what you can observe: whether people still use it, how often they correct it, and whether it has caused any incident. Then make the first fix the creation of a written success criterion and a basic way to measure it, so the next review has real evidence.

Is retiring a workflow a sign the AI project failed? No. Processes change, tools change and some workflows were useful only for a period. Retiring one cleanly frees attention, reduces risk and makes room for a better-scoped role, and it shows that your review is doing its job.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.