Monthly Service Review With Your AI Provider: A Meeting Agenda

A monthly service review with your AI provider is a short, fixed-agenda meeting that turns the provider's report into decisions, named owners and dated next actions. Run it as a working session rather than a presentation: read the evidence, decide what changes, and leave with a written record. A good agenda covers outcomes, quality, incidents, access, usage, change requests and the priorities for the coming month.

Key takeaways

• The review exists to produce decisions. A report that ends with no owner and no date has not been reviewed, only read.
• Send the report before the meeting and spend the meeting time on questions, not on reading slides aloud.
• Use the same agenda every month so trends become visible and surprises become easier to spot.
• Every item on the agenda should end in one of four outcomes: keep, change, investigate or stop.
• Keep a one-page decision log. It is the memory of the relationship and protects you when people change roles.

What is a monthly service review meant to produce?

Most provider reports describe activity: how many tasks the agent handled, which integrations ran, what was patched. Activity is not the same as value. The review exists to connect that activity to the one useful role the agent was set up to perform, and to the success criterion you agreed when the engagement began.

Think of the meeting as a small governance loop. The provider brings evidence. Your side brings business context: what changed in the team, the customers, the process or the rules this month. Together you decide whether the agent is doing its job, whether it is safe to keep doing it that way, and what should be different next month.

If you have not yet written down what the agent is supposed to achieve, fix that first. The guide on how to define success criteria for an AI agent pilot is a good starting point, because a review without a criterion turns into a conversation about impressions.

The output of each meeting is small and concrete:

• A short list of decisions, each with a named owner.
• A dated next action for every decision.
• Updated notes on risks, open incidents and pending changes.
• A clear statement of what will be reported next month.

Who should attend, and what should arrive before the meeting?

Keep the room small. Three roles matter most. The business owner of the agent's role, the person who feels the benefit or the failure in daily work. A budget or executive sponsor who can approve or decline changes. And the provider's named operator, who can answer technical questions without escalating. Add a security or IT contact only when access or data topics are on the agenda.

The provider should send the report at least a working day before the meeting, and your side should read it. If nobody on your side has time to read it, that is itself a finding: the review is too heavy, or the role has no real owner.

A useful pre-read contains these items, each short:

• Outcomes measured against the agreed success criterion, with the method of measurement stated.
• A sample of real work the agent performed, including some that went badly, with the reasons.
• Incidents and near misses, with cause, fix and what was done to prevent a repeat.
• Access changes: who or what gained, lost or still holds permissions.
• Usage and cost trend against the limits you set.
• Open change requests, with status.
• Maintenance performed on integrations, and anything that is about to expire or be deprecated.

On your side, bring two things the provider cannot know: feedback from the people who use the agent, and any change in the business that affects the role. For a method to collect the first, see how to collect employee feedback on an AI agent and act on it.

What is the agenda, item by item?

The agenda below fits in about an hour. The time boxes are guides, not rules. Spend longer on whatever the report suggests is going wrong, and shorten what is stable.

• Opening, five minutes: review last month's actions. Each is closed, carried over with a reason, or dropped on purpose.
• Outcomes, ten minutes: compare results with the success criterion. Ask whether the trend is improving, flat or declining, and whether the measure still matches what the business cares about.
• Quality and failures, ten minutes: read real examples, not summaries. Look at the worst cases first, then a few ordinary ones.
• Incidents and near misses, five minutes: confirm cause, containment and prevention. A near miss that was caught is useful information, not an embarrassment.
• Access and permissions, five minutes: confirm that the agent, its tools and the people who can change it still match what the role needs.
• Usage and cost, five minutes: compare actual use with the limits and alerts agreed, and ask what is driving any change.
• Changes and backlog, ten minutes: decide which requests to approve, defer or decline, and who signs each.
• Close, five minutes: read the decisions, owners and dates aloud and confirm them in writing.

A few notes on the items that usually need the most care.

On quality, insist on seeing the evidence behind the numbers. Evaluations are only useful if the test cases reflect your real work, so ask how the provider's recurring checks were built and when they were last refreshed. The approach in reviewing an AI agent like a role works well here: goals, evidence, feedback and changes, in that order.

On access, the review is a light check, not the full exercise. Confirm that nothing was added without approval and that departures from your team have been reflected. Schedule the deeper exercise separately, following how to run a permission review for your AI agents.

On cost, look for the reason behind a movement, not only its size. Usage can rise because adoption is succeeding or because something is looping. The first deserves a capacity conversation, the second an investigation. A sound setup of usage limits and budget alerts makes the question much easier to answer, and a unit view of cost per completed piece of work, as the FinOps community describes it, is more informative than a total.

On changes, keep the discussion tied to a written process. If requests arrive informally in chat, the review becomes the place where scope quietly grows. The process in how to request and approve changes to a managed AI agent keeps the meeting focused on decisions rather than on negotiation.

How do you turn a report into a decision?

The common failure is leaving an item at the level of observation: "usage went up", "two incidents occurred". Push each one to an outcome. The guide below maps typical findings to a default response. It is a starting point, and your own judgment about risk should override it.

• If outcomes improve and quality is steady: keep the current setup, and ask whether the role can take on a small, adjacent task next month.
• If outcomes are flat for more than one review: investigate. Ask whether the criterion is wrong, the agent lacks context, or the process around it has changed.
• If quality failures cluster around one kind of request: change. Request a defined fix, such as better source material or a narrower scope for that case.
• If a failure involved wrong data being shared or an action taken without approval: treat it as an incident first, then review the permission and approval boundaries before anything else on the agenda.
• If usage rises with no matching rise in outcomes: investigate before approving more budget.
• If the team is working around the agent: ask users why, and decide whether the problem is training, trust or fit. See how to get your team to actually use an AI agent.
• If the role no longer matters to the business: stop, deliberately, and save what was learned.

For every decision, ask the same three questions. Who owns it? By when? How will we know in the next review that it worked? An action without all three is a wish.

What does a review look like in practice? An illustrative scenario

This is an illustrative scenario, not a real engagement.

A regional distributor has an agent that drafts first replies to customer questions about order status. The monthly report shows that the share of drafts accepted without editing has been steady, but the sample of poor drafts contains several cases where the agent quoted a delivery window that the warehouse team had already changed.

In the quality item, the customer service lead explains that warehouse updates are posted in a shared channel the agent cannot read. The provider confirms that the agent only reads the order system. The group decides on three things. The provider will propose, as a change request, a read-only connection to the warehouse update source, owned by the provider operator and due before the next review. The service lead will decide whether the agent should escalate to a person whenever a delivery window is older than the latest warehouse note, due this week. The sponsor will approve or decline the new connection after the provider describes exactly what it can see.

Notice what happened. A pattern in the evidence became a diagnosis, then three actions with owners and dates. Nobody argued about whether the agent was good or bad.

What should you record after the meeting?

Write the record within a day, and keep it to one page. Longer notes are rarely read. A useful format has four blocks:

• Decisions made, each in one sentence.
• Actions, each with an owner and a date.
• Risks and open incidents, with status.
• Topics deferred to next month, with the reason.

Store the record where a new colleague can find it. Over several months, this page becomes the best account of how the agent has changed and why. It helps when a sponsor moves on, when a contract comes up for renewal, or when you need to explain a decision to leadership. The notes also feed a later conversation on what to review first when renewing an AI services contract.

Use the record to improve the review itself. If an agenda item never produces a decision, shorten it. If a topic keeps appearing in the deferred list, give it its own session.

How does Be Human handle the monthly review?

Be Human operates agents for companies as managed infrastructure, so the review is part of the service rather than an extra. Before each meeting, the operator prepares a report that follows the same structure every month: outcomes against the agreed criterion, quality samples, incidents, access state, usage against limits, open change requests and integration maintenance. Anything that needs a decision is marked as such, so the meeting can start from questions.

Approval boundaries stay with your side. The provider proposes changes, explains their effect and waits for sign-off before applying them. Decisions and actions are written into a shared log, so the next review begins by checking what was done. Observability data, such as traces and logs of what the agent did, supports the report, in the spirit of the observability concepts that the OpenTelemetry project describes. The service is remote, in English and Portuguese.

Frequently asked questions

How long should a monthly AI service review take? About an hour is enough for one agent with a single role, provided the report is sent and read beforehand. If the meeting regularly overruns, the usual causes are a missing pre-read or too many topics. Split out the deep topics, such as a full permission review, into their own sessions.

Who should lead the meeting, the provider or the client? The client should chair it, because the purpose is to decide what the business needs. The provider presents evidence and answers questions, and also writes the first draft of the record. Having the sponsor or role owner lead keeps decisions tied to business priorities and not to technical activity.

What if the provider's report is mostly activity counts? Ask for outcomes against the success criterion and for real examples of work, including failures. Activity counts such as tasks handled say little about value on their own. If the provider cannot connect its numbers to your criterion after a couple of reviews, treat that as a signal to revisit the scope of the engagement or how it is measured.

Should the review happen if nothing went wrong? Yes. Steady months are when you notice slow drift, rising use, expiring credentials and changes in the business that the agent has not caught up with. A shorter meeting is fine, but skipping it removes the habit and the record that make the hard months easier to handle.

What should we do when the review reveals a serious problem? Handle it outside the normal agenda: contain it, name an owner and agree a date for the cause and the fix. If the agent needs to stop, decide that deliberately and preserve the work done so far, as described in how to stop an AI pilot without wasting the work already done. Then bring the lessons back to the next regular review.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.