AI Agent Support and Maintenance: A Monthly Routine for Healthy Agents

A monthly maintenance routine for an AI agent is a fixed set of checks that confirms its connected tools still respond, its permissions still match the work it does, and its answers still pass a stable test set. Run it once a month, write down what you found, and change the agent only when a check fails or a model or policy change requires it. This is how an agent stays reliable while the models, APIs and business rules around it keep moving.

Key takeaways

• An agent depends on parts that change on their own schedules: model versions, vendor APIs, documents and staff roles.
• Run four kinds of checks every month: integrations, permissions, evals, and usage and cost.
• Keep one versioned test set and rerun it before and after every model or prompt change.
• Review access against the job the agent does today, not the job it had at launch.
• Record what changed, what passed, and who approved each change, so the next review starts from evidence.

Why agents need a monthly routine

An AI agent is not a finished piece of software that stays the same after launch. It calls a model that the provider may update or retire, it reads from systems whose fields and endpoints change, and it acts under permissions granted for a version of the business that may no longer exist. Each of these parts can drift without any visible error. The agent keeps answering in a confident tone while a tool quietly returns partial data or a document template changes its layout.

Weekly checks catch the failures that show up fast, such as a tool that stops responding. A monthly routine looks at slower problems: scopes that have widened, test results that have softened, and costs that have grown because a prompt got longer. If you want the lighter weekly pass first, the weekly maintenance checklist for managed AI agents covers the signals to watch between monthly reviews.

The monthly routine in order

Work through these steps in sequence, because later steps depend on earlier ones. A permission review means little if the inventory is wrong, and an eval score means little if a tool it depends on was broken during the run.

• Confirm the inventory of tools, data sources and channels, and check that each one has a named owner.
• Run read-only tests on every integration and log the results with timestamps.
• Review permissions for each tool against the current job, and remove scopes nobody uses.
• Rerun the versioned eval set and compare the results with last month's run.
• Check usage, cost per completed task and rate limit headroom.
• Review recent escalations and incidents, and note any failure that repeated.
• Record each decision as keep, adjust or change, with the person who approved it.

Keep the whole pass to one page of notes per agent. If a step reveals a problem that needs engineering work, record it as a task with an owner rather than fixing it in the middle of the review.

Integration checks: confirm every connection still works

Start with an inventory. List every tool, data source and channel the agent uses, the account or service identity behind each one, and the person inside your company who can answer questions about it. Many teams discover in this step that an integration has no owner, which is the first thing to fix.

Then run each connection through a read-only test. Confirm that authentication still succeeds, that responses contain the fields the agent expects, and that errors come back in a form the agent can handle. For systems that send events to the agent, check that deliveries arrive and that signatures are verified rather than assumed. Read the vendor change notices for each service, because deprecated endpoints and renamed fields are the most common reason an integration that worked last month fails today.

Log every test with a timestamp and a result. [OpenTelemetry's observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) explains the logs, metrics and traces that make such records useful, and a dated log turns a later failure into a traceable change instead of a general feeling that something broke.

[figure]

Permission reviews: match access to the current job

Permission reviews ask a narrower question than most teams expect: what can this agent do without asking a person first? Start with the list of actions, not the list of tools. Read access to a customer record and the ability to issue a refund are different permissions, even when both come from the same integration. For each action, write down whether it is allowed, whether it needs human approval, and what limits apply, such as a maximum amount or a restricted set of records.

Remove what the job no longer needs. Agents often accumulate scopes during setup and testing, and nobody removes them afterward. The [OWASP authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) is a useful reference for checking access at the point of action rather than assuming it from earlier configuration. Also confirm that write actions are safe to repeat, because a retried request should not create a second refund or a second booking. The guide to idempotent agent actions covers that pattern.

Finally, confirm that approval rules still match the company. A policy that required a manager's sign-off above a certain amount may have changed since launch. The agent should follow the current rule, and that rule should live in a document someone can point to.

Eval runs: measure quality on a fixed test set

An eval is a set of realistic tasks with known good outcomes, run against the agent and scored by rules or by reviewers. Keep the set fixed from one month to the next, so that a change in score reflects a change in the agent or its inputs rather than a change in the test. Include the cases that have failed before, the ambiguous requests, and the cases that touch money, records or customers. The [OpenAI evals guide](https://platform.openai.com/docs/guides/evals) and [Anthropic's notes on demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) both describe grading outcomes rather than impressions.

Score the outcome first. A reply can read well and still book the wrong slot or quote a superseded policy. If you use an automated judge to score free-form answers, check its verdicts against a sample that a person has graded, because judges can favor particular styles. Scores can also slide slowly without any single failure, so compare several months of results rather than only the latest run. When the line moves in one direction, the guide to spotting agent drift describes what to check next.

Illustrative scenario: a small appliance repair company uses a scheduling agent to book visits and send reminders. In one monthly run, simple bookings still passed, but several cases that moved an existing visit began failing. The cause was a calendar field that a vendor update had changed from local time to a UTC value, which the agent read as the wrong slot. The fix was a small change to how the tool's response was handled, and the failing cases were added to the test set so the next update is checked against them. This scenario is illustrative and does not describe a real client.

Model changes and the decision after each run

Model providers update and retire models on their own timelines, and a change in the underlying model can alter behavior without any change in your configuration. When a new version becomes available, run the full eval set against it before switching, and compare results task by task rather than by overall average. A small average gain can hide a regression on the tasks that matter most. The guide to AI model update regression checks describes how to structure that comparison.

Each run should end in one of three decisions. Adopt the change if the important tasks hold or improve. Adjust if the regression comes from a prompt, a tool description or a missing field, change that one thing, and rerun the set. Roll back to the previous version if the regression affects high-risk tasks and the fix is not clear. Write the decision and its reason in the record, so the next month does not reopen the question from scratch.

How Be Human runs the routine

Be Human handles this routine as part of managed operations. The first engagement defines one useful role, the memory and tools it needs, the approval boundaries and an observable success criterion. Ongoing work covers monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. Each change is recorded against the agent's approval boundaries before it is applied, so the routine stays with the agent over time rather than depending on whoever set it up.

Record results and decide what changes

The routine only helps if its output is kept. A one-page record per agent is enough: the date of the review, the result of each check, every failure, the decision taken and the person who approved it. Store it where the next reviewer will look, and link it to the audit trail for any permission or tool change.

Changes should always go through a written decision. A silent edit to a prompt or a permission is the most common reason an agent behaves differently next month with no explanation. Keep the review small and regular, and let the record show how the agent has changed over time.

If you are not sure which task should go to an agent first, the free Be Human AI diagnostic is a practical place to find the one worth starting with.

Frequently asked questions

Who should own the monthly routine? A named person in operations should own it, because the review involves business decisions about access and acceptable errors that an engineer alone cannot make. Technical staff or a managed provider can run the integration tests and the eval harness. The owner signs off on the permission review and on the decisions recorded in the review notes.

Do I need to run the full eval set every month? Yes, run the full set once a month and again after any model, prompt or tool change. Between those runs, spot-check a sample of live outputs each week to catch problems early. The full run is what keeps monthly scores comparable with each other.

What if a model update improves some tasks and weakens others? Decide task by task rather than on the average. Keep the new version for the tasks where it holds up, and keep the previous version or add a correction for the high-risk tasks where it does not. Record the reason so the choice can be revisited when the next update arrives.

Can a small team run this without engineers? The inventory, the permission review and the decision record need only business owners and a clear checklist. Integration tests and eval runs usually need someone with access to the systems, which can be an internal engineer or an outside operator. Keep the steps that require judgment with your team, even when someone else runs the tests.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.