AI Agent Maintenance Checklist: A Monthly Calendar After Go-Live

An AI agent needs a maintenance calendar from the day it goes live, because its answers, permissions and connected tools drift as the business, the underlying models and the data change. A workable calendar assigns recurring checks across four areas, evaluations, permissions, integrations and memory hygiene, with one owner and one written record for each. The sections below give a month-by-month version you can adapt, along with the evidence to keep.

Key takeaways

• Give every recurring check a named owner, a due date and a written record before the first month ends.
• Rerun a fixed set of real evaluation cases every month so quality changes show up before users report them.
• Review permissions and connected tools on a set rhythm, and treat every new tool or wider scope as a change request.
• Clean agent memory on purpose: remove stale facts, confirm sources and log each deletion with its reason.
• Use the first quarter to tune the calendar, then keep the checks that caught real problems and retire the ones that never did.

Why an agent needs a maintenance calendar after go-live

A pilot proves that an agent can do a task under controlled conditions. Production adds the things that change without anyone touching the agent. A model provider may update the model behind it, a policy in your business may change, a connected tool may alter its scopes or response format, and the agent's stored memory keeps growing. Monitoring shows that something happened. A maintenance calendar makes sure a named person looks at it, decides what to do and records the decision.

Each check should answer three things: the question it settles, the person who runs it and the record that proves it was done. Without the record, a team cannot tell whether a quiet month meant nothing went wrong or simply that nothing was checked. For the first months in more detail, see What the First Months With a Managed AI Agent Look Like.

Evaluations: keep a fixed test set and rerun it

Evaluations answer one question: is the agent still doing its job at the quality you accepted? Build a fixed set of real cases from the work the agent actually does, starting with a few dozen, and include the cases it handled badly during the pilot. Write the expected outcome next to each case so a reviewer can judge an answer without reconstructing the whole context. Anthropic's guidance on evals for AI agents describes this approach of grading agents against defined tasks, and it is a useful reference when you design the set.

Rerun the set every month, and also after any change to the model, the prompts, the tools or the memory. Compare each result with the success criterion agreed before go-live, not with an impression from last month. When a score drops, the first question is whether the cause was your change, the provider's change or the data's change, because each one has a different owner and a different fix.

The month-by-month calendar

Run these checks in the same order every month so the review takes a predictable slot in the calendar. The first three months carry most of the risk, because real users find the cases the pilot never covered.

Month one, after go-live:

• Confirm the evaluation set reflects live traffic, and add every failure from the first two weeks.
• Check that logs capture each request, tool call and escalation with timestamps.
• Test the escalation path to a human and confirm that the alerts reach someone who will act on them.
• Record baseline scores and the list of permissions the agent holds today.

Month two:

• Rerun the evaluation set and compare the results with the baseline.
• Review every escalation and label whether the agent should have handled it.
• Check each connected tool for errors, timeouts and changed responses.
• Open a change request for every fix, and close the requests already completed.

Month three:

• Run a permission review: list each tool and data source and confirm the agent still needs it.
• Test a restore from the latest backup of memory and configuration, if you keep one. The steps are covered in AI Agent Backup and Restore: What to Save and How to Test It.
• Review the memory entries added during the quarter.
• Ask the people who work with the agent what it got wrong and what it could not do at all.

Months four to six:

• Rerun evaluations monthly and read the trend across months, not only the latest score.
• Rotate credentials and confirm which accounts and service identities the agent uses.
• Check provider notices for model, API or pricing changes that could affect the agent.
• Clean memory by merging duplicates and expiring facts that no longer apply.

Months seven to twelve:

• Repeat the monthly checks and run the full permission review every quarter.
• Compare the year's escalations and change requests to see which tasks grew and which shrank.
• Decide whether the agent's scope should widen, stay the same or narrow, and record that decision.
• Update the calendar itself by dropping checks that never found anything and adding checks for new risks.

Permissions and integrations: review access and connections

Permissions drift through convenience. An agent granted read access to a shared folder for one project may keep that access long after the project ends. Apply least privilege: the agent should hold only the scopes its role needs, granted through the authorization flow each tool provides. OAuth 2.0 scopes are a practical place to start, since they define what a token is allowed to do. For the full procedure, see How to Run a Permission Review for Your AI Agents, Step by Step.

Treat a new tool or a wider scope as a change, not as a routine update. In the monthly review, decide what happens to each permission with a simple rule set:

• Keep it when a current task in the scope document needs it and the owner confirms that need.
• Narrow it when the agent uses only part of the access, such as reading records but never writing them.
• Remove it when no task has used it during the review period and no owner claims it.
• Escalate it when the access touches personal data, payments or customer records, and send it to a named approver.

Integrations fail quietly. Check webhook delivery and signature verification, API version changes, credential expiry and error rates for each connection. Where an action can be retried, confirm that repeated requests cannot create duplicate records. Some providers document idempotency keys for exactly this purpose, and Stripe's documentation on idempotent requests is a clear example of the pattern. Test this behaviour once a quarter in a sandbox if the tool offers one.

Memory hygiene: keep what is true, correct what is stale

Agent memory is useful only when its contents are current and attributable. Each month, sample the entries added since the last review and ask three questions: is it still true, does it have a source, and should the agent be allowed to hold it at all? Remove duplicates, expire facts tied to campaigns, policies or people who have left, and delete any credential or personal detail that the role does not require. Log every deletion with its reason so the history remains explainable to an auditor or a new team member.

Illustrative scenario: a support team's agent still remembers a discount rule from a campaign that ended two months earlier. During the memory review, the operations lead finds the entry, confirms with the campaign owner that it has expired, removes it and records the change. The next evaluation case for that topic now expects the standard answer, and the score reflects the correction.

The NIST AI Risk Management Framework is a useful reference for organizing these reviews around risk, documentation and accountability rather than around habit.

How a managed approach keeps the calendar running

Be Human designs and operates these checks as part of managed operations, which can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. Each check produces a short written record. Changes to scope, permissions or memory pass through an approval step with a named owner, as described in How to Request and Approve Changes to a Managed AI Agent. The calendar is adjusted at each quarterly review as the agent's role changes, and it is built from the first engagement's definition of the role, its memory, its tools and its approval boundaries rather than from a generic template.

If you are not sure which of these checks should come first for your agent, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

How often should an AI agent be evaluated after go-live? Run the fixed evaluation set at least monthly, and again after any change to the model, prompts, tools or memory. Monthly runs reveal drift that individual complaints tend to miss. Compare each result with the success criterion agreed before go-live and keep the scores in a shared record.

Who should own the maintenance calendar? A named operations owner should hold it, with a technical owner for integrations and a business owner for scope decisions. When ownership is shared without a name, checks tend to be skipped in quiet months. The owner does not need to run every check personally, but must see and approve each record.

What does memory hygiene mean for an AI agent? It means reviewing what the agent stores so that entries remain true, sourced and permitted for its role. Stale facts are removed, duplicates are merged and sensitive data the role does not need is deleted. Each change should be logged with a reason.

Do we need a maintenance calendar if the agent only answers questions? Yes, although the checks can be lighter. A question-answering agent still depends on a model, a knowledge source and often a few tools, and its answers can become wrong when policies change. A short calendar with monthly evaluations and a quarterly permission check is usually enough for that case.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.