AI Knowledge Base Maintenance: Owners and a Review Routine

AI knowledge base maintenance works when every body of knowledge has a named human owner, a review interval and a clear rule for what happens when something is wrong. Without those three things, an agent keeps answering confidently from documents that stopped being true a long time ago. This article gives you the roles, the cadence and a step-by-step routine you can start this week.

Key takeaways

• Every collection of knowledge needs one named person as owner, not a team and not a shared inbox.
• Review frequency should follow how fast the content changes, not a single calendar rule for everything.
• Separate four jobs: owning the content, running the routine, maintaining the technical pipeline and approving what the agent itself writes down.
• A short register listing owner, source, last review and review interval is the most useful maintenance tool you can build.
• Test the agent against real questions after each review, so you know the knowledge works and not only that it exists.

Why does an AI knowledge base decay?

A knowledge base rarely fails in one dramatic moment. It erodes through small changes that nobody connected to the agent. A policy is updated in one document but the older version is still indexed. A team renames a process and the agent keeps using the old name. A folder is reorganized and some files quietly stop syncing. The agent does not notice any of this, because from its side every retrieved passage looks equally authoritative.

There are five common ways it happens:

• Facts change at the source and the copy the agent reads does not.
• Two versions of the same rule exist, and retrieval picks whichever matches the question wording better.
• Documents lose their owner when someone changes role or leaves, so nobody feels responsible for them.
• Notes the agent saved during work accumulate without anyone confirming them.
• A connection or permission changes and part of the source silently disappears from view.

The result is the pattern described in how to stop stale context from misleading your AI agent: the answers sound fluent, the sources look legitimate and the content is wrong. Maintenance is the routine that catches this before a customer or a colleague does.

It also helps to be clear about what you are maintaining. If the agent has a defined source of truth for each kind of question, maintenance means keeping that source accurate and keeping the copies the agent reads in step with it. If there is no defined source, the first maintenance task is to choose one.

Who should own each part of the knowledge base?

The most common failure is vague ownership: "the operations team looks after it." In practice that means nobody does. Assign four distinct roles, and make each one a person with a named backup.

• Content owner: accountable for the accuracy of one topic area. This is usually the person who would be asked to explain it if the agent gave a wrong answer, such as the support lead for refund rules or the finance lead for invoicing procedures.
• Knowledge steward: runs the review routine, keeps the register, chases owners and records what changed. This is a coordination job, not a writing job, and one steward can serve the whole company in a smaller organization.
• Technical custodian: looks after ingestion, indexing, connections and permissions. When a source stops syncing, this person finds out and fixes it.
• Memory approver: a human who confirms or rejects things the agent proposes to remember, so that its own notes do not become unreviewed policy.

Two rules keep this honest. First, an owner must have the authority to change the source, not only to comment on it. Second, ownership follows the person's role, so when someone changes jobs the register is updated the same week. The handoff practice in an AI agent handoff template that preserves the work applies to knowledge too: ownership should transfer with a short note, not by accident.

Owners also decide what is not in the base. Not everything a company knows should be available to an agent. Personal notes, client-specific material and drafts belong elsewhere, and where to draw those lines is covered in shared vs personal memory in company AI agents.

How often should the knowledge be reviewed?

Use a cadence based on volatility. Content that changes constantly needs frequent checks, while stable reference material needs only occasional ones. A single quarterly review for everything is too slow for some content and wasteful for the rest.

• On change: any time a policy, procedure, product detail or contact changes at the source, the owner confirms the agent's copy was updated. This is the most important trigger and costs the least.
• Weekly: the steward scans a short list of what the agent could not answer, what users corrected and what failed to sync. Fifteen minutes is enough if the list is well kept.
• Monthly: owners of fast-moving areas (pricing rules, availability, active procedures) re-read their material and confirm or update it. The memory approver clears the queue of proposed notes.
• Quarterly: the whole register is reviewed. Stable material is confirmed, duplicates are merged, orphaned documents are assigned or retired, and the test questions are refreshed.
• After an incident: any time the agent gives a wrong answer traced to the knowledge, the owner reviews the surrounding material, not just the single document.

These intervals are starting points. Adjust them after a few cycles. If a document is confirmed unchanged several reviews in a row, lengthen its interval. If corrections keep arriving for an area, shorten it.

What is a step-by-step routine you can start this week?

You do not need new software to begin. You need a list, some names and a recurring calendar slot. Work through these steps in order.

• Step 1: List every source the agent reads, including folders, wikis, shared drives, ticket histories and anything it has saved itself. Include sources people forget, such as an old export that is still indexed.
• Step 2: Group the sources into collections by topic, such as policies, product information, procedures and client-facing templates.
• Step 3: Create a register with one row per collection: owner, backup, source location, last reviewed, review interval, volatility (fast, medium, slow) and what the agent is allowed to do with it (quote, summarize or only reference).
• Step 4: Ask each owner to confirm they accept the role. If nobody will own a collection, that is a signal to remove it from the agent's reach until someone does.
• Step 5: Set review intervals from volatility, and put the dates in the owners' calendars with a short checklist attached.
• Step 6: Write the review checklist: is it still true, is there a newer version elsewhere, does it contradict another collection, is the title and date clear, should any part be removed or restricted.
• Step 7: After each review, run a handful of real questions through the agent and compare the answers with what the owner expects. The method in how to test retrieval quality in an AI knowledge base fits well here.
• Step 8: Record what changed in a one-line log entry. A short history makes it much easier to explain a surprising answer later.

Preparing the documents themselves matters as much as reviewing them. Clear titles, visible effective dates and one topic per document make every later review faster, as described in how to prepare company documents for an AI agent.

What does this look like in practice? An illustrative scenario

This scenario is illustrative, not a description of a real engagement.

A small services company has an agent that answers customer questions about cancellations. The cancellation policy lives in a shared document owned by the head of customer care. The policy changes, and the head of customer care updates the document the same afternoon. Under the routine, that edit triggers the on-change step: she tells the steward, who checks whether the agent's index has picked up the new version.

The technical custodian finds that the sync picked up the new file but the old version was exported to a separate folder last year and is still indexed. The agent could therefore retrieve either one. The custodian removes the old export, the steward adds a log entry, and three test questions about cancellation timing are run against the agent. Two answers match the new policy. One still mixes in wording from an older help article, which the owner then corrects.

Nothing here required special tooling. It required a named owner who noticed the change, a steward who followed it through and a short test that exposed the remaining inconsistency. Without the routine, the old export would likely have gone on producing the occasional wrong answer for months.

How do you know the routine is working?

A routine that nobody checks becomes a ritual. Watch a few signals that show whether knowledge is staying healthy:

• Overdue reviews: how many collections have passed their interval without confirmation. This is a plain process measure, and it should trend toward zero.
• Unanswered and corrected questions: the topics where the agent said it did not know, or where a person corrected it. Clusters point to gaps or outdated content.
• Conflicts found: how many contradictions between documents each review turns up. A falling count means ownership is taking hold.
• Test results: whether the agent still answers the standing test questions correctly after each change. If you maintain a golden test set, add a question whenever a real error is found.
• Orphaned content: documents with no owner, or owners who have left. Each should be assigned or removed.

The weekly report can carry these signals alongside other results, as outlined in an AI weekly report that shows coverage and decisions. Keep it short enough that the people who need to act will read it.

Removal deserves its own discipline. Some knowledge should be corrected, some should expire on a date and some should be deleted outright. How to delete AI agent memory walks through those three options. The general principle also appears in Anthropic's guidance on [effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents), which treats context as a limited resource to be curated rather than a store to be filled. A smaller base that is current usually serves an agent better than a larger one that nobody checks.

How does Be Human handle this in managed operations?

Be Human treats knowledge maintenance as part of operating an agent, not as a one-time setup task. In an engagement, the knowledge sources an agent relies on are listed with owners and review intervals agreed with the client. Be Human monitors the integrations that feed them, runs recurring evaluations against agreed test questions, flags overdue reviews and conflicting material, and routes memory proposals to the person who approves them. Changes to what the agent knows are made within the approval boundaries the client has set, and the content owners remain people inside the client company.

What should you do first?

Start small. Pick the one collection that causes the most visible damage when it is wrong, usually customer-facing policy or procedure. Name its owner, set a review date, write five test questions and run them after the next change. Once that works, extend the register to the rest. The goal is not a perfect knowledge base. It is a base where every part has someone who is responsible for it and a date when it will next be looked at. Good retrieval technology, such as the approaches described in [AWS's overview of retrieval-augmented generation](https://aws.amazon.com/what-is/retrieval-augmented-generation/), can only return what the sources contain, so the quality of the sources is a human responsibility.

For teams that want a governance frame around this, the [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) treats clear roles, monitoring and ongoing review as basic practice. A review routine for the knowledge base is a small, concrete way to put that into effect.

Frequently asked questions

How often should an AI knowledge base be reviewed? It depends on how quickly the content changes. Review fast-moving material monthly and stable material quarterly, and always check it when the source changes. Adjust the intervals after a few cycles based on how many corrections each area produces.

Who should own the knowledge base in a small company? One person can hold the steward role, but content ownership should still sit with whoever knows each topic best. In a very small team, one person may cover several roles, as long as the register shows who is responsible for what and who is the backup. Avoid assigning ownership to a group, since a group tends to mean nobody in practice.

Should the agent be allowed to update its own knowledge? It can propose additions, but a human should approve anything that will influence future answers about policy, pricing rules or commitments. Low-risk working notes can be accepted with lighter review, and they should expire unless someone confirms them. Keeping the agent's notes separate from official documents makes this much easier to manage.

What if nobody wants to own a collection? Treat that as information. If no one will stand behind a set of documents, the agent should not present them as reliable. Remove the collection from its reach, mark it as reference only, or assign an owner before it is used again.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.