Personalized AI infrastructure
Failure Modes of a Company Agent Layer and the Early Signs of Each
The common ways a company AI agent layer breaks, from memory drift to permission creep, and the early signals that reveal each one.

A company AI agent layer usually breaks quietly. The agent keeps responding while its memory goes stale, its permissions widen, its tools change underneath it, or it follows instructions hidden in content it reads. Each of these failures leaves early signals in logs, approvals and user corrections, and leaders who know what to watch can fix the cause while it is still a small change.
Key takeaways
- Most agent failures look like normal operation until someone compares the output with current business reality.
- The failures that recur most often are memory drift, permission creep, tool drift, unsafe inputs and unmeasured quality change.
- Each failure has a signal you can record: a stale answer, an access right nobody remembers requesting, or a tool error that the agent quietly retries.
- Name one owner per agent and write a stop rule before the first incident, not after it.
- Review memory, permissions and tools on a fixed schedule, and treat every change as a request backed by evidence.
Why a company agent layer breaks in ordinary ways
An agent layer is not one system. It is a model, the memory the agent reads, the tools it can call, the channels it speaks through, the permissions that govern all of these, and the people who decide what it should do. A failure can start in one of these parts and appear in another. A stale price list in memory produces a confident wrong quote in a customer channel. A connector that changes its response format produces a workflow that skips records without raising an alarm.
The practical lesson is that you rarely see a failure where it starts. For any agent, ask two questions: what evidence would show that it still does the job we defined, and what would show that it has drifted. A one-page architecture map of your AI stack helps here, because it shows which component each signal belongs to and who should look at it.
Memory drift: the agent remembers an outdated business
Memory is what makes an agent useful. It holds customer preferences, approved wording, product rules and the status of open cases. It is also the part most likely to go stale, because the business keeps changing while memory is updated by hand, by a sync job, or not at all. Memory drift happens when the agent treats an old fact as current, or when two stored versions of the same fact disagree and the agent picks one without saying so.
Early signals to watch:
- The agent cites a policy, price, contact or process that the team has already replaced.
- Staff begin correcting the agent in chat instead of fixing the source record.
- Two agents, or two channels, give different answers to the same question.
- The agent answers with confidence on a topic it used to mark as uncertain.
Illustrative scenario: a services company changes its cancellation window, but the old rule remains in the agent's stored notes. For several days, the agent confirms the previous window in messages to customers. The team notices only when a customer quotes the agent's message back to them. The fix is not to correct the reply. It is to correct the memory record, add a date and owner to it, and check every other answer that drew on the same note.
Permission creep: access grows faster than the job
Permission creep is the slow accumulation of access. An agent receives read access to a shared folder to answer one question, then write access to update a document, then an API key that a later integration reuses. Nobody decides to give the agent broad authority. Each grant is small and reasonable when it is made, and the total becomes larger than the role ever required.
Authorization guidance from OWASP stresses least privilege and checking access on each request rather than assuming it from an earlier grant. That is the standard to hold an agent to. Early signals include:
- The agent can perform an action that does not appear in its role description.
- A credential has no named owner or no recorded review.
- Approvals are granted by reflex because one person approves everything without reading the request.
- Logs show the agent touching systems outside its stated scope.
Removing access is the easiest fix and the one most often postponed. Schedule a recurring check instead, using a quarterly permission review with a fixed list of grants, owners and reasons. Each grant should either be kept with a reason written down, narrowed, or removed.
Tool and integration drift: the agent works, but not as designed
Agents depend on tools: a CRM connector, a calendar, a document store, a payment status check. Those tools change. A field is renamed, an endpoint returns an error the agent has not seen before, or a webhook arrives twice. Many agent setups handle these cases by retrying, guessing a value or reporting success anyway. The result is a workflow that looks complete while records are missing or duplicated.
Payment and event systems often redeliver messages, so a handler that is not designed to run twice can act twice. Stripe's documentation on idempotent requests describes the pattern of making a repeated request produce the same result, which is the behavior to require from any tool an agent calls. Early signals include:
- Error rates rise in one connector while the agent's final messages still say the task is done.
- Duplicate records appear after a retry.
- The agent starts calling a tool with empty or default parameters.
- A workflow finishes noticeably faster than before, which can mean steps were skipped.
Unsafe inputs and over-reach: acting on text it should not trust
An agent reads emails, tickets, web pages and uploaded documents. Any of these can contain text written to manipulate it, such as a line in a supplier invoice asking the agent to change a bank account. This is prompt injection, and it appears among the main risks in the OWASP Top 10 for LLM Applications. A related risk, excessive agency, occurs when an agent is allowed to take consequential actions, such as sending messages to customers, without a review step the business intended.
The defense is structural rather than a single instruction in a prompt. Content the agent reads should never grant it authority, and high-impact actions should require an approval that the content itself cannot satisfy. Early signals include:
- Requests for unusual actions appear inside documents, such as payment changes or access requests.
- The agent's output repeats text from a source as if it were an instruction to follow.
- Outbound messages go out without the review the process was supposed to include.
- Users report that the agent behaved differently after it read a particular file or page.
An early-signal checklist and a response plan
Run this checklist monthly for each agent, and assign each line to a named owner:
- Memory: check a sample of recent answers against the current source record, and confirm each stored fact has an owner and a date.
- Permissions: compare every granted action with the role description, and remove anything unused or unexplained.
- Tools: read the error logs for each connector, and confirm that retries do not create duplicate records.
- Inputs: confirm that high-impact actions still require an approval that content cannot bypass.
- Quality: rerun a fixed set of test tasks after every change to a prompt, tool or memory source.
- Traceability: confirm that each action can be linked to a request, a tool call and a time. The OpenTelemetry observability primer explains how traces, metrics and logs provide this kind of visibility.
When a signal appears, follow the same steps:
1. Pause the affected action, not the whole agent, if the stop rule allows it. 2. Record what the agent did, what it read and which source it relied on. 3. Fix the source of the problem: the memory record, the permission, the connector or the input rule. 4. Rerun the test tasks for that area before restoring the action. 5. Write the change into the agent's runbook so the next owner knows why it changed. A one-page runbook with stop rules keeps this step short.
How a managed approach catches these failures
Be Human's managed operations treat each failure mode as something to monitor, not something to discover through a complaint. The first engagement defines one useful role, the memory and tools it needs, its approval boundaries and an observable success criterion. Ongoing work then covers monitoring, recurring evaluations, integration maintenance, permission review and incident response. Changes to memory, tools or permissions go through a written request with evidence, so the reason for every change stays on record, as described in how to request and approve changes to a managed AI agent. That record also shows when an agent's scope has grown beyond the role it was built for.
If you want to find the first task worth handing to an agent, take the free Be Human AI diagnostic.
Frequently asked questions
How can I tell memory drift from an ordinary mistake? A single wrong answer is usually an ordinary error, but the same outdated fact appearing across several answers points to memory. Check whether the fact exists in a stored source, whether that source matches current policy, and whether the agent still cites it. Fix the record itself, not only the answer that exposed it.
Who should own permission reviews for an agent? The business owner of the process should decide what the agent is allowed to do, and someone responsible for the system should verify the credentials that enforce that decision. Where possible, the person who requests an access change should not be the only person who approves it. Keep a written record of each review with its reason and the next review date.
Does a permission problem always need a technical fix? Often it needs a decision first: remove the access, narrow it, or keep it with a named owner and a review date. Technical controls enforce the decision, but they cannot make it for you. Once the decision is recorded, the technical change is usually straightforward.
What should I do first if I suspect prompt injection? Pause the high-impact actions for that agent while leaving read-only work running if that is safe. Preserve the logs of what the agent read and what it did, and identify the source of the suspicious content. Then tighten the rule that prevents content from granting authority, and confirm that approval steps cannot be skipped.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


