How to Respond to an AI Agent Incident: A Practical Playbook

To respond to an AI agent incident, stop the harmful action first, then tell the people who own the affected work, preserve the evidence, and only then diagnose and repair. Keep that order: contain, notify, preserve, diagnose, correct, review. The goal is to limit damage quickly and leave the company with a changed system, not just a quiet apology.

Key takeaways

• Contain before you investigate. Pause the agent's ability to act, not necessarily the whole system, so the problem stops growing while you look.
• Name one incident owner. Shared responsibility during an incident usually means nobody decides.
• Tell people in layers: those who must act now, those affected by the mistake, and those who need to know later. Each group needs different information.
• Record the facts while they are fresh: what the agent did, what it saw, who approved it, and what was undone.
• Close every incident with a review that changes something concrete: a permission, a test, a rule, or a memory entry.

What counts as an AI agent incident?

An incident is any time an agent takes or prepares an action that the company did not intend and that has, or could have, a real effect on a customer, a record, money, or a commitment. A wrong answer in a draft that a person caught is a quality issue. The same wrong answer sent to a customer is an incident.

It helps to agree on categories before anything goes wrong, because under pressure people argue about labels instead of acting. A workable set looks like this:

• Wrong action: the agent did something it was permitted to do, but in the wrong case (a refund to the wrong account, a message to the wrong contact).
• Out-of-bounds action: the agent did something it should never have been able to do, which points to a permission problem.
• Wrong information: the agent stated something false or outdated to a person outside the team.
• Data exposure: the agent revealed, copied, or stored information in a place it should not be.
• Manipulation: text inside a document, email, or web page steered the agent away from its instructions. The [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) lists this and related risks, and it is a useful vocabulary for the review.
• Silent failure: the agent stopped doing its job and nobody noticed.

You do not need a long taxonomy. You need enough categories that the person on call can say which one this looks like, because the category decides who gets called.

What should you do in the first minutes?

The first minutes are for containment, and containment is narrower than people expect. You are not trying to understand the incident. You are trying to make sure it does not produce a second, third, or tenth instance while you work.

A plan the on-call person can follow without thinking:

• Stop the action path. Disable the specific tool, channel, or permission the agent used. Pausing outbound email is better than shutting everything down if the damage came from email.
• Halt queued work. Check for scheduled sends, pending approvals, and retries that will fire later. A paused agent with a full queue can still do harm.
• Do not delete anything. Logs, conversation history, memory entries, and tool outputs are your evidence. Leave them as they are.
• Note the time and who made the call. Write down when you contained it and by what means, even in a rough message to yourself.
• Assign the incident owner. One person decides what happens next and who is told. Everyone else supports.
• Check whether the same pattern is happening elsewhere. If one customer received a wrong message, look for others in the same batch before assuming it was a single case.

Whether the agent can be stopped quickly is mostly decided long before the incident. If the only way to pause it is to ask the person who built it, the design has a gap. An agent should have a documented off-switch for each channel and tool, and a person in the company who is allowed to use it without asking permission.

This is also where least privilege pays off in practice. An agent that could only draft, or only touch one folder, limits how much there is to contain. An agent with broad access turns every incident into a wide search. The [OWASP authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) makes the same point for software generally: grant the minimum needed and check it on every request.

Who needs to be told, and what do they need to hear?

Notification fails in two ways: too few people hear early, or everyone hears the same long message and nobody knows what is theirs to do. Work in layers, and give each layer only what it needs.

The people who act now. These are the owner of the affected process, whoever can reverse the action, and whoever manages the agent's access. Tell them directly, not through a general channel. The message should say what happened, what has been contained, what you need from them, and when you will update them. This is the same principle as in human escalation for AI agents: send the decision, not the mess.

The people affected. If a customer, a candidate, a patient, a student, or a supplier received something wrong or had their data exposed, someone needs to contact them. That contact should come from a person, be plain about what happened, say what you are doing about it, and not speculate about cause. If you are unsure, say you are still confirming details and will follow up. Do not wait for the full investigation to send a short first message.

Leadership and legal or compliance. Anything involving personal data, a contract commitment, a regulated activity, or a financial effect should reach whoever is responsible for those areas quickly. Whether a formal notification duty applies depends on your sector and jurisdiction, and that is a question for them, not for the person running the technical response. Frameworks like the [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) are useful for structuring who is accountable for managing AI risk in the first place.

The wider team. Colleagues who work alongside the agent should know what is paused, what to do instead, and what not to repeat. A two-line note is enough: the agent is not handling X right now, do it manually, send questions to the incident owner.

Avoid a common mistake: blaming the model in the first message. Early statements about cause are usually wrong, and they shape how people think about the fix. Describe the effect, not the explanation.

What should you record while it is fresh?

A good incident record is written during the response, not reconstructed a week later. It does not need to be elaborate. It needs to be complete enough that someone who was not there could understand what happened and why you decided what you did.

Capture these items:

• Timeline: when the wrong action happened, when it was noticed, who noticed, when it was contained, when affected people were told.
• The action itself: what the agent did, in which system, to which records or people, and the exact output or message where relevant.
• Context at the time: the instruction it was following, the documents or memory it relied on, the tools it called, and the version of the configuration or model in use.
• Authority: whether a person approved the action, whether it fell inside the agent's permissions, and who set those permissions.
• Impact: who or what was affected, whether it is reversible, and what has been reversed so far.
• Decisions: what you chose to do and why, including what you chose not to do.

Most of the context items are only available if you log the right things in advance. That means recording tool calls, inputs, outputs, and decisions in a form you can search. The [OpenTelemetry observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) explains the idea of reconstructing what a system did from its traces, logs, and metrics, and our article on what to log for AI agents covers the agent-specific details. If the evidence you need was never recorded, say so in the incident record. That gap is itself a finding.

Take a snapshot of the agent's memory and configuration at the moment of the incident, before anyone edits them. Later fixes will change the state, and you will want to compare.

How do you find the cause without guessing?

Once the situation is stable, diagnose. The temptation is to pick the first plausible explanation and patch it. Resist it. Work through the layers in turn, from the cheapest to check to the hardest.

• Instruction: was the instruction ambiguous, missing a case, or in conflict with another? Many incidents are a gap in the written procedure.
• Information: did the agent read something outdated, wrong, or irrelevant? Stale memory and conflicting documents are frequent causes. See how to stop stale context from misleading an agent and giving the agent a reliable source of truth.
• Input: did a message or document contain text that redirected the agent? Treat anything from outside the company as potentially hostile.
• Permission: should the agent have been able to take this action at all? If not, the fix is access, not wording.
• Approval: was a human check supposed to happen, and did it? A review step that people click through is not a control.
• Tool: did a tool return something unexpected, fail partway, or run twice? Repeated actions after errors are covered in a retry policy that avoids repeated mistakes.
• Model or configuration change: did something change recently? Updates can alter behavior without anyone touching the instructions.

Try to reproduce the failure in a safe setting using the recorded context. If you can reproduce it, you can confirm the fix. If you cannot, say so honestly in the record and treat the cause as unconfirmed, then add monitoring to catch a recurrence.

Repair in this order: reverse the harm to people and records, fix the cause, add a test that would have caught it, and only then restore the agent's authority. When you restore it, do so gradually. Start with drafting only or approval required, and widen again once the behavior is observed to be correct.

What does the review afterward need to change?

A review that ends in "we will be more careful" has changed nothing. Hold a short, blameless review within a reasonable time after the incident is closed, with the people who handled it and the owner of the process. Ask four questions: what happened, why did our controls not stop it, how did we find out, and what will be different next time.

Then require at least one concrete change from this list:

• A permission narrowed or removed.
• An approval step added, or an existing one made meaningful.
• A new case added to your golden test set so the same failure is checked on every future change.
• An instruction or procedure clarified.
• A memory entry corrected, or a source document fixed.
• A new alert for a signal you missed.
• A line in the operating runbook so the next person on call knows what to do.

Assign each change an owner and a date to check it was done. Also decide whether the incident changes the agent's scope. Sometimes the right outcome is that a task should return to people for now. That is a legitimate result, not a failure of the project.

An illustrative scenario

This is an illustrative example, not a real case. A services firm uses an agent to send appointment reschedule notices. One morning a staff member notices that several clients received notices for appointments that were not changing.

The on-call manager, named incident owner within minutes, disables the agent's outbound email tool and clears the queue of scheduled notices. She leaves the agent's logs and memory untouched. She messages the operations lead and the person who manages client relationships directly, with what happened and what is paused. A short, plain correction goes to the affected clients from a named person at the firm.

The record shows the agent matched clients to a calendar export that had been regenerated with shifted rows, so it was reading a stale mapping. No approval step existed for batch notices. The fixes: the agent now reads from the live calendar, batch sends need a person to approve a sample, a test case reproduces the shifted export, and the firm adds an alert for unusual send volume. The agent returns to work in draft mode first, then regains sending rights after it behaves correctly in review.

How does a managed setup handle this?

Be Human designs and operates agents with incident response in mind from the start. Each agent has defined permissions, approval boundaries, and a documented way to pause individual tools or channels. Logging is set up so that actions, inputs, and decisions can be reviewed afterward. Under managed operations, Be Human can support monitoring, incident response, permission review, and controlled improvements, while the client keeps the authority to decide what the agent may do and when it resumes. Findings from an incident go back into tests, instructions, and the runbook, and the weekly checks described in our maintenance checklist look for the same weak points.

Frequently asked questions

Should we shut the whole agent down when something goes wrong? Usually not. Disable the specific tool, channel, or permission involved, and halt queued work, so the problem stops without losing the parts that are working. Shut everything down only if you cannot tell what the agent can reach or if the cause is unclear and the risk is high.

Who should own an AI agent incident? One named person with the authority to pause the agent, contact affected people, and pull in help. It is often the owner of the business process the agent supports, backed by whoever manages the technical side. Without a single owner, decisions stall.

Do we need to tell customers every time an agent makes a mistake? Not every mistake. If a person caught it before it reached anyone, an internal record is enough. If it reached a customer, exposed personal data, or created a commitment, a person should contact them promptly and plainly, and your legal or compliance contact should decide on any formal duties.

How do we prepare before an incident happens? Write down who can pause the agent and how, keep logs detailed enough to reconstruct actions, limit permissions to what the task needs, and run a practice scenario so the first real response is not the first attempt. Review the plan whenever the agent gains a new tool or channel.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.