Skip to content
be-human.ai
← Back to the journal

Testing & reliability

AI Agent Incident Postmortem: A Blameless Template for Failed Runs

A blameless postmortem template for failed AI agent runs. Record the trigger, tool calls, memory state and the control that should have stopped it.

Be Human typographic cover with the title: AI Agent Incident Postmortem: A Blameless Template for Failed Runs
Conceptual illustration created with AI · Be Human

An AI agent incident postmortem is a blameless written review of a run that produced a wrong, harmful or unsafe result. It records the trigger, the tool calls and the memory state the agent used, and it names the control that should have stopped the run before the damage spread. The purpose is to change the system so the same failure is caught earlier, not to decide who was at fault.

Key takeaways

  • Write a postmortem for any run that caused a wrong external action, exposed private data or needed manual repair, not only for outages.
  • Reconstruct the run in order: trigger, context, each tool call with inputs and outputs, memory reads and writes, and the final action.
  • Treat memory state as evidence. Record what the agent knew at the moment it acted, including stale or conflicting facts.
  • Always name the control that should have stopped the run, then check whether it existed, was configured correctly and actually fired.
  • Close each review with one or two concrete changes, each with an owner and a test that proves the change works.

When a failed run deserves a full postmortem

Not every error needs a full review. A harmless retry that finished cleanly can be recorded in a single line in the run log. A full postmortem is worth the time when one of the following is true:

  • The agent sent a message, changed a record, issued a booking or made a commitment to a customer or supplier.
  • The agent read or exposed information it should not have had access to.
  • A person had to undo, correct or apologize for work the agent produced.
  • The same failure has appeared before, even when the visible symptom looked different.
  • The agent kept running after a signal that should have stopped it.

Decide the severity before the review starts. Severity determines who attends, how quickly the review must happen and whether the agent should be paused while the team investigates. A postmortem that takes a week to start has lost most of its evidence, especially the memory and log details that disappear quickly.

The postmortem template, field by field

Keep one page per incident and fill every field. Where the evidence is missing, write unknown and explain what would have been needed to know. That gap is itself a finding, because it tells you which logging or memory decision to revisit.

  • Incident ID, agent name and role, environment (test or production) and the exact time window of the run.
  • Summary: one plain paragraph describing what happened in business terms, without blaming a person.
  • Trigger: the input, schedule, event or message that started the run, with its source.
  • Impact: what changed in the outside world, who was affected and how the failure was discovered.
  • Tool calls: an ordered list of each call with target system, parameters, response and whether it was retried.
  • Memory state: the memory and context items available at each decision point, with their sources and timestamps.
  • Stop control: the control that should have halted the run, its configured threshold and whether it fired.
  • Contributing factors: conditions that made the failure likely, such as ambiguous instructions, a stale record or a missing status check.
  • Corrective actions: changes, each with an owner, a verification method and a review point.
  • Open questions: what the team still cannot explain and who will investigate it.

The structure is deliberately close to a runbook entry. Teams that already keep operating notes for their agents usually find the postmortem easier to write, and the fields double as a checklist for future incidents. For a broader view of the notes worth keeping, see AI agent runbook operating notes.

Reconstruct the trigger and every tool call

Start from the trigger, not from the symptom. A complaint about a wrong invoice may trace back to an inbound email that contained instructions the agent treated as data rather than as content to summarize. Note the exact input, where it came from and when it arrived. If the trigger was a schedule or a webhook, record the event identifier as well.

Then list the tool calls in the order they happened. For each call, capture the target system, the parameters sent, the response received and any retry. A common gap appears when a call succeeds at the tool level but the business outcome is wrong, for example a write that returns success but writes to the wrong record. Record that gap explicitly, because it points to a missing verification step rather than a tool failure.

Reconstruction depends on logs that share one run identifier across reasoning steps, tool calls and external writes. The OpenTelemetry observability primer explains the general concepts of traces, logs and metrics that make this kind of reconstruction possible. If a tool call has no log entry at all, that absence becomes the first corrective action. The question of what to capture at each step is covered in AI agent observability: what to log.

Record the memory state at the moment of failure

Memory is often the least examined part of an incident, and it is frequently the cause. For each decision point, record which memory items were retrieved, from which store, with what timestamp, and whether the agent used them in its output. Also record the important items that were not retrieved. An agent cannot be blamed for ignoring a fact it never received, but the review should explain why the fact was not in its context.

Illustrative scenario: a scheduling agent confirms a meeting room that a facilities note released a few days earlier. The old release note was retrieved and used. A newer cancellation had been stored under a different customer field and was never retrieved. The postmortem records both facts. The corrective action is not to label the agent careless, but to add a rule that the agent checks the current status of a resource before confirming it, and to test that rule with the same scenario.

Memory snapshots can contain personal data about customers or staff. Limit who can read the postmortem, store it under the same access rules as the source records and remove details that the review does not need. A useful postmortem names the memory item by identifier and describes its content in general terms.

Find the control that should have stopped the run

Every postmortem should answer one question directly: which control was supposed to stop this, and why did it not? Controls in agent systems include approval gates for sensitive actions, permission scopes on each tool, step and cost limits, idempotency on writes, dead-letter queues for tasks that cannot finish and human escalation when confidence is low. Check each one against the run.

  • Was the action in a category that requires approval, and did the approval actually happen before the write?
  • Did the agent's credentials allow the write it made, and should those credentials have allowed it?
  • Did a step limit or cost limit exist for this task, and was the threshold set at a level that would have caught this run?
  • Was the write idempotent, or did a retry repeat the action?
  • Did an alert fire, and did the person who received it have the authority to pause the agent?
  • Could a person pause the agent without losing its queue, memory or pending work?

If no control existed, name the missing control as the root cause. If a control existed but was set too loosely, name the threshold. If it fired but nobody acted, name the handoff. Each of these leads to a different fix, and mixing them up is one of the most common reasons repeat incidents happen.

Retries deserve special attention. Stripe documents provider-defined idempotency keys as a way to make repeated requests safe, and the same principle applies to any write an agent performs. Without it, a timeout followed by a retry can duplicate a booking or a message. The pattern is explained in idempotent tool calls for agents.

Turn findings into changes that hold

Corrective actions should be testable. Phrases such as be more careful or review more often cannot be verified and usually disappear from the next review. Better actions are specific: add a required status check before confirming a resource, move a category of write behind an approval step, lower a spend threshold for one task, add the failing scenario to the test set or route a specific case to a person instead of letting the agent decide.

Each action needs an owner and a verification method. Before closing the incident, replay the failing run against the change, or the closest safe copy of it, and confirm the outcome is different. Record that result in the postmortem. When a fix involves handing a decision to a person, make the handoff itself precise: send the decision and the evidence needed to make it, not the whole conversation. The pattern is described in human escalation for AI agent failures.

In Be Human's managed operations, postmortems are part of the routine rather than something written after a bad week. Each agent is set up with a defined role, a memory scope, a list of tools, approval boundaries and an observable success criterion, so the reviewer already knows what the agent was allowed to do. Monitoring keeps the run records that reviews depend on, incident response follows a written sequence, and permission reviews check whether the controls that should stop a run still match current business rules. Changes to agent behavior are made in controlled steps and tested against the cases from the review.

Keep the review blameless, short and useful

Write about the system, not about the person who configured it or the reviewer who missed the alert. Use the agent's name and the control's name, and describe decisions by what was known at the time. Blame tends to produce vague reports, and vague reports produce vague fixes. A short review with specific evidence is more useful than a long one that avoids the central question.

After the review, store the postmortem where the team that operates the agent will read it, and tag it by failure type so patterns become visible. The NIST AI Risk Management Framework describes managing AI risk across the lifecycle, including monitoring and response, and it offers a shared vocabulary that makes these reviews easier to compare over time. A single postmortem fixes one run. A collection of them shows where the controls are weakest. If you want to find the first task worth handing to an agent, take the free Be Human AI diagnostic.

Frequently asked questions

Who should write the postmortem for an agent incident? The owner of the agent's role should draft it, because that person understands the expected behavior and the business rules. Engineers or operators who can read the logs should contribute the tool call and memory sections. Someone outside the immediate workflow should review the draft for blame and missing controls.

Do we need a postmortem if the agent caught and corrected its own error? Yes, if the error reached an external system, a customer or a record that others depend on. The self-correction is evidence that a control worked, but the review should still ask why the first action happened and whether the correction could have failed. A short note is enough when nothing left the internal workspace.

How do we handle personal data inside memory snapshots during a review? Restrict access to the people who need the evidence, keep the postmortem in the same access-controlled place as the source records and describe memory items by identifier rather than by full content. Remove any personal details the findings do not depend on. Your data protection obligations apply to the review itself, not only to the agent's production use.

How do we know that a corrective action actually worked? Replay the failing scenario against the change, or a safe copy of it, and record the new outcome in the postmortem. Then watch the next comparable runs for the same failure pattern over a reasonable period. If the failure returns, reopen the incident rather than writing a new one.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →