How to Report an AI Agent Problem So It Gets Fixed Fast

To report an AI agent problem so it gets fixed, send one message with four parts: the task you gave the agent, the result you expected, what actually happened, and the context around it (where, when, which account or channel). Attach the exact output and say how much it affects your work. A report built this way lets the person running the agent reproduce the problem without a follow-up conversation.

Key takeaways

• Describe the task, the expected result, the actual result and the context. Those four parts answer most of the questions a fixer would otherwise ask.
• Keep the exact output. Paraphrasing an agent's answer removes the detail that usually explains the fault.
• Say what you expected and why. Without it, nobody can tell whether the agent made a mistake or followed an unclear instruction.
• Treat an unwanted action or exposed information differently from a wrong answer: report it at once and ask for a pause if it could continue.
• Use one agreed reporting channel and one template, so reports arrive complete and nothing depends on who happened to hear about the problem.

Why do vague AI agent reports take so long to fix?

An AI agent rarely fails the way a printer does. A printer jams in the same place each time. An agent can answer correctly for one person and wrongly for another, depending on how the request was worded, which documents it could reach, which tool it called and which instructions it was following that day. A report that says the agent is wrong again gives the person fixing it almost nothing to work with.

Most of the first hour of a fix goes to questions. What exactly did you ask? What did you expect to see? Which customer or document was involved? Was this the first time? Each question costs a round trip, and by the time the answer arrives the employee has moved on, the conversation has scrolled away and the details are half remembered.

The purpose of a good report is to remove those round trips. It does not need to diagnose the cause. Diagnosis is the job of whoever operates the agent. Your job is to describe what you saw so clearly that someone else can see it too.

What should an AI agent problem report contain?

Use five short parts. The first four carry the facts. The fifth tells the fixer how urgent the matter is.

• Task: what you asked the agent to do, copied word for word if you can.
• Expected: what a correct result would have looked like, and why you expected it (a policy, a past result, a document you checked).
• Actual: what the agent said or did, with the exact output attached.
• Context: where and when it happened, which channel or account you used, which document, customer record or file was involved, and whether it has happened before.
• Impact: what the problem affected, whether anyone outside your team saw it, and whether you can keep working.

The expected result is the part people skip, and it is the most useful one. Agents are often right about a different question than the one you meant. If you write that you expected the latest version of a document and the agent used an older one, the fixer knows to look at what the agent can read and how it chooses between versions. If you only write that the answer was wrong, they have to guess.

Context matters because an agent's behavior depends on its surroundings. The same request can produce different results in a chat window and in an email channel, or for two users with different access. Naming the channel, the account and the time lets the operator find the matching record in the agent's logs. Observability practice in general rests on this idea: when a system records what it did and in what sequence, a reported symptom can be traced to a cause (the [OpenTelemetry observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) explains the concepts). Your report is the pointer that tells the operator which record to open.

What does a good report look like in practice?

Here is an illustrative scenario. A sales operations coordinator asks the company's agent to prepare a renewal summary for a customer account. The summary lists a contract term that was replaced by an amendment. The coordinator notices before sending it on.

A weak report reads like this:

The renewal summary is wrong, please fix.

The operator now has to ask which customer, which summary, what was wrong, and what the right term should have been. A strong report for the same event looks like this:

• Task: I asked the agent in the sales channel to prepare a renewal summary for one named customer account, using the contract on file.
• Expected: The summary should show the notice period from the amendment, because the amendment replaced the original clause.
• Actual: The summary shows the notice period from the original contract. The agent did not mention the amendment. Output attached.
• Context: Sales channel, my own account, this morning. Both documents are in the customer folder. I have not seen this with other accounts, but I have only checked this one.
• Impact: Caught before sending. I can continue by checking terms manually, but I would not trust the summaries for other accounts with amendments until this is looked at.

This report is short, and it already points toward a likely cause: the agent found the original document and either could not see or did not prioritize the amendment. The operator can open the same folder, repeat the request and check what the agent could read. No meeting is needed.

How do you describe different kinds of problems?

Not every problem deserves the same handling. Use this guide to decide how to word the report and how fast to send it.

• The answer was wrong or incomplete: send the standard report. Include the source you checked against, since that is how the fixer knows which of you is right.
• The agent did not do something it should have: say what you expected it to do, what it did instead (including nothing), and whether it showed an error or just stopped.
• The agent did something it should not have done, such as sending a message, changing a record or booking something: report immediately. Say what changed, where, and whether it could happen again. Ask for a pause if it could.
• The agent showed information to someone who should not have seen it, or appeared to follow instructions hidden inside a document or message: treat it as security-sensitive. Report it straight away through the agreed channel, and do not forward the content more widely than necessary. These categories appear in the [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) as sensitive information disclosure and prompt injection.
• The agent was slow, unavailable or inconsistent: note the time, what you were trying to do and whether retrying helped.

When in doubt, report. A report that turns out to be a misunderstanding costs a few minutes. A real problem that nobody mentioned because it seemed minor can quietly repeat across a team.

What should you do between noticing the problem and sending the report?

The moments right after a failure are when evidence is easiest to lose. A short routine helps.

• Stop and keep the evidence. Do not delete the conversation, retry repeatedly or overwrite the file the agent touched.
• Capture the exact output. Copy the text or take a screenshot that shows the request and the response together.
• Write down the details while they are fresh: the time, the channel, the record or document involved.
• Write the five parts in plain language. Two or three sentences each is enough.
• Send the report through the one agreed channel, not through a side conversation with a colleague.
• State whether work should pause. If the agent is still acting on something wrong, say so in the first line.

A quick check before you send:

• Could someone who was not there repeat the request from what I wrote?
• Did I say what I expected, not only what went wrong?
• Is the exact output attached?
• Did I name the channel, the account and the record involved?
• Is it clear whether the matter is urgent?

Try not to include more than you need. Do not paste sensitive customer details into a report if a record name or reference is enough. The operator can look the record up with their own access.

What happens after you send the report?

A well-run process follows a predictable path. The operator confirms receipt, tries to reproduce the problem, and works out which part of the setup is responsible. The cause usually sits in one of a few places: the instructions the agent follows, the knowledge or memory it can draw on, a tool it calls, a permission it has or lacks, or the underlying model's behavior. Knowing which one decides the fix.

If the cause is an instruction or a tool, the fix is a change to the agent, and a change to a live agent should be approved and tested rather than improvised. The article on how to request and approve changes to a managed AI agent describes that step. If the report points to access that is too broad or too narrow, it feeds the permission review.

The best operators also turn a confirmed problem into a saved test case: the same task, the same context and the known correct result. After a fix, they run it again to confirm the problem is gone and keep it so a later change does not bring it back. This is the basic idea behind evaluations of agents, which [Anthropic's engineering guidance on evals](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) discusses. A report with a clear task and a clear expected result converts into such a test almost directly. A vague report does not.

Finally, the reporter should hear back. Even a short note saying what was found and what changed tells the employee that reporting works, which is what keeps the next report coming.

How can a company make reporting easy enough that people do it?

People report problems when it is quick, when it is clear where to send them, and when something visibly happens afterwards. Three habits make the difference.

• One channel: a single place or address for agent problems, named in the same message that introduces the agent to the team.
• One template: the five parts above, pinned where people will see them, so a report takes a couple of minutes.
• One acknowledgment rule: every report gets a reply saying who has it and what happens next, even if the fix will take time.

These habits belong with the rest of adoption. Employees who know what the agent is for, what it should not do and how to flag a fault are far more likely to keep using it. The guidance on getting your team to actually use an AI agent covers that wider picture, and the piece on collecting employee feedback and acting on it covers the review of reports as a pattern rather than one by one.

How Be Human handles this: in a managed engagement, reporting is agreed at the start alongside the agent's role, tools and approval boundaries. Employees get one channel and a short template. The operations team logs each report, reproduces it using the agent's records, identifies the layer responsible, proposes a change for the owner to approve, and checks the result against the original case. Recurring reports are reviewed periodically for patterns, and anything involving exposure or unwanted actions is handled as an incident rather than a routine request.

Frequently asked questions

Do I need technical knowledge to report an AI agent problem? No. You only need to describe what you asked, what you expected and what happened, in ordinary language. The operator handles the technical diagnosis. Exact output and a clear statement of the context are more valuable than any technical explanation.

Should I report every small mistake the agent makes? Report anything you would correct by hand more than once, anything that could reach a customer, and anything involving access, data or actions. One-off wording preferences can go into regular feedback rather than a formal report. If you are unsure which it is, send the report and let the operator decide how to treat it.

What if I cannot repeat the problem? Report it anyway, and say that you could not repeat it. Include the time, the channel and the exact output from when it happened. Agents can behave differently from one request to the next, and the operator can often find the original event in the agent's records even when it does not recur on demand.

When should I ask for the agent to be paused? Ask for a pause when the agent is taking actions that are wrong, when it may have exposed information to the wrong person, or when you cannot safely keep working with its output. Say so in the first line of the report. Stopping a pilot or a workflow is a normal control, and the operator can resume it once the cause is understood.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.