Turn Staff Corrections Into Agent Rules: A Feedback Loop Method

A reliable AI agent improves when staff corrections are captured, reviewed and turned into written rules or memory entries that the agent reads next time. The method has four moves: record each correction with its context, sort it by cause, promote only approved changes, and test every change against past cases before it goes live. Done this way, one correction improves every future run instead of being repeated by the next person who spots the same mistake.

Key takeaways

• A correction is only useful once it is recorded with the input, the wrong output and the corrected output.
• Sort each correction by cause before changing anything, because a missing fact, a bad rule and a bad tool result need different fixes.
• Promote a correction into a rule or memory entry only after a named owner approves it.
• Test each promoted change against a fixed set of past cases before it reaches live work.
• Keep a change log so anyone can see what the agent learned, when, and who approved it.

Why staff corrections rarely make the agent better

Most corrections happen in the moment. A reviewer edits a draft, a team member replies differently to a customer, or someone pastes a note into a chat window to explain what went wrong. The fix works for that one task, then it sits in an inbox, a messaging thread or the memory of the person who made it. The next week a different colleague meets the same error, corrects it again, and the agent never learns anything durable.

There is a second problem. A correction is not automatically a rule. Some corrections describe a one-off exception, such as a customer who was promised a delivery date by a manager. Some reflect a policy that has since changed. Some contradict a correction made by someone else last month. If every correction is written straight into the agent's instructions, the loop does not improve the agent. It teaches the agent whatever the most recent person happened to say.

Governance frameworks such as the NIST AI Risk Management Framework point in the same direction: risk is managed across the whole life of a system, not only at launch. A correction loop is one practical place where that idea becomes a daily operating habit.

Capture each correction in a record you can reuse

A correction needs enough context that someone who was not in the room can judge it later. A short record is usually enough. It should contain:

• The task and the input the agent received, with personal and confidential data removed or replaced by a placeholder.
• The output the agent produced, quoted exactly rather than paraphrased.
• The corrected output, or the specific sentence or value that changed.
• The reason for the change, written in one sentence by the person who made it.
• The name of the person who made the correction and the date it was recorded.

Quoting the exact output matters. A paraphrase hides the detail that caused the error, and that detail is what the rule must address. If your team already keeps run logs, these fields should map onto them. For the signals worth keeping, see what to log for AI agent observability.

Sort each correction by cause before you change anything

Many corrections are fixed in the wrong place. A staff member changes a price in the reply, so someone edits the prompt, when the real problem was that the agent read an outdated price list. Ask one question first: what kind of gap produced this error? Use this guide:

• If the agent lacked a fact, such as a current policy, a branch's opening hours or a replacement part for a product, the fix is a memory entry with a source, an owner and a last verified date.
• If the agent followed a rule that is wrong, incomplete or unclear, the fix is a rule change.
• If a tool returned stale, partial or malformed data, or the call timed out, the fix belongs to the integration or the tool definition, not the prompt.
• If the case needed a judgment the business does not want automated, the fix is an approval boundary that routes that kind of case to a person.
• If the correction was a one-off exception, record it as an exception note and do not generalize it.

Use the same categories across the whole team. A shared vocabulary for failures makes trends visible, and it stops every correction from looking like a unique event. For a fuller set of categories, see AI agent failure modes for business agents.

How do you promote a correction into a reviewed rule?

Promotion is a deliberate step with a named owner. The sequence below works for most business agents and can run weekly without slowing daily work. Apply it to every candidate change:

• Group the recorded corrections each week by the rule, fact or tool they touch.
• Wait for a pattern before drafting a rule. Two or more independent corrections pointing to the same gap is a reasonable threshold for routine issues. A single error that caused real harm goes to review immediately.
• Draft the change in plain language: the situation, the required behavior, the exception, and the source of any fact.
• Ask the accountable owner of that process to approve, change or reject the draft. The owner is the person who answers for the outcome, not necessarily the person who noticed the error.
• Replay the original case and a set of similar past cases against the candidate change, using a fixed test set rather than memory of what used to work. Building that set is covered in how to build a golden test set for an AI workflow.
• Publish the change with a version note, then check the next sampled outputs for regressions in the areas it touches.

In Be Human's managed model, this loop is part of operations rather than a side task. The first engagement defines the agent's role, the memory it may read and write, and the approval boundary for each category of change. Managed operations then review corrections on a set cadence, keep the change log and test sets current, and route changes that affect permissions or customer commitments to the named owner before publication. The client keeps control of what the agent is allowed to learn.

Write rules an agent can follow and a reviewer can check

A rule that says "be professional and accurate" cannot be tested, and an agent cannot reliably follow it. Write rules as conditions and actions. Name the situation, the behavior required, what to do when the situation does not match, and where the supporting fact lives. For example:

When a customer asks for a refund on an order placed outside the return window stated in the policy memory entry, do not confirm a refund. Create a handoff to the support lead with the order number, the purchase date and the customer's message.

A rule like this can be reviewed in seconds. A reviewer can see what triggers it, what the agent should do, and what would count as a violation. Avoid stacking rules that conflict. When a new rule touches an existing one, read both together and state which takes precedence. Keep credentials, personal data and confidential figures out of rule text. Rules describe behavior, and facts belong in memory with their sources.

Keep memory entries sourced, owned and due for review

Memory entries record facts about the business: current policy, product details, team responsibilities and approved wording. Each entry should carry three things: the source it came from, the person who owns it, and the date it was last verified. Without an owner and a verification date, a memory entry quietly becomes a rumor the agent repeats with confidence.

Review memory on a schedule, not only after errors. Entries about prices, schedules and policies age fastest. Remove an entry when its source is retired, and do not keep an old fact beside its replacement, because the agent may read both and blend them. Stale facts are one of the slower failure modes to notice, which is why scheduled checks matter. For how to watch for gradual decline, see how to spot quality decline in AI agents over time.

Anthropic's engineering guidance on context engineering treats an agent's context as a limited resource that should hold only what is useful for the current task. The same discipline applies to memory. A smaller set of current, well-sourced entries is easier to trust and easier to review than a long archive of old corrections that nobody has checked.

An illustrative scenario: correcting a quote follow-up agent

Illustrative scenario: a small distribution company uses an agent to draft follow-up emails after sales quotes. A sales coordinator notices that a draft offers a discount that was withdrawn earlier. She corrects the draft and records the case with the quoted email, the corrected line and the reason: the discount is no longer offered.

The team sorts the case first. The agent lacked a current fact, so the fix is a memory entry, not a prompt edit. The sales operations lead, who owns pricing communications, approves a memory entry stating that the discount is withdrawn. The entry cites the current price policy document and carries the date it was verified. Before publishing, the team replays several earlier quote follow-ups that mentioned the discount. The corrected drafts omit it, and a draft that asks for an exception is routed to a person. The change is logged with its version and date.

In the following week, sampled follow-ups are checked for the same error, and none appear. The coordinator's correction is closed. The lesson is not about one discount. The correction was resolved in the place where the fact lives, so the next person who opens a draft does not need to fix it again.

If you are unsure which recent correction in your team should start this loop, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

How many corrections should trigger a new rule? There is no universal count. Two or more independent corrections pointing to the same gap is a reasonable starting threshold for routine issues, while a single error that caused real harm should go to review at once. Judge the pattern and its consequences, not the number of complaints.

Who should approve changes to an agent's rules? The accountable owner of the process the agent supports should approve, such as the sales lead for quote communications or the support lead for refunds. That person answers for the outcome, so approval should not sit with whoever happened to notice the error. Changes that touch permissions, data access or customer commitments also need review from whoever governs the system.

Should every correction go into the agent's memory? No. Memory is for facts about the business that stay true across tasks. One-off exceptions, personal preferences of a single employee and opinions about tone should be recorded but not stored as memory. Every memory entry that remains needs a source, an owner and a review date.

How do you know a rule change made things better? Replay the original failing case and a fixed set of similar past cases before publishing, then check sampled live outputs in the affected area afterwards. If the change fixes the case but breaks an unrelated behavior, roll it back and revise. A change log lets you connect a later drop in quality to the change that caused it.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.