Shadow Mode for AI Agents: Watch Before You Grant Autonomy

Shadow mode is a rollout stage in which an AI agent works through the same tasks as your staff, sees the same inputs, and writes down the actions it would take without executing any of them. You compare those proposals with what people actually did, then use the differences to decide which action types can move to supervised execution and which should stay with a person. Done well, it turns autonomy into a measured decision instead of a hope.

Key takeaways

• Shadow mode lets an agent propose actions on real work while nothing reaches customers, suppliers or systems of record.
• Log every proposal with its inputs, the tools it would have called and the reason it gave, so each gap can be reviewed later.
• Score agreement by action type rather than by overall impression, and separate harmless differences from wrong decisions.
• Grant autonomy one action class at a time, with a named owner, a rollback path and a scheduled review.
• Keep a sampled shadow run going after go-live, so changes in the work or in the agent's behavior are caught early.

What shadow mode means for an AI agent

In shadow mode, the agent receives the requests, documents or messages that a staff member receives, works through the task, and records what it would have done: the reply it would draft, the field it would update, the approval it would route. None of those actions leave the agent. Its tool permissions are restricted during this stage, so a proposed update cannot reach the CRM, a supplier or a customer by accident. The restriction belongs in the tool layer, not only in the instructions, because instructions can be ignored or misread.

Shadow mode is easy to confuse with two neighboring ideas. A sandbox tests the agent against invented or copied data, which is useful for early engineering but says little about how real requests arrive, in what order, and with what missing details. A pilot usually gives the agent live actions on a small slice of work, which carries real consequences from the first day. Shadow mode sits between them: real inputs, real timing, no live effect. Its value comes from comparing what the agent proposed with what the person did on the same work, and that comparison only exists when both happen on the same case.

Scope one task and its action boundaries

Shadow mode fails quietly when the task is too broad. An instruction such as handle customer email contains dozens of decisions with very different risks. Start with one role and one recurring task, then list the actions that task could involve and group them by consequence: reading and summarizing, drafting, changing a record, sending something outside the company, and committing money or dates. Early rollouts usually need to test only the first three groups. The last two should stay human-owned until the evidence is strong and the permissions are narrow.

Write one success criterion that you can check without opinion. The proposed routing matches the approver who finally signed off is observable. The agent is helpful is not. If you cannot verify the criterion from the logs, the rollout will end in debate rather than in a decision. A role-level description of this kind is covered in AI Employee Onboarding, which explains how an agent's responsibilities sit beside those of the people already doing the work.

A step-by-step plan for running shadow mode

The plan below is deliberately plain. Most failed rollouts were not technically hard. They lacked an owner, a log, or a clear rule for what happens after the evidence arrives.

• Name an operations owner who decides whether a gap is acceptable. This should be the person accountable for the outcome, not the engineer who built the agent.
• Connect the agent to the same inputs the staff member sees, and remove write access at the tool layer for the duration of the run.
• Create the proposal record before the human acts. If the agent can read the human's final answer, it cannot be fairly graded.
• Run in parallel for at least one full cycle of the task, including busy periods and unusual cases, not only the easy ones.
• Review gaps at a fixed interval with the same reviewers each time, and record a decision for every gap that matters.
• Decide per action class whether to keep observing, grant supervised execution, or retire the proposal entirely.

Treat the end of the run as a decision point with a written outcome. If the evidence is mixed, the outcome may be to narrow the task, add context, or keep the agent as a drafting assistant. Any of those is a valid result. Continuing indefinitely without a decision is not.

How to log proposals so the gap is useful

A shadow log is only as useful as its structure. Free-text notes make it hard to tell whether the agent made a systematic error or a one-off mistake. Each record should include enough detail to reconstruct the decision later:

• A reference to the input, such as the message identifier or document version, so the case can be found again.
• The exact proposed action, including its target record, recipient, or approver.
• The tools the agent would have called and the parameters it would have sent.
• The rule, policy or source the agent cited for its choice.
• The human action taken afterward, linked to the same case reference.
• The timestamp of each step, so you can see whether the agent proposed before or after the person acted.

Observability practice helps here. Tracing concepts, where each step of a process is recorded with its context and linked to the steps before and after, let you follow a decision from input to proposal to outcome. You do not need a specialized platform to apply the idea. You need consistent identifiers and a rule that every case leaves a trace. Keep only the personal data the review requires, and apply the same retention limits you would apply to the original records.

Illustrative scenario and how to read the gap

Illustrative scenario: a small services company runs its shared inbox with a team of coordinators. During shadow mode, the agent categorizes each incoming message and drafts a reply, but the coordinators keep handling every message as usual. In the review, routing to the billing queue matches the coordinators almost every time. The drafts diverge in a different way: the agent promises refund timing that the company does not actually guarantee. The team does not grant drafting autonomy yet. They fix the source of the timing rule, rerun the same cases, and only then move routing to supervised execution. The workflow pattern is described in AI Shared Inbox Management.

When reviewing a gap, classify it before deciding anything. A decision guide that works for most rollouts looks like this:

• The proposal matches the person's action and the difference is cosmetic, such as a greeting or a formatting choice. Keep the case as evidence of agreement.
• The proposal differs and the person was wrong for a documented reason. Correct the rule or the reference the agent uses, then rerun the case.
• The proposal differs because the agent lacked context that the person had. Add that context through an approved source, rather than hoping the agent infers it.
• The proposal would have caused external harm if executed. Stop promotion for that action class and investigate the cause before any further runs.
• The proposal was right where the person was slow or inconsistent. Record it as a candidate for supervised execution, with a human still approving at first.

When to grant more autonomy

Granting autonomy should be a sequence rather than a switch. For each action class, move through stages: supervised execution, where a person approves each action before it runs; exception-based execution, where the person reviews only flagged cases; and bounded autonomous execution, with limits on amount, recipient, record type or time. Each stage needs a named owner, a written definition of the limits, and a way to revert quickly. The permissions themselves should follow least privilege, granting the agent only the scopes the action requires, as described in [OWASP's authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html).

Shadow logs also reveal risks that are easy to miss. Incoming messages sometimes contain text that tries to instruct the agent, such as an email asking it to ignore its rules. Treat every input as data to be evaluated, not as a command, and flag any case where the agent's proposal changed because of unusual instructions inside the content. Prompt injection concerns are covered in [OWASP's prompt injection guidance](https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html), and the pattern should be part of your gap review.

How Be Human handles this: in a managed engagement, Be Human defines the single role, the action classes and the success criterion first, then runs the agent in shadow mode with logs the operations owner can read without engineering help. Monitoring, recurring evaluations and permission reviews continue after any action class is promoted, so the boundary set during the rollout is still checked later. Changes to permissions, memory or approval rules go through a recorded review rather than a quiet edit.

If you are not yet sure which task deserves a shadow run first, the free Be Human AI diagnostic is designed to help you find the first task worth handing to an agent.

Frequently asked questions

How long should shadow mode run before anyone grants autonomy? The length depends on how often the task occurs and how varied its cases are, so a rare task needs a longer window than a daily one. The run should cover normal days, busy periods and the unusual cases that cause most trouble. Decide the end condition in advance, so the rollout does not drift on without a verdict.

Does shadow mode expose customer data to the agent? Yes, the agent sees the same inputs the staff member sees, so it needs the same access rules, retention limits and purpose restrictions. Check the obligations that apply to your data, and for Brazilian operations review the guidance of the national data protection authority, ANPD. Minimizing what the agent receives is often the simplest way to reduce this exposure.

Will staff change how they work because the agent is watching? They can, and it is worth saying openly that the agent is a measurement tool, not a performance review. If people start rushing or over-documenting, the gap will look better or worse than reality. Keep the review focused on the action classes and the rules, not on individual speed, and let the same reviewers see the cases consistently.

Is shadow mode useful for an agent that only answers questions? It is useful, and the comparison works in the same way. Instead of comparing actions, you compare answers against responses the team already considers correct, and you note where the agent cites sources it should not use. The approach is close to recurring evaluations, which the [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) treats as part of ongoing risk management rather than a one-time test.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.