Shadow Mode for AI Agents: Test Proposals Before Any Write Access

Shadow mode is a test in which an AI agent receives the same work as your staff, writes down the action it would take on each case, and leaves every live system untouched. You then compare each proposal with the decision the person actually made. Write access is granted only after the gap is small, understood and narrow enough to put in writing.

Key takeaways

• Shadow mode lets an agent propose actions on real work while people keep every write permission.
• Pick one repeated decision with a clear owner and a visible outcome before you set anything up.
• Score agreement with written rules, and separate harmless variation from proposals that would have caused harm.
• Grant write access one action at a time, and only after the gap stays low for the whole test window.
• Keep proposals, human decisions and reasons in one audit trail from the first day.

What shadow mode is and what it is not

Shadow mode is an operating pattern, not a product feature. The agent reads the inputs a staff member sees, such as the request, the order record or the attached document. It produces a structured proposal, for example "route to billing with this summary" or "issue a partial refund on the disputed line," and stores that proposal in a log. A person performs the real action. Nothing is sent to a customer, nothing is posted to a ledger, and the agent changes no record.

Shadow mode is also different from a sandbox demo. A demo shows how the agent behaves on cases someone chose, usually clean ones. A shadow period shows how the agent behaves on the cases that actually arrive, including duplicate requests, missing fields and notes written in a hurry. That mess is the reason to run the test at all.

Shadow mode does not remove risk. An agent that reads live data still handles personal and commercial information, so access scope, retention and handling rules apply from the first day. Read permissions should cover only the fields staff use for that decision. Authorization guidance such as OWASP's cheat sheet on authorization is a useful reference when you define those scopes, because the principle is the same whether the reader is a person or a process: grant the least access the task needs.

Pick one decision the agent will shadow

Choose a decision that happens often enough to produce a meaningful sample within your test window, has an observable outcome, and has one accountable owner. Avoid decisions that depend on judgment nobody has written down. Also avoid decisions where one wrong action is severe and hard to reverse, at least until the team has seen the agent's behavior for a while.

Good first candidates are routine and bounded: classifying inbound requests, drafting the next step for a standard case, or checking whether a submitted document is complete before a human review. Weaker candidates include open negotiations, exceptions to policy, and anything that touches money in amounts your team has not formally discussed.

Before you configure anything, write one sentence that names the decision, the input the agent may read, the output it may propose and the person who owns the final call. If that sentence is hard to write, the task is not ready for an agent, and the shadow period will mostly measure confusion.

A step-by-step plan for the shadow period

The test works when the setup is boring and the rules are fixed before the first case arrives. The following sequence keeps the comparison honest.

• Write a decision card with the decision, the readable inputs, the proposal format and the owner of the final call.
• Freeze the policy version staff are following, so that people and the agent are judged against the same rules.
• Give the agent read-only access to the exact fields used for the decision, and log every read.
• Require proposals in structured fields: action type, parameters, a one-sentence reason that cites the input, and a confidence note if your system provides one.
• Run the agent on live cases in parallel with staff. Its output is stored and never executed.
• Record the human decision for each case, with a reason whenever it differs from the proposal.
• Do not ask staff to change their behavior to match the agent. The gap is the information you are collecting.
• Review disagreements weekly with the owner, and tag each one using the categories described below.
• Fix the window length in advance, and do not extend it because the results look almost right. Extending a test after seeing the numbers biases the outcome.
• Close the window with a written decision: keep shadowing, narrow the scope, or propose one specific write permission.

How to score agreement without fooling yourself

A single match rate hides the cases that matter most. Two proposals can differ in wording and still produce the same business result, while a proposal that looks close can route a sensitive case to the wrong team. Use categories instead of one percentage, and write down the rule for each category before the review begins.

• Agreement: the same action with the same parameters.
• Acceptable variation: different wording or step order, same business result, confirmed by the owner.
• Wrong proposal: a different action that would have produced a worse outcome. Record the reason.
• Unsafe proposal: an action that would have crossed a policy boundary, exposed data, or reached outside the agent's scope. Any single unsafe proposal pauses the test for review.
• Unclear: the human decision itself was debatable. Track these separately, because they usually point to a policy gap.

If you use another model to grade agreement, treat that grader as a component that needs its own check. Have the owner grade a sample by hand first, then compare. Anthropic's writing on evals for agents describes the same discipline: define what success means before you read the scores. A shadow period without written categories tends to end with a feeling rather than a decision.

Illustrative scenario: a request triage queue

Illustrative scenario, not a client case: a shared inbox receives questions about invoices. Staff tag each message and forward it to billing, sales or the account manager. The agent is shadowed on the same messages for a fixed window and proposes a destination for each one.

In the first review the agent matches staff on most messages, which looks promising. The disagreements, however, form a pattern. Messages that mention a payment plan go to billing in the agent's proposals but to the account manager in the staff decisions, because the team knows that payment plans for certain accounts are negotiated by the account manager. The owner records this as a routing rule, the policy is updated, and the next review shows the disagreement shrinking. The category had been hiding in the match rate until someone read the reasons.

Only after that rule holds across the window does anyone consider a write permission, and then only for routing, with a log entry for each move and a single switch that returns the inbox to manual handling.

When to grant write access, and how Be Human handles it

The end of a shadow period should produce one of four decisions, and the choice should be written down with its evidence. Use the following guide as a checklist, not a score.

• Keep shadowing if the disagreements cluster in a category you have not yet explained.
• Narrow the scope if the agent is reliable on routine cases but consistently wrong on a sub-type you can name and exclude.
• Propose one write permission if unsafe proposals are zero, the unclear cases have been resolved by policy changes, and the owner accepts the rollback path.
• Do not grant broad access because the test went well. Grant the single action you actually tested, with the same inputs and boundaries.

A write permission is only ready when someone can disable it in one step, knows who is allowed to do so, and can see from the logs what the agent did and why. Each action should also be recorded with enough detail to reconstruct the decision later. The fields that make this possible are covered in the guide on audit trail fields for agent actions.

Be Human designs and operates this kind of setup as a managed service. A first engagement defines the single role, the memory and tools it needs, the approval boundaries and the observable success criterion, so the shadow period tests the same boundaries the agent will operate under later. Managed operations then keep the logs, run recurring evaluations, review permissions as people and systems change, and handle incidents if a proposal would have caused harm. The service is remote and available in English and Portuguese.

If you are not yet sure which decision in your team deserves the first shadow test, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Does shadow mode need a separate test environment? It does not have to. The agent reads live inputs with read-only access, and its proposals are stored rather than executed, which is the core of the pattern. A separate environment is useful for testing tool calls, but it does not replace a shadow period on real traffic, because sandbox data rarely matches the mess of production cases.

How long should the shadow period last? Set the length before the test starts, based on how often the chosen decision occurs and how much variety you need to see. Keep the window fixed even if early results look strong or weak. Changing the length after you have seen the results makes the outcome hard to trust.

Can the agent read live customer data during shadow mode? It can, but only within a scope you define and document. Limit read access to the fields the decision requires, keep logs of what was read, and apply the same retention and handling rules you use for staff. If the data is sensitive, review the restrictions with whoever is responsible for privacy before the first case is processed.

What if staff start following the agent's proposals during the test? That undermines the comparison, so the rule should be stated plainly: people keep working as they normally would, and the agent's output is only for review. If staff begin copying the proposals, you lose the ability to measure the agent against real judgment. Record the cases where this happens, and treat the test as contaminated for those cases.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.