Replay Logs for AI Agents: Log Fields to Reproduce a Bad Run
Which fields a harness must log so a failed AI agent run can be replayed step by step in a test environment, with a practical checklist.

To replay an AI agent run, the harness must record, for every step, the context the model actually received, the model and settings it used, each tool call with its arguments and returned result, and the versions of prompts, tools and permissions in force at that moment. A test environment can then feed those recorded inputs back through the same steps, with side effects stubbed, and show where the run departed from what was expected. If the logs hold only the final answer, the bad run can be described but not reproduced.
Key takeaways
- A replay needs the context each step actually saw, not the context you intended to send.
- Log the model identity, its settings, and the versions of prompts, skills, tool schemas and permissions for every step.
- Record tool arguments and full returned results, with redaction rules agreed before the first run.
- Store a run identifier, a sequence number and a parent link so steps can be ordered and branches traced.
- Replay in a sandbox with live writes stubbed, then compare the new behavior against the original record.
Why a bad run cannot be reproduced from its final answer
An agent run is a chain of decisions. Each model call depends on everything earlier in the context: the instructions, retrieved documents, previous tool results and the model's own prior output. A tool may also change state, so the same request can return a different result on a second attempt. A final message saying the order was wrong tells you the outcome, but not which step introduced the error.
Replay does not mean asking the model to produce the same words again. Models do not guarantee identical output, and a provider may have changed behavior since the run. What you need is the ability to re-run the harness with the recorded inputs at each step, so you can see whether the failing decision was caused by the context, the tool, the permission setup or the model. When the model is rerun at a single step, that rerun is a test, and its result should be stored as its own record.
The fields each step record needs
Think of the log as a sequence of step records under one run identifier. Each record should contain at least the following fields.
- Run identifier, step identifier, parent step identifier and sequence number, so steps can be ordered and branches traced.
- Start and end timestamps in UTC for each model call and each tool call.
- Harness version, prompt version, and the identifiers of every skill file and tool schema loaded for that step.
- Model name, provider version string, and sampling settings such as temperature and maximum output length.
- The full input context sent to the model, or a stored copy of each context block with a content hash that shows it was not altered.
- The raw model output, including the stop reason and any tool call requests exactly as emitted.
- For each tool call: the tool name, the validated arguments, the permission decision, and the approver's identifier if a human gate was involved.
- For each tool result: the response as received, the version passed to the model after any truncation, and a flag showing what was removed.
- Status, error class, retry count, and the rule that selected the next step.
- Token counts and cost for the step, so a budget breach can be explained as well as stopped.
Keep the raw result and the truncated result as two separate fields. Teams often truncate to protect the context window, then discover that the field which explained the failure was the one cut. Storing both lets you replay what the model saw and audit what the harness held back.
Some of this data is personal or confidential. Log references to secrets rather than the secrets themselves, mask customer identifiers at capture where the workflow allows it, set retention periods before the first run, and limit who can read full context blocks. Under Brazilian and European data protection rules, a log that contains identifiable people is still personal data, so its retention belongs in the same review as the rest of your data handling.
A replay plan you can follow
When a run goes wrong, work through the record in this order. The aim is to reproduce the decision in isolation before changing anything live.
- Freeze the failing run by copying its records into a read-only store and noting who requested the investigation.
- Restore the harness, prompt, skill and tool schema versions named in the first step record, not the versions currently deployed.
- Load the recorded permissions, then replace every tool that writes to a live system with a stub that returns the recorded response.
- Feed each recorded tool result back in sequence, and compare the arguments the model produces against the originals.
- Stop at the first divergence and write down what differed: the model output, the arguments, the retrieved context or the harness decision.
- Rerun the model at that single step several times to find out whether the failure is stable or intermittent.
- Write the cause as a specific change to a prompt, tool description, permission, budget or stop condition, then replay the same run against that change before any release.
The test environment should be separate from production and have no route to live credentials. The guidance in Sandbox for AI Agents: Test Changes Before They Reach Live Systems explains that separation in more detail.
An illustrative scenario: the duplicate order
Illustrative scenario: a small operations team runs an agent that updates supplier orders from email confirmations. One afternoon the order system responds slowly, a tool call times out, and the agent sends the order again. The team has only a final message reporting two orders, with no explanation of how the second one happened.
The step records tell a more useful story. The first tool call returned a timeout, but the harness stored only the status code, not the error class. The truncation rule kept the first lines of the response, which contained no error field. The retry logic therefore treated the call as failed without knowing it had partly succeeded, and it issued a new write without an idempotency key. Replaying the run with the stubbed tool showed the divergence at the retry step, not at the model call.
The fixes were specific. The truncation rule now always keeps the error block, every write stores its idempotency key, and any order creation that follows a timeout passes through a human approval gate. Each change was tested by replaying the same run before it reached the live system. For the truncation rule, Tool Result Truncation for AI Agents: Keep the Fields That Decide covers which fields to protect.
How Be Human handles replay in managed operations
Be Human treats the log schema as part of the first engagement, not as an afterthought. When a useful role is defined, the memory it reads, the tools it calls, its approval boundaries and its success criterion determine which fields must be recorded, so the logs describe the decisions the agent actually makes. In managed operations, the same records support monitoring, incident response and recurring evaluations. Retention rules are applied, permission changes are reviewed against the logged history, and past failures are replayed before any controlled improvement reaches the live harness. An incident review then starts from recorded evidence rather than from memory of what the agent was asked to do.
Mistakes that make a bad run impossible to replay
Most replay failures come from a small set of habits. Logging the final answer but not the steps is the most common. Storing the prompt template without the rendered prompt is another, because the template does not show the retrieved documents the model read. Leaving versions out of the record makes it hard to tell whether a change in behavior came from a new prompt, a new tool description or a model update. Some teams also replay against live systems, which turns an investigation into a second incident. A harness that changes without release history cannot say which configuration a run used, a problem covered in AI Harness Versioning: Release Notes for Every Agent Change.
Where to start with replay logging
Start with the one workflow where a bad run would be hardest to explain. Add the step fields to that workflow first, run it in a test environment until a replay reproduces a real failure, and only then widen the schema to other workflows. If you are not sure which workflow deserves this attention first, take the free Be Human AI diagnostic to find the first task worth handing to an agent.
Frequently asked questions
Can a replay reproduce the exact output the model gave the first time? Not reliably. Model output can vary between calls, and providers update their models, so a replay should reproduce the recorded inputs and tool results rather than promise identical words. Rerunning the model at the failing step shows whether the behavior is stable, and that rerun should be stored as its own record.
Is it safe to store full prompts and tool results? Only with controls in place. Mask personal and confidential values where the workflow allows it, keep secrets out of logs entirely, limit who can read full context blocks, and set retention periods before the first run. Treat the log store as sensitive data, because it often holds more detail than the original system did.
How long should agent logs be kept? The right period depends on your legal obligations, your sector and your internal review cycle, so it should be decided with the people who own data protection and compliance. Write the period into the logging policy, apply it automatically, and document deletions. Shorter retention reduces exposure, but problems often surface after the fact, so set the period with that delay in mind.
Do I need replay tooling before the first agent goes live? You need the logging schema before the first run, because records cannot be recreated afterward. The replay tooling can start simpler, such as a script that feeds recorded results into a stubbed environment. Build fuller tooling as the number of workflows grows and the cost of an incident becomes clearer.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


