Tool-Call Test Fixtures for AI Agents: Test Without Live Systems

To test AI agent tool calls without touching live systems, record the responses your tools return, store each one as a versioned fixture, and replay those fixtures whenever the agent is tested. The agent makes the same calls it would make in production, but every read and write goes to a stored copy instead of your live CRM, ledger, inbox or calendar. This article explains what to record, how to organize the fixture set, and how to check the calls the agent makes.

Key takeaways

• A tool-call fixture is a saved request and response pair for one tool, captured once from a real interaction and replayed in every test run.
• Test the agent's decisions and arguments, not only its final text, because a polite reply can hide a wrong write.
• Record fixtures from real responses, then remove personal data and keep the files readable for reviewers.
• Version fixtures alongside prompts and tool descriptions, so a change in one is visible next to the others.
• Keep write tools in replay-only mode during tests, and move an agent toward live actions only after its replay results and a supervised period agree.

What a tool-call fixture actually contains

An agent works through tools: a lookup in the order system, a draft in the email client, a ticket update, a calendar hold. Each tool call has two sides that matter for testing. The first is the request the agent sends, including the tool name, the arguments and any identifiers. The second is the response the tool returns, which the agent then reads and reasons over. A fixture stores both sides for one interaction, keyed by a stable description of the request, so a test can run the agent, intercept each call, and answer it from the file instead of calling the live system.

This matters for three reasons. Tests become repeatable, because the same input always produces the same tool reply. Tests become safe, because no test run can issue a credit, send a message or change a record. Tests become cheap to run often, because nobody has to coordinate a sandbox account or wait on a vendor's rate limits. Anthropic's engineering notes on writing tools for agents stress that tool names, descriptions and return formats shape how well an agent uses them. Fixtures should therefore capture the real return format, not a tidied version of it.

A fixture is not a mock written from memory. A mock reflects what the developer believes the tool returns. A fixture reflects what the tool actually returned on a specific day, which is why recording from real responses is the core habit.

Which tools should be recorded, and which should be replayed only?

Not every tool needs the same treatment. Sort the agent's tools into three groups before recording anything:

• Read tools that return records, such as customer lookups, order status or stock levels. These are the easiest to record and the most valuable, because the agent's reasoning depends on their exact shape.
• Write tools that change state, such as issuing a credit, updating a status or sending a message. In tests, these must never reach the live system. Record what the agent tried to send, and have the replay return a success or an error you chose on purpose.
• Tools with side effects outside your control, such as a payment provider or a third-party messaging API. Record their responses, and pay close attention to rate limit and error codes, because the agent must handle those in production as well.

Also decide which fields are stable and which change on every call, such as timestamps, generated IDs and session tokens. Pin or normalize the unstable fields inside the fixture. Otherwise tests fail for reasons unrelated to the agent, and the team learns to ignore red results.

A step-by-step plan for building the first fixture set

A small team can build a useful fixture set without specialist tooling by following this sequence:

• Step 1: List every tool the agent can call, and write one sentence describing what each is allowed to do.
• Step 2: Run the agent in a controlled session with read-only access, and log each tool call with its arguments and response.
• Step 3: Choose realistic tasks that cover the common path, the edge cases and the failures you already know about, then save the logged calls for each task as one fixture set.
• Step 4: Remove personal and confidential data, replacing it with consistent placeholders so references still match across calls.
• Step 5: Write the expected behavior for each task: which tools should be called, in what order, with which required arguments, and what the final answer must contain.
• Step 6: Store fixtures and expectations in version control, next to the prompt and tool definitions they depend on.
• Step 7: Run the replay suite whenever any of those files change, and review every fixture diff with the same care as a code change.

Start small. A few well-chosen fixtures teach more than a large set copied from one busy afternoon.

Illustrative scenario: a support agent looks up an order, checks whether the return window is open, and drafts a reply to the customer. The team's fixtures include one lookup for a shipped item, one for a delivered item outside the window, one for an order with a missing address, and one timeout from the shipping API. Each comes from a real response, with the customer's name and contact details replaced by placeholders. The credit tool is replayed with a fixed success response, and the test checks that the agent calls it only in the case where the window was open.

What should the test actually check?

Checking only the final answer is the most common mistake. An agent can write a correct-sounding reply after calling the wrong write tool, or after reading a stale record. For every task, check four things:

• The sequence of tool calls, including whether the agent stopped when it should have.
• The arguments of each write call, especially identifiers, amounts and recipients.
• The handling of error and timeout responses, including whether the agent retried within its policy or escalated to a person.
• The final output against the facts in the fixtures, so every claim in the reply traces back to a recorded response.

OpenAI's evals guide describes defining tests against expected outcomes, and the same logic applies here, with the tool trace treated as part of the outcome. Anthropic's notes on evals for AI agents make a similar point about grading the path an agent took, not only where it ended. For write calls, add a check that an idempotency key or equivalent identifier is present whenever the tool supports one. The reasoning is covered in Idempotent AI Agent Actions: How to Prevent Double Refunds, and the fields worth keeping in each call trace are listed in Superintelligent Agent Audit Trail: Fields Every Action Needs.

Include permission tests as well. A fixture in which the agent's identity lacks the right to a write should produce a refusal or an escalation, not a workaround through another tool.

How do you keep fixtures current without letting them rot?

Fixtures age. A vendor renames a field, a status code changes meaning, or a policy shifts a return window. A suite that still passes against stale fixtures gives false confidence. Three habits guard against this:

• Re-record fixtures on a schedule and whenever a tool changes, and mark each file with the date it was captured.
• Keep a small set of live, read-only checks that compare current responses with the stored shape, so drift shows up before the replay suite fails.
• Treat a failed replay as a possible sign of real change in the outside world, not only as a bug in the agent.

Fixtures also deserve the discipline of production data. If the original responses were sensitive, restrict who can read the fixture set. Never let a fixture contain a live credential. Recorded responses sometimes include tokens, internal notes or other people's details, so the cleaning step is a review step, not an afterthought. For the ongoing side of this work, see A weekly maintenance checklist for managed AI agents.

How Be Human approaches the fixture set in managed operations

In Be Human's managed model, the fixture set is part of the agent's operating record from the first engagement. The first engagement defines one useful role, the tools that role needs, the approval boundaries and an observable success criterion, and the initial fixtures are built from exactly those tools and tasks. During managed operations, fixtures are re-recorded when tools change, permission reviews confirm which write tools the agent can reach in any test environment, and replay failures enter the same incident process as production issues. The agent's identity, memory and permissions remain under the client's control throughout, so test access never becomes a route to live systems.

If you are not sure which of your workflows is the right candidate for this kind of testing, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Can I test an agent with a sandbox account instead of fixtures? A sandbox is useful, but it is slower to set up and still depends on the vendor's behavior, which may differ from production. Fixtures are fast, repeatable and fully under your control. Many teams use fixtures for routine test runs and keep a sandbox for occasional end-to-end checks.

How many fixtures do I need before testing starts? Enough to cover the common path, the main edge cases and the known failures of the task you are testing. Begin with a small set for one role, then grow it from real escalations and incidents so each new fixture reflects a situation that actually happened.

Do recorded fixtures replace evaluations on real business tasks? No. Fixtures test how the agent behaves under known tool responses, while evaluations on real tasks test whether the work is good enough to use. Keep both, and treat fixture failures and evaluation failures as separate signals with separate owners.

What if a write tool has no test mode? Replay it and never call it. The test harness should intercept every write and record the attempted call. If a tool cannot be intercepted, do not give the agent that tool in any test environment until it can be.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.