Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Synthetic Test Data for AI Agents: Realistic Records, No Real Customers

Build synthetic test records that mirror real business cases, so you can evaluate AI agents without exposing customer or employee data. A practical method for operations leaders.

Be Human typographic cover with the title: Synthetic Test Data for AI Agents: Realistic Records, No Real Customers
Conceptual illustration created with AI · Be Human

To test an AI agent without exposing customer or employee data, build synthetic records that copy the shape of real work: the same fields, the same messy inputs and the same decision points, with no real person's details inside. Write down the decisions the agent makes, generate records that force each decision, and attach the outcome a skilled person would accept. Keep the generator, the expected answers and the access rules under version control so the same test set can run again after every change.

Key takeaways

  • Synthetic records should match the structure and difficulty of real work, not the identities of real people.
  • Every test case needs an expected outcome written by someone who runs the process today.
  • Include errors, missing fields, conflicts and unusual requests, not only clean examples.
  • Generate and run test data in an environment where the agent holds no production credentials.
  • Review the test set on a schedule, because work changes and stale cases give false confidence.

Why real records should stay out of agent tests

Testing with production data feels faster because the data already exists. Every copy of a customer file, however, creates another place where personal data can leak, be kept longer than intended, or be read by people and tools that never needed it. Employee records carry the same risk with added sensitivity, since they often include pay, performance notes and health-related absences.

Governance frameworks such as the NIST AI Risk Management Framework ask organizations to document how AI systems are tested and what data they touch. Privacy rules, including Brazil's data protection law (LGPD), expect personal data to be used only for a stated purpose. You do not need a legal essay to act on this. The test environment should hold only what the test requires, and nothing more.

Start from the decisions, not the database

Most teams begin by copying a database schema. A schema tells you the shape of the data but not which parts matter for the agent's work. Begin instead with the decisions the agent makes: which category to assign, which priority to set, whether to reply automatically, whether to write to a system or hand the case to a person. For each decision, list the inputs a person would look at before deciding. Those inputs define the fields your synthetic record must contain.

The type of decision also determines what the test must check:

  • If the agent only drafts text for a person to approve, test the facts used, the tone and whether it invents details.
  • If the agent writes to a system, such as a refund, a booking or a status change, test the exact fields it writes and the conditions that should block the write.
  • If the agent retrieves documents to answer a question, test whether it finds the correct document and cites it.
  • If the agent routes work to people, test whether the handoff contains the facts the receiver needs without the full history.

Build the generator in six steps

A useful generator is simple enough that a process owner can read it. Work through these steps in order:

  • Write a field dictionary with name, type, allowed values, format and the person who fills each field in the real process.
  • Generate names, addresses and identifiers from a fictional pool, never by editing real records and changing a few letters.
  • Use reserved or obviously invalid formats for identifiers such as tax or national numbers, so a synthetic value cannot be mistaken for a real one.
  • Set the distribution by hand for the cases that matter, such as rare product types, unusual regions or uncommon approval paths, instead of matching production averages.
  • Attach an expected outcome to every record, written by a person who knows what a correct result looks like.
  • Label each record with the scenario it covers, so failures can be grouped and the weak areas become visible.

Store the generator and the expected outcomes together, with a version number. When the process changes, you update both, and the test history shows which version each result came from.

Avoid training a data generator on real records to make the output look more realistic. A model trained on customer data can reproduce rare combinations that point back to a real person. A rule-based generator is less impressive in a demo, but it is easier to explain and to audit.

Add the messy cases real work contains

Clean synthetic data produces clean scores, and clean scores tell you little about production. Real requests include misspelled names, duplicate submissions, two messages that contradict each other, a missing postal code, a request written in Portuguese with English product names, dates written in several formats, and an attachment that is a scanned image with a crooked page. Include these deliberately, and weight the set toward the cases where a mistake would cost the most.

Illustrative scenario: a service company tests an agent that answers billing questions. The generator creates a fictional customer with two invoices, one of which was paid twice, and a message asking for a refund on the other invoice. The expected outcome is a refund request routed to a person, with both invoices listed, and no automatic payment. An agent that passes only the simple cases will often fail this one, and that failure is the reason the case belongs in the set.

When a real case goes wrong, convert it into a synthetic version the same day. Keep the structure and the difficulty, replace every identifying value, and add the corrected outcome to the set. The agent then has to keep passing a case that already caused trouble once.

Keep test data away from production permissions

Synthetic data protects people only if the agent cannot reach real systems while it runs the tests. Run evaluations against sandbox tools with credentials scoped to test accounts. The authorization principles in the OWASP Authorization Cheat Sheet apply here: each identity should hold only the rights its task needs, and nothing that lets a test reach a live customer record.

In practice, this means:

  • Separate credentials for test and production, with no shared secrets between them.
  • Tool mocks that record the intended write instead of executing it, so you can inspect what the agent tried to do.
  • Memory isolated per environment, cleared between test runs unless a specific case needs persistence.
  • Logs that store the generator's identifiers, never values copied from production, even when the logs are shared with vendors or reviewers.

Check these settings after every integration change. A new tool or a widened token scope can quietly undo a boundary that worked last month.

How to check that the test set is realistic enough

Realism means the structure matches your work. It does not mean the names look plausible. Ask a process owner to read a sample of records and answer two questions: would a record like this appear in a normal week, and would the expected outcome be the one they would choose? Records that fail either question should be rewritten before the test set is trusted.

Then check whether the synthetic results predict real behavior. Run the agent in shadow mode on real work, where it proposes actions but a person decides, and compare its proposals with the synthetic results. If the agent handles every synthetic billing case correctly but makes errors on real ones, the set is missing a pattern, and the generator needs a new scenario. The shadow mode pilot approach is the usual next step before granting any authority.

Keep the set current. Retire cases that describe a process you no longer run, and add cases for new products, new channels or new rules. A golden test set that stays small and reviewed is more useful than a large one nobody maintains. For the broader question of which work should be measured first, see how to evaluate AI agents on real business tasks.

How Be Human handles synthetic test data in managed operations

In a Be Human engagement, the test data for each agent role is designed in the first phase, alongside the role itself: the fields the agent needs, the synthetic generator, the expected outcomes written with the process owner, and the sandbox tools used during evaluation. Permission reviews and recurring evaluations then keep the test set aligned with the work as it changes, and the generator is updated when a new case type appears in production.

If you are not sure which task to test first, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Can I use real records if I remove names and email addresses? Removing direct identifiers is a useful first step, but it is rarely enough. Order details, dates, locations and rare combinations can still point to one person, and employee records often reveal more than they appear to. Treat any copy of real data as sensitive, and confirm the approach with your privacy or legal lead before using it.

How many synthetic cases do I need before the test means something? Start with a small set that covers every decision path and every failure you have already seen in production. Adding many similar records mostly adds cost without adding information. Grow the set when a new scenario appears, and retire cases that no longer reflect the work.

Will synthetic data show how an agent will behave on real work? It will show how the agent handles the situations you thought to include, which is valuable but incomplete. Pair synthetic evaluations with a supervised period on real work, where the agent proposes actions and a person approves them. The gap between the two results tells you where the test set needs new cases.

Does synthetic data automatically satisfy privacy rules? Synthetic data reduces exposure when it contains no real personal information and cannot be linked back to a real person. Whether your specific use meets the rules depends on the purpose, the jurisdiction and how the data was produced, so document the generation method and review it with the people responsible for privacy.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →