Skip to content
be-human.ai
← Back to the journal

Tools & integrations

Structured Outputs for AI Agents: Schemas Your Systems Accept

Learn how to define output schemas for AI agents so results are validated before they reach billing, CRM or scheduling systems. Steps and checks included.

Be Human typographic cover with the title: Structured Outputs for AI Agents: Schemas Your Systems Accept
Conceptual illustration created with AI · Be Human

Structured outputs for AI agents are results in a fixed, machine-readable format, usually JSON, that follow a schema the receiving system defines. Before a result reaches a billing, CRM or scheduling system, a validator checks its types, required fields, allowed values and business rules, then accepts it, holds it for review or rejects it. The agent proposes, the schema constrains, and the target system only receives what it can safely accept.

Key takeaways

  • A schema fixes what a valid result looks like before the agent runs, so downstream systems never have to guess what a field means.
  • Validate in layers: structure first, then allowed values and existing identifiers, then the business rules of the target system.
  • Validate before any write, and treat a failed check as a routed exception with a named owner, not a silent retry.
  • Take field names, types and required fields from the receiving system's documentation, not from what the model happens to produce.
  • Test the schema with ordinary inputs and deliberately messy ones before any agent writes to live records.

What does an output schema actually guarantee?

A schema is a contract. It says which fields a result must contain, what type each field holds, which values are allowed and which combinations are impossible. It guarantees shape and vocabulary. It does not guarantee that the content is true. An agent can return a well-formed invoice with a total that is wrong, or a booking with a valid time that nobody offered. The schema's job is to make malformed or out-of-scope results visible and rejectable, while other checks confirm the facts.

Many model platforms can constrain generation to a JSON schema, which reduces malformed output. Treat that as a helpful first filter, not as validation. OWASP's Top 10 for LLM Applications describes model output as untrusted input that needs checking before a downstream component uses it. Your application still has to test every result against the rules of the system that will receive it.

Start from the receiving system, not the model

Begin with the destination. A billing system typically expects a customer identifier, a currency code, line items with quantities and tax treatment, and an issue date in a specific format. A CRM expects a contact identifier, a pipeline stage from a fixed list and an owner who exists as a user. A scheduling system expects a start time with a time zone, a duration, a resource and no overlap with existing bookings. Copy those requirements into the schema as they are. If the agent invents its own names, a translation step appears, and translation is where errors hide.

Collect these details from each target system before you write any schema:

  • Required fields, using the exact names the system uses
  • Types and formats, including dates, time zones and decimal precision
  • Enumerations: the closed lists of statuses, categories or stages
  • Identifiers that must already exist in the system before a write can succeed
  • Write rules: which fields can be set on creation, which are read-only, and which need a specific role

For a deeper treatment of how to define the contract around a tool, see AI tool contracts: design inputs and outputs that work. The same discipline applies to outputs: a schema is only useful if the receiving side agreed to it.

How do you structure a schema in three layers?

Separate the checks into three layers so that a failure tells you what went wrong.

  • Shape: required fields exist, types are correct, and unknown fields are rejected rather than silently dropped.
  • Vocabulary: values come from closed lists, and identifiers resolve to real records in the target system.
  • Business rules: totals equal the sum of their lines, the appointment falls inside working hours, the discount does not exceed the limit that the approver set.

Layer one is cheap and deterministic, so run it first and stop on failure. Layer two needs a lookup against the live system, ideally through a read-only view. Layer three is where domain knowledge lives and where most of the real operational risk sits. Keep these rules in code or configuration that a person can read and audit, not buried in a prompt that changes with every edit.

A step-by-step plan for validating before writes

Use this sequence for each action the agent can take:

  • Define the schema for one action only, such as creating a draft invoice or proposing a calendar slot, rather than a general schema for an entire system.
  • Write the schema from the target system's documentation, then ask the person who owns that system to review it.
  • Check every agent output against layer one first, and return the errors to the agent only if a retry is allowed.
  • Resolve every identifier and enumerated value against the live system before reading the business rules.
  • Run the business rules and produce one decision: accept, hold for review, or reject with a reason in plain language.
  • Send only accepted results to the write step, and attach an idempotency key so a retry cannot create a second record.
  • Log the input, the proposed output, the validation result and the write outcome together, so any record can be traced later.

Providers document this pattern. Stripe's API, for example, supports idempotency keys so that a repeated request does not repeat the action. For how to design those keys for your own actions, read idempotency for AI agents: prevent duplicate actions.

What happens when a result fails validation?

Decide in advance what each class of failure does. A schema failure usually means the agent misread the request, so one retry with the specific validation errors included is reasonable. A lookup failure, such as a customer identifier that does not exist, should not be retried blindly, because the same lookup will fail again. It should go to a person or to a search step designed for that purpose. A business rule failure is a decision, not a bug. A discount above the threshold belongs to whoever approves discounts.

The agent must never adjust its own output to pass a rule it failed. Set a retry limit, and when it is reached, move the item to a review queue with the original input, the attempted output and the error text. Scheduled runs need the same treatment, which is covered in scheduled AI agent tasks: runs, retries and alerts that reach a person.

How Be Human handles this: in a managed engagement, Be Human designs each schema with the owner of the receiving system, implements the validator as part of the agent's tool layer, and limits write permissions to results that have been accepted. Monitoring tracks rejections by field and by rule, so a vendor's format change or a shifted business rule shows up as a visible pattern rather than a surprise in the ledger. Scheduled permission reviews and controlled schema changes keep the contract current as the systems around it change.

Illustrative scenario: a booking that does not fit

Illustrative scenario: a field service company uses an agent to turn customer messages into proposed visits. A message asks for a Sunday visit. The agent's output contains a start time without a time zone, and the technician identifier it produced does not exist in the scheduling system. The shape check rejects the missing time zone before anything else runs. The lookup then confirms the technician identifier is invalid. Nothing is written. The request, both errors and the customer's original message go to the dispatcher, who selects an available technician by hand and confirms the slot with the customer. The agent did not fail silently, and the scheduling system never received a booking it could not place.

Test the schema before it touches live records

Build a test set from real past requests, and include deliberately difficult cases: missing fields, messages written in two languages, dates in day-first and month-first formats, amounts with thousand separators, and duplicate submissions. Run the agent against this set after every change to the prompt, the model or the schema. Score validation outcomes, not only whether the text reads well. Anthropic's guidance on writing tools for agents stresses testing how agents actually use a tool, because the failures that matter are often in the details of inputs and outputs. Use this checklist before go-live:

  • Every required field is present, or the output is rejected.
  • Unknown fields are rejected, not quietly removed.
  • Dates include a time zone and match the target format exactly.
  • Money fields follow the precision and rounding rules of the billing system.
  • A repeated request produces the same idempotency key, not a second record.
  • Every rejected case reaches a named person with a clear reason.

If you are not sure which of your tasks is the right first one to validate this way, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.

Frequently asked questions

Do I still need validation if the model supports structured outputs? Yes. Schema-constrained generation reduces malformed JSON, but it cannot confirm that a customer identifier exists, that a discount is within policy or that a time slot is free. Your application needs its own checks against the receiving system before any write. Treat the platform feature as the first filter and your validator as the gate.

Should the schema live in the prompt or in code? Keep it in code or in versioned configuration. A schema inside a prompt is hard to test, easy to change by accident and invisible to the people who own the receiving system. When the schema is a file under version control, you can review it, test it and roll it back when the target system changes.

What happens when the receiving system changes its fields? The schema has to change with it, through a controlled review rather than a quick edit. Monitor rejections by field so that a renamed or newly required field appears as a pattern within the normal review cycle. Pause writes for that action until the schema is updated and the test set has been rerun.

Can a schema catch wrong facts? Only partly. A schema can confirm that a value is well-formed, allowed and consistent with the rules you wrote down, and a lookup can confirm that a record exists. It cannot confirm that the customer really asked for the Sunday visit. For facts that matter, add a review step or a second source, and keep the agent's proposal and the evidence it used in the log.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →