Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Silent Failure Detection for AI Agents: Reconciliation and Sampled Review

Learn how to detect AI agents that finish without errors but write wrong records, using reconciliation counts, ID-level comparison and sampled human review.

Be Human typographic cover with the title: Silent Failure Detection for AI Agents: Reconciliation and Sampled Review
Conceptual illustration created with AI · Be Human

A silent failure is an AI agent run that completes with no error, no timeout and a success status, yet leaves wrong or missing records in the systems the business relies on. Detecting it takes checks that do not depend on the agent reporting on itself: reconciliation counts that compare what the source contains with what the agent wrote, and sampled human review that compares selected records against their original evidence. Used together, these two checks find the errors that logs and status codes never show.

Key takeaways

  • A silent failure looks like success in every log, so detection must compare outcomes against the source, never against the agent's own summary.
  • Reconciliation counts catch missing, duplicated and extra records cheaply, but only when each task has a written relationship between its inputs and outputs.
  • Sampled human review catches wrong values, wrong owners and wrong dates that totals cannot reveal.
  • Sample by risk, use a fixed checklist, and calibrate reviewers on the same cases before trusting their results.
  • Every discrepancy should end in a named fix, such as a rule, a test case or a tool change, not only a corrected record.

Why logs and status codes miss silent failures

A tool call that returns success confirms that the request was accepted and the system responded. It says nothing about whether the payload was the right one. The OpenTelemetry documentation on observability describes how traces and logs show what a system did, step by step. That is valuable for diagnosis, but a trace can faithfully record a correct sequence of steps that wrote the wrong customer identifier. Observability is a prerequisite for detection, not a verdict on business correctness. For what to capture alongside those records, see what to log.

Common silent failures in business agents look ordinary. The agent reads a source list, stops early because a pagination cursor was not followed, and reports completion. It matches a new invoice to the previous supplier because two names are similar. It writes a status copied from yesterday's export. It places the value of one field into a neighboring field after a schema change. In each case the final message says the work is done, and nobody is paged, because nothing technical failed.

The practical consequence is that the agent's own report cannot serve as evidence of success. A useful test is simple: if the agent's summary were removed, could someone still confirm the outcome from the source and the system of record alone? If not, the workflow has no detection layer yet.

Build reconciliation counts that compare sources, not summaries

Reconciliation means comparing two independent records of the same work. Counts are the fastest form of this check, and they work best when the relationship between input and output is written down before the agent runs.

Start with one task and write its expected relationship in a single sentence. For example: every intake message received in a shared mailbox should produce exactly one record, and every record should cite its message. Then:

  • Name the source of truth, such as the mailbox export, the order system or the ticket queue.
  • Write a query that counts source items for a fixed time window, stating the window's boundaries in one time zone.
  • Count the records the agent wrote in the same window, using a field that links each record back to its source.
  • Compare the two numbers and state in advance which difference is acceptable. For most record-creating tasks the tolerance is zero, with explained exceptions.
  • When the numbers differ, compare identifiers rather than totals to find which items are missing, duplicated or extra.

Totals alone hide errors. One missing record and one duplicate produce the same count, so a count-only check reports success while two items are wrong. Comparing identifiers closes that gap.

Illustrative scenario: a team uses an agent to log supplier intake emails into a tracking table. On a busy morning the mailbox export shows more threads than the table has new rows. The identifier comparison reveals that the agent stopped at the end of the first results page and still marked the run complete. The agent did not refuse any message. The fix belongs in the retrieval step, and a test case with more than one page of messages goes into the evaluation set.

Add sampled human review for what counts cannot see

Counts verify that the right number of records exists. They cannot verify that each record is right. A supplier name can be attached to the wrong invoice while the totals still match. For that, a person needs to compare a sample of records against their evidence. The NIST AI Risk Management Framework treats measurement and monitoring as continuing functions rather than a one-time test, and this kind of review is a practical way to keep measurement running.

Sampling works when it is deliberate. Divide the work into groups by risk, for instance by value, by whether the action can be reversed, or by whether the task type is new. Sample more often from the high-risk groups, and choose items at random within each group so reviewers do not see only the easy cases. The size of each sample should follow the risk and the volume, and it should be small enough that a reviewer can finish it in one sitting. For a method on sizing, see how many outputs to review.

Reviewers need a fixed checklist that compares each record field by field against the source: the identifier, the owner, the dates, the quantities, the status, and any claim the source does not support. Reviewers should work from the source document, not from the agent's explanation. Before the first real review, have two reviewers score the same set of cases independently and discuss every disagreement. Without that calibration, a review produces a feeling of control rather than a measurement.

Record each finding with a category such as wrong value, wrong entity, missing field, stale input or unsupported claim. Categories turn a list of mistakes into a pattern you can act on.

Turn each discrepancy into a named fix

A detected silent failure should change the system, not only the single record. For each finding, decide which kind of fix addresses its cause. A retrieval problem calls for a change in how the agent reads sources. A mapping problem calls for a schema check at write time. An ambiguous instruction calls for a rule, written so the next run follows it. A missing test calls for a new case in the evaluation set, so the same failure is caught before it reaches production. The evaluation guidance from Anthropic's engineering team makes a similar point: evaluations should use realistic tasks and run repeatedly, so that regressions surface early.

Keep the fix log visible to the people who own the workflow. Each entry should name the failure category, the affected records, the suspected cause, the change made and the test that now covers it. When a finding suggests that wrong data may have been acted on more widely, treat it as an incident. The steps for that are in the incident response playbook.

A weekly routine for detecting silent failures

The routine below fits in one working session per workflow. It is a starting structure, to be adapted to the risk of each task.

  • Pull the source and system-of-record counts for the last period, and compare them by identifier.
  • List every missing, duplicated or extra item, and classify each one by likely cause.
  • Draw the review sample from each risk group, and record how the selection was made so it can be repeated.
  • Review each sampled record against its source with the fixed checklist, and log every disagreement with its category.
  • Check whether any record was edited by a person after the agent wrote it, since manual edits can hide or create problems.
  • Add each new failure type to the test set, and assign one owner to every open fix.
  • Note any change in volume, source format or tool behavior, because these often come before silent failures.

How Be Human handles silent failure detection

Be Human's managed operations treat detection as part of the agent's design rather than an afterthought. The first engagement defines the useful role, the memory and tools it needs, the approval boundaries and the observable success criterion, and the reconciliation rules and review categories are written down at that stage. Ongoing operations then run the counts on a schedule, keep the review sample and checklist current, track each fix to completion, and review permissions whenever a workflow changes. The aim is that the business can see, in plain terms, whether the agent's records match their sources.

If you want to find the first task worth handing to an agent and the checks it would need, take the free Be Human AI diagnostic.

Frequently asked questions

How is a silent failure different from a normal error? A normal error produces a signal, such as an exception, a timeout or a rejected request, so someone sees it and responds. A silent failure completes with a success status while the output is wrong or incomplete, so it stays hidden until someone compares the result with the source.

How often should reconciliation counts run? Run them at a frequency matched to how quickly wrong records cause harm. For tasks that feed payments, customer commitments or deadlines, a daily comparison is a reasonable starting point, while lower-stakes reporting may need only a weekly one. Choose a frequency you can keep without skipping, because a missed check is the same as no check.

Do I still need human review if the counts match? Yes. Matching counts show that the right number of records exists, not that each record contains the right values, owner or date. Sampled review checks content, and it can continue at a lower rate once the workflow proves stable, with more frequent review after any change to tools, sources or instructions.

Can an automated judge replace the human reviewer? An automated judge can help triage, for example by flagging records for a person to check, but it should be calibrated against human decisions before anyone relies on it. Its agreement with reviewers must be measured on known cases. Final decisions about whether a record is wrong should stay with a person who can read the source.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →