When a Superintelligent Agent Misreads a PDF: Checks First

An AI agent misreads a PDF when it turns a scanned or table-heavy document into structured fields and then acts on those fields as if they were correct. The practical fix is not only a better model. It is a set of checks that run before any payment, booking, record update or outgoing message is triggered: totals reconciliation, field-level confidence flags, required-field rules and a hold path to a person.

Key takeaways

• A PDF becomes usable data only after its fields pass checks that do not depend on the model's own judgment.
• Totals reconciliation is the cheapest strong check, because line items, subtotal, tax and total must agree with each other.
• Confidence flags should route uncertain values to a reviewer, not simply label them.
• A failed check blocks the action, and the agent must not adjust a value to make the check pass.
• Keep the page reference, extracted values and check results in the log so a wrong read can be traced later.

Why extraction errors reach real actions

Most PDF misreads are quiet. A decimal moves, two columns of a table merge, a date is read in the wrong order, or a stamp covers part of a figure. The output still looks tidy, and the agent proceeds because nothing in its own process says otherwise. The risk rises with write access. An agent that can create an invoice record, change a vendor's bank details or schedule a delivery turns one bad read into a real operational event.

The underlying problem is that the agent's confidence in its own reading is the only control in the loop, and that is the control that failed. Sound operations design adds a second path that does not rely on the same reading. Checks that compare extracted values with each other, with the layout of the source and with records already on file give you that second path.

Start with totals reconciliation

Reconciliation asks whether a document agrees with itself. Each line total should equal quantity multiplied by unit price, within a rounding rule you write down in advance. The line totals should sum to the stated subtotal, and the subtotal plus tax and fees should equal the stated total. When these relationships fail, the document is not usable as read.

A mismatch tells you that something is wrong, but not which value is wrong. That is why the agent should never choose the number that makes the arithmetic work. It should hold the item and show the reviewer the conflicting values with their location on the page. Reconciliation rules worth writing down before a pilot starts:

• Line totals equal quantity multiplied by unit price, within a stated rounding tolerance.
• Line totals add up to the stated subtotal on the same page or within the same table.
• Subtotal plus tax and fees equals the stated total.
• Dates fall inside an expected window and use one format across the whole document.
• Currency, tax identifier and vendor name match the record already on file, or the difference is flagged.

Use field confidence flags to route work

Extraction tools and models often report a confidence value for each field. That value is useful only when it changes what happens next. Set three bands in advance: accepted, reviewed and blocked. Accepted fields flow to the next step. Reviewed fields go to a person with the page image beside them. Blocked fields stop the action until someone resolves them.

Treat the score as a signal, not as a probability you can rely on without testing. The article on confidence scores in business workflows explains how to check whether a score actually tracks accuracy on your documents. Two rules matter in practice. Flag low confidence on any field that carries consequence, such as an amount, a due date or a bank account, even when the rest of the document looks clean. Treat scans, handwriting and low-resolution images as higher risk by default. A high score on a field that fails a reconciliation check should still be held, because the two signals disagree, and that disagreement is the event to surface.

A validation sequence before the agent acts

A short, fixed sequence covers most cases. Keep it as a written procedure that the agent follows and a reviewer can audit.

• Classify the document and confirm it belongs to a type the agent is allowed to process.
• Extract text and tables while keeping the page and position of every value.
• Check the schema: required fields present, types valid, identifiers in the expected format.
• Run reconciliation on totals, dates and currency.
• Compare identifiers with the system of record, and treat any mismatch with the vendor or customer on file as blocking.
• Route each consequential field by confidence band, sending reviewed and blocked items to a person.
• Let the agent draft the action only when every check passes, and apply the approval rule for that action class before anything executes.
• Write the document reference, extracted values, checks run and final decision to the log.

An illustrative scenario: the transposed total

Illustrative scenario, not a client case: a supplier invoice arrives as a scanned PDF with a readable vendor name and due date. The agent extracts the line items and the total. Two digits in the total are transposed during extraction, so the line items add up to a different figure than the stated total. The reconciliation check fails, and the agent marks the total as blocked instead of passing the record to the payables step.

The reviewer sees the page image, the extracted total, the line items and the calculated sum side by side. They confirm that the line items were read correctly and that the printed total is the value in question, so they request a corrected copy from the supplier. The record is created only after the corrected document passes every check. The useful outcome is not that the agent was clever. The error stopped before it reached a payment file, and the log shows exactly why.

What the agent should never decide on its own

Some actions should stay outside the agent's discretion however clean an extraction looks. It should not alter a value to make a check pass, override a failed check, or change payment or bank details on the strength of a single document. Those changes need a second source and a written approval rule. When a blocked item cannot be resolved, send the decision to a person rather than forwarding the whole file, as described in human escalation for AI agents.

Documents also carry text that was never meant as instructions for an agent. A footer, a hidden field or a line in an attachment can try to steer the agent's behavior. Guidance on prompt injection treats external content as untrusted input, so document text belongs in the data path and never in the instruction path. The article on prompt injection in business documents covers what to protect and how to test for it.

How Be Human keeps these checks running

Be Human's managed approach treats document reading as one defined role with written rules, not as a general assistant with open access. The engagement records the document types, reconciliation rules, confidence bands, blocked actions and approval rule for each action. Managed operations then review those rules as the document mix changes, sample held and accepted items, and update integrations so the checks keep reading the same fields. The agent's memory and tools stay limited to what that role needs, and every decision is logged with its source reference. If you want to find the first document task worth handing to an agent, take the free Be Human AI diagnostic.

Frequently asked questions

How do I stop an AI agent from acting on a misread PDF? Place validation between extraction and action, so the agent must pass schema and reconciliation checks before it drafts anything that changes a record. Route low-confidence fields that carry money, dates or bank details to a person, and block the action until those gates clear. Write the approval rule before the pilot starts, not after the first error.

Are confidence scores from a PDF tool enough to skip human review? No. A score describes the tool's internal certainty, which may not match real accuracy on your documents. Use scores to route work, test them against a sample of corrected documents, and keep review on any field where an error would be costly.

Should the agent correct a total that does not add up? No. A reconciliation failure shows that the document or the extraction is inconsistent, but it does not show which value is right. The agent should hold the item, show the reviewer the conflicting values with their location on the page, and wait for a corrected source or a human decision.

Do scanned documents need stricter checks than digital PDFs? Usually yes. Scans, handwriting, stamps and low-resolution images raise the chance of character errors, so treat them as higher risk by default, lower the threshold for review and sample more of their output. Digital PDFs still need reconciliation, because the arithmetic check does not depend on how the text was produced.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.