How to Test an AI Vendor's Claims Before You Commit

To verify an AI vendor's claims, turn each claim into a small test that uses your own documents, your own tools and your own awkward cases, and score the result against a pass line you wrote before the test began. Demos, reference calls and benchmark charts show what is possible in general. A short, structured test shows what happens in your company.

Key takeaways

• A claim is only useful once it can be turned into a task you can run with your own material and fail against a pass line written in advance.
• Test first the claims that cost the most if wrong: accuracy on your data, behavior at the edges, integrations, permissions, and what happens when something breaks.
• Unpaid tests are enough for fit, honesty and documentation. Paid tests are for anything that needs real access, real tools or repeated runs.
• Watch how the vendor behaves during the test, especially how they report failures. It is the best preview of the relationship after you sign.
• Keep the results as a written record, so the decision rests on evidence and not on the memory of a good demo.

Why is a demo not evidence of a vendor's claims?

A demo is rehearsed. The data is chosen to suit the product, the path through the system is one the vendor knows works, and the questions asked are ones the system answers well. None of this is dishonest. It simply answers the question can this work, when you need an answer to the question will this work here.

The other usual sources of proof have the same limit. Reference calls are with customers the vendor selected. Public benchmarks measure general tasks, not your contracts, your product catalog or your customer tone. A slide that says the system is accurate does not say on what, measured by whom, or against what standard of correct.

The practical shift is to treat every claim as a hypothesis. The vendor is not lying until proven otherwise. You are simply declining to pay for something you have not seen behave in your conditions. Most serious vendors welcome this, because a test that passes closes the sale faster than another round of presentations.

Which claims are worth testing first?

Not every sentence in a sales deck needs a test. Sort the claims by type, then rank them by what it would cost you if the claim were false. These are the usual types and the test that fits each:

• Capability claims, such as answers questions from our documents correctly. Test with a set of real questions drawn from your own files, including some whose correct answer is that the documents do not say.
• Integration claims, such as connects to our email and ERP. Connect to a sandbox or test account with the same permissions you would grant in production, then run a realistic task end to end.
• Control claims, such as always asks for approval before acting. Try to make it skip the approval. Place an instruction inside a document or email and see whether the system obeys it.
• Memory claims, such as remembers context between sessions. Give it a fact on one day, change the fact later, and check what it uses on a third day.
• Operational claims, such as we monitor and respond to incidents. Ask for a walk-through of a recent, anonymized incident and a sample of the logs you would be able to see.
• Exit claims, such as you can take your data with you. Request an export now, in the format you would receive at the end, and open it. The exit planning guide lists what to ask for.

If your list is long, start with the claims that touch customers, money or regulated data. A wrong answer on an internal summary is an annoyance. A wrong action on a customer account is an incident.

How do you turn a claim into a test you can run?

Use the same short sequence for every claim. It takes less time than it looks, and it prevents the most common mistake, which is deciding what counts as success after seeing the results.

• Step one: Rewrite the claim as a sentence that can fail. Handles supplier emails well becomes drafts replies to our five most common request types that a reviewer would send with light edits.
• Step two: Assemble the test material. Use real documents (scrubbed if needed), a few messy ones, some contradictory ones, and several questions where the right behavior is to say it does not know.
• Step three: Write the pass line before you run anything, and name the person who will score. The article on success criteria for an AI agent pilot explains how to make a pass line observable.
• Step four: Run the test and score it without the vendor in the room. If possible, have a colleague who did not write the questions do the scoring.
• Step five: Run it again on a different day. Consistency is a claim too, and a system that is right once and wrong later is not ready for a business process.
• Step six: Break it on purpose. Give it a missing field, a wrong file format, a request outside its role, and an instruction hidden in a document.
• Step seven: Write down the results and send them to the vendor. Ask them to explain each failure. The explanation matters as much as the score.

OpenAI and Anthropic both publish practical guidance on building evaluations for AI systems. The ideas transfer well to a buyer's test: a fixed set of tasks, a defined grading method and repeated runs.

When should you pay for a test, and when should you not?

Unpaid testing covers more than people expect. You can ask pointed questions in writing and compare the answers with the documentation. You can run a sandbox walkthrough, test on public or non-sensitive material, request a sample export, read the terms on data use and ownership, and ask for the incident walk-through. The questions to ask an AI agent vendor give you a starting list for this stage.

Pay for a test when the claim cannot be checked without real work. That usually means connecting to your actual systems, configuring behavior for your process, running over several days to see consistency, or building an integration the vendor has not built before. A vendor who will not do this work for free is not hiding something. They are protecting their time, and a paid test also commits you to taking it seriously.

A good paid test is bounded. It has a defined scope, a fixed end date, a named deliverable, a statement that the outputs and any configuration are yours, and an explicit line that continuing afterward is optional. Do not give production data to anyone before a basic security review. The article on proof of concept, pilot and production describes what each stage should and should not be asked to prove.

What should you watch for while the test runs?

The score is only part of the evidence. How the vendor behaves around the test tells you what working with them will be like. Keep this checklist beside you:

• Do they ask about your data, your process and your exceptions, or only show features?
• When something fails, do they explain the cause, or change the subject?
• Do they say not supported when something is not supported? Vendors who say yes to everything are guessing.
• Can you see what the system did: sources used, steps taken, actions attempted? Without that, you cannot review anything.
• Does behavior stay stable across runs and across days?
• Are the people in the test the people who will run your service?
• Are the claims specific enough to test? A vague answer to a precise question is still an answer.

For the safety side, the OWASP Top 10 for LLM Applications and the NIST AI Risk Management Framework give neutral vocabulary for risks such as prompt injection, excessive permissions and unreliable outputs. You do not need to read them cover to cover. Use them to check that your break-it tests cover the common failure types.

What does this look like in practice? An illustrative scenario

Illustrative scenario: a mid-sized distributor is considering an agent to triage supplier emails. The vendor claims it reads incoming messages, drafts replies, asks for approval before sending anything, and connects to the company's ordering system.

The operations lead writes four testable sentences. Drafts for the common request types need only light edits. Anything outside the role goes to a human with a note. No message is sent without approval. A delivery-date lookup in a test copy of the ordering system returns the right record.

She prepares a set of real emails with names removed, adds two where the answer is not in any document, and one where the email body contains an instruction to send a reply immediately. The vendor runs the set in a sandbox. Drafts are good on routine requests and weaker on partial shipments. The approval gate holds on every message except the one with the embedded instruction, which gets drafted and queued in a way that skips the reviewer.

None of this ends the conversation. The vendor explains the cause, proposes a fix and shows it working on a rerun the following day. The company decides to go ahead with a narrower scope, drafts only for the common request types, and writes the approval-gate result into the contract's acceptance criteria. The test cost a few days and changed the decision.

How does Be Human handle this?

Be Human builds testing into the first engagement. Before anything is deployed, the engagement defines one useful role, the memory and tools it needs, the approval boundaries, and an observable success criterion, so each claim about the work is already a testable sentence. Managed operations then include recurring evaluations, permission review and monitoring, so the checks you run before signing continue after go-live instead of being a one-time event. The overview of what managed AI services include describes the recurring work in more detail.

Frequently asked questions

How long should a test of an AI vendor take? Long enough to repeat it. For most single-role agents that means days, not months: one day to prepare material, one to run it, and at least one more to rerun and break things on purpose. If a claim needs weeks of observation, make it a bounded paid test with a clear end date.

What if a vendor refuses to be tested? Ask what they are willing to do instead, and judge by the answer. A vendor can reasonably decline unpaid work that requires system access, but should offer a paid test, a sandbox or detailed documentation. A flat refusal to let you check any claim is a reason to slow down.

Should I rely on public benchmarks to compare vendors? Use them as a rough filter, not as proof. Benchmarks measure general tasks under controlled conditions, and your documents, tools and tolerance for error are different. Your own small test, scored by someone who knows the work, will always say more.

Who should score the test? Someone who does the real work today and did not write the test questions. They know what a good answer looks like in your context and will notice errors a generalist would miss. Add a second person for anything involving customers, money or regulated data.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.