How to Evaluate AI Case Studies Without Being Misled

To evaluate an AI case study, treat it as a claim to be checked, not a story to be believed. Check four things: the scope of what was actually deployed, the baseline the result is compared against, how the outcome was measured, and what the write-up leaves out. If a case study cannot answer those four questions, it is marketing, however well it is written.

Key takeaways

• A case study can be true in every sentence and still mislead, because the choices about what to include are made by the seller.
• Scope matters first: a result from one narrow task in one team says little about a broad deployment in yours.
• A result without a stated baseline has no meaning. Ask what the work looked like before, and who measured it.
• Measurement method decides whether a number is evidence or decoration. Look for a defined metric, a defined period and an independent check.
• Omissions are the most informative part. Failures, rework, human review effort and ongoing costs are usually missing, so ask for them directly.

Why do AI case studies mislead even when nothing is false?

A case study is a selected account. Someone chose the project, the time window, the metric and the words. That does not make it dishonest, but it does make it a document written to persuade, and persuasion rewards a clean arc: a problem, a deployment, an improvement.

Real AI work rarely follows that arc. Agents behave differently across tasks, people adapt around them, data is messy, and the first version is rarely the one that runs a year later. A write-up that compresses all of this into a single improvement is leaving out the part you most need, which is how the result was produced and whether it would survive in your conditions.

There is also a structural bias. Companies publish the engagements that went well, and they publish them after they went well. You rarely see the ones that were narrowed, paused or rebuilt. That is normal, and it means the burden of checking falls on you, the reader. The goal is not cynicism. It is to ask the questions that turn a story into something you can decide on.

What was the scope, and does it match yours?

Scope is the boundary of what the AI actually did. Many case studies blur it with phrases such as transformed support or modernized operations. Those phrases describe an ambition, not a deployment.

Look for the specific role. What task did the agent perform, for which users, using which data and tools, with what authority? An agent that drafts replies for a human to approve is a different thing from one that sends them. An agent that reads a single document library is a different thing from one connected to billing, calendars and customer records. If the case study does not say, the scope is probably narrower than the headline suggests.

Then compare it to your situation:

• Is the process similar in volume, variety and exceptions, or is it unusually simple?
• Was the data clean and well organized, or comparable to what you actually have?
• Were the users a handful of enthusiastic volunteers or a typical team?
• Did the deployment have a dedicated owner and review routine, or does the story assume one that you do not have?

A result from a tidy, narrow task can be real and still irrelevant to a messy, broad one. For a sense of what each stage of a project should and should not prove, see Proof of Concept, Pilot or Production: What Each AI Stage Must Prove. A case study from a proof of concept should not be read as evidence about production.

What was the baseline?

Every improvement is relative to something. The baseline is what the work looked like before the AI arrived, and it is the most common place for a case study to quietly flatter itself.

Watch for these weak baselines:

• A comparison to a process that was already broken, so almost any change would look like progress.
• A comparison to an estimate or a memory of how long things used to take, rather than recorded data.
• A comparison to the worst week, the slowest team or the oldest tooling, instead of typical operation.
• A before-and-after that ignores other changes made in the same period, such as new staff, a new policy or a seasonal shift.

A good case study says how the baseline was captured, over what period, and whether anything else changed at the same time. Better still, it compares the AI-assisted work with a parallel group or with the same work done the old way on the same material. If none of this is stated, assume the comparison is informal.

The question to ask is simple: compared with what, measured when, and by whom?

How was the result measured?

A number is only as good as its method. Two case studies can report the same improvement and mean entirely different things.

Start with the metric itself. Time saved can mean time to first draft, time to a finished and checked output, or time the employee believes they saved. Accuracy can mean agreement with a reviewer on a sample, or the absence of complaints. Satisfaction can be a survey sent to the people who chose to use the tool. Each of these is a legitimate measure, and each answers a different question.

Then look at how the measurement was done:

• Was the metric defined before the deployment or chosen afterward because it looked good?
• Was it measured on all the work or on a curated sample?
• Did someone independent of the seller verify it, or is it self-reported?
• Were errors counted, including those caught by human reviewers before they reached a customer?
• Does the number include the effort of reviewing and correcting the AI output?

That last point is frequently missed. If a person must check everything the agent produces, the saving is the difference between doing the work and checking it, not the full time of the original task. Reputable guidance on evaluating AI systems, such as the material on evals from [Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) and [OpenAI](https://platform.openai.com/docs/guides/evals), stresses defined test cases, clear grading criteria and repeated measurement. A case study that describes none of this is describing an impression.

If you are setting up your own measures, How to Define Success Criteria for an AI Agent Pilot covers how to decide what to measure before the work begins.

What was left out?

The omissions are where most of the useful information sits. A case study will not volunteer them, so you have to ask. Use this list as a starting point:

• Cost of operation over time, including the human work of review, maintenance and monitoring, not only the build.
• Failure cases: what the agent got wrong, how often it was noticed, and what happened next.
• Changes made after launch, and how many iterations it took to reach the reported result.
• Who owns the system today and who handles incidents.
• What data the agent can access and under what permissions.
• How long the result has held. A strong first month is different from a stable year.
• What happened to the people involved: whether they adopted it, worked around it or stopped using it.
• What would not transfer: the conditions that made this engagement easier than yours.

Notice what a reader could do with a case study that answered these. They could judge risk, plan staffing and compare providers fairly. A case study that is silent on all of them gives you a result and withholds the conditions required to reproduce it.

A practical plan: how to read a case study in six steps

Use the same routine every time, so that you compare providers on equal terms.

• Restate the claim in one plain sentence, stripped of adjectives. If you cannot, the claim is vague.
• Write down the scope: task, users, data, tools, level of autonomy and human approval.
• Identify the baseline and how it was captured. Mark it as recorded, estimated or unstated.
• Identify the metric, the period and who measured it. Mark it as independent, shared or self-reported.
• List what is missing from the omissions list above, and turn each gap into a question.
• Ask the provider to answer the questions in writing, and note which answers are specific and which are evasive.

The last step often tells you more than the case study did. A provider who answers plainly, including with a limitation, is showing you how they will behave during a real engagement. For further ways to verify what a vendor tells you, read How to Test an AI Vendor's Claims Before You Commit.

### An illustrative scenario

This is an illustrative example, not a real engagement. A case study says an AI agent reduced the time a distributor's team spent on order queries. Reading it with the routine above, the reader finds that the agent handled only one category of query, for one team, with every reply approved by a person. The baseline was the team's recollection of how long queries used to take. The metric was the time until a draft was ready, not the time until the query was closed. Nothing was said about corrections or about who maintains the connection to the order system.

None of this makes the story false. It means the honest conclusion is narrower: drafting a single category of query is faster with the agent. The reader can now ask better questions, such as how often drafts were corrected, and decide whether that narrow claim matters for their own process.

How does Be Human approach this?

Be Human designs, implements and operates AI agents for companies, with identity, memory, tools and permissions under human control. Because of that, we frame an engagement the same way a careful reader should frame a case study: one defined role, the memory and tools it needs, approval boundaries and a success criterion that can be observed. The baseline and the measurement method are agreed at the start, and managed operations include recurring evaluations so that results are checked over time instead of reported once. We do not publish prices or client results, and we suggest you apply the same scrutiny to anything we write. If you are comparing providers, How to Choose an AI Implementation Partner for Your Company lists further criteria.

Frequently asked questions

How many case studies should I ask a provider for? Ask for as many as are relevant to your process, but weigh quality over quantity. One case study with a clear scope, a recorded baseline and a stated method is worth more than several that describe only outcomes. If a provider cannot share detail because of confidentiality, ask for a conversation that covers the method without naming the client.

Are anonymous case studies less trustworthy? Not necessarily, since many companies have good reasons not to be named. What matters is whether the case study is specific about scope, baseline and measurement. An anonymous case study that answers those questions is more useful than a named one that does not.

What if a provider has no case studies yet? A newer provider may have little to show, and that is not disqualifying on its own. In that case, ask for a small, bounded pilot with success criteria agreed in advance, and judge the provider on how they define and report results. Your own measured pilot is better evidence than anyone else's story.

Should I trust numbers from independent reports or industry surveys more than case studies? Treat them with the same four checks. A survey has a scope too, in who was asked and what they were asked, and it may rely on self-reported answers. Broad reports can help you understand general trends, but they cannot tell you how an agent will perform on your data and processes. That is something only a defined test in your own environment can show.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.