LLM Judges for AI Agent Evals: Where They Mislead and How to Check Them

An LLM judge is a language model that grades another model's output against a written rubric, which makes it practical to score many agent answers that no person could read. It works best for criteria that are concrete and checkable, such as whether an answer cites the record it used or follows a required structure. It misleads when it shares the agent's blind spots, when the rubric rewards fluent wording over correct facts, or when nobody has tested the judge against human decisions.

Key takeaways

• A judge is a measuring instrument, so it needs its own calibration before its scores drive any decision.
• Use judges for narrow, observable criteria: format, required fields, citation presence, and policy wording.
• Be cautious with judges for factual accuracy, domain judgment, and anything where judge and agent come from the same model family.
• Spot-check the judge against human labels on a fixed sample, and read every disagreement rather than relying on a single agreement figure.
• Keep the rubric, the judge version, and the sample under the same review schedule as the agent they grade.

What an LLM judge does, and why teams reach for one

Human review remains the reference point for quality, but it does not scale to every output an agent produces. A judge gives fast, repeatable scores on the same criteria each time, so a team can watch trends across hundreds of runs instead of reading a handful by hand. Anthropic's writing on agent evaluations separates code-based graders, model-based graders, and human graders, and the useful question is which kind fits each criterion. A judge is a model-based grader. It is valuable precisely because it can read language, and that same ability is the source of most of its failures.

The mistake many teams make is to treat the judge as an oracle. It is closer to a new employee who reads quickly, follows instructions literally, and has no access to anything you did not show them. Before you trust its verdicts, you need to know what it can see, what it is inclined to reward, and how often it agrees with a careful person on the cases that matter.

Where an LLM judge is reliable

Judges perform best when the criterion is narrow and the evidence is inside the text they are grading. Whether a reply includes a case reference in the required format, whether a refund answer names the policy document it relied on, or whether an out-of-scope request was declined politely are all criteria a judge can usually check. Ask one question per judge call, give a short rubric with clear pass and fail descriptions, and include two or three examples of each. Narrow questions reduce the room for the judge to invent its own standard.

Reliability also depends on giving the judge the same evidence a reviewer would use. If the agent answered from retrieved records, the judge should receive those records alongside the answer. A judge that sees only the final text can judge tone and structure, but it cannot tell whether a number in the answer matches the source. Treat that as a design requirement, not a nice-to-have.

Where an LLM judge systematically misleads

The failures are predictable, which is why they can be checked for. The first is fluency bias. A confident, well-organized answer that is wrong often scores higher than a plain answer that is right, because the judge responds to structure and assurance. The second is length bias: longer answers look more thorough, so they pass criteria they have not earned. The third is self-preference. A judge tends to favor text that resembles its own writing, so a judge from the same model family as the agent can approve mistakes the agent and the judge both find natural.

The fourth failure is context blindness, already mentioned above, and the fifth is rubric drift. A vague criterion such as "is the answer helpful" lets the judge set its own threshold, and that threshold can change when the judge model is updated. The sixth is a ceiling effect: when almost every output passes, the eval stops separating good behavior from poor behavior, and the scores look reassuring while telling you little. Finally, agent output is not always neutral text. Documents and messages the agent processes can contain instructions aimed at whatever reads them, including a judge. The prompt injection guidance for business documents covers how to limit that exposure.

A judge that always agrees with itself has not been tested. It has only been repeated.

How to spot-check the judge itself

Spot-checking the judge is a separate activity from spot-checking the agent, and it should be done on a schedule. The steps below are a practical sequence for a team without a dedicated evaluation group.

• Draw a fixed sample of outputs that covers normal cases, edge cases, and recent failures. Keep this sample stable so results can be compared over time.
• Have two people label each output on the same rubric, without seeing the judge's score. Where they disagree, settle the label by discussion and write down the reason.
• Compare the judge's verdicts with the human labels, and read every case where they differ. Do not rely only on an overall agreement figure, because a high number can hide a single systematic error.
• Sort each disagreement by cause: unclear rubric, missing context for the judge, style preference, or a mistake by a human labeler.
• Fix the cause. Rewrite the criterion, supply the source record, or move that criterion out of judge scope and into human review.
• Repeat the check after any change to the agent, the judge model, or the rubric, and on a regular cadence after launch.

The golden test set approach is a good foundation for the fixed sample, because the same reference cases can measure both the agent and the judge.

Illustrative scenario

Illustrative scenario: a support agent drafts answers about refund windows, and a judge scores each draft for policy accuracy. Over several weeks the judge keeps giving high marks to polite drafts, including some that state the wrong refund window. A spot check shows the judge never received the policy excerpt, only the draft, so it judged the wording and not the facts. The team gives the judge the excerpt, adds a rule that any stated deadline must match the excerpt word for word, and re-labels a sample. The judge's scores then fall on the drafts that were wrong, which is the signal the team needed from the start. The agent did not change. The instrument did.

When to rely on a judge, and when to keep a person decide

Use this guide to decide where a judge belongs in your workflow.

• Use a judge when the criterion can be checked against a visible source or a short, example-backed rubric.
• Use a judge with caution when the answer depends on specialist knowledge that the judge may not hold reliably.
• Keep a person in the decision when the output commits money, creates a legal or contractual statement, or gives medical or safety guidance.
• Do not let a judge from the same model family be the only grader for high-stakes criteria.
• Route low-confidence verdicts and disagreements to a reviewer, using a clear handoff such as the one described in human escalation for AI agents.

The NIST AI Risk Management Framework is a useful reference for documenting these choices, because it frames measurement and oversight as ongoing activities rather than a single test before launch.

How Be Human handles this in managed operations

Be Human designs and operates AI agents with defined memory, tools, permissions, and approval boundaries. In a managed engagement, the judge rubric is written alongside the agent's success criterion, so both describe the same outcome. The judge is checked against a human-labeled sample before its scores influence any decision, and disagreement cases are reviewed as part of normal operations. Monitoring, permission review, and integration maintenance run on the same calendar as the evaluations, so a judge that drifts is caught alongside the agent it grades.

Keeping the judge honest after launch

A judge that passed its first check can still drift, because the agent's inputs change, the judge model is updated, and the rubric is reinterpreted by people who were not in the original discussion. Schedule the spot check as a standing task, record which version of the rubric and judge produced each score, and treat a sudden shift in scores as a question to investigate before celebrating it. A weekly maintenance routine is a practical place to keep this check visible. If you are not yet sure which of your agent's tasks are worth grading automatically, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.

Frequently asked questions

Can an LLM judge replace human reviewers? It can take over the repetitive part of quality review, but it should not replace human judgment on the criteria where errors are costly. Keep people responsible for the labeled sample, for disagreements, and for any decision with financial, legal, or safety consequences. The judge then handles volume while people keep the standard honest.

How often should I spot-check a judge? Check it before the judge influences any decision, then repeat the check after every change to the agent, the judge model, or the rubric. Between changes, a regular cadence matched to your risk level is reasonable, with more frequent checks for high-stakes tasks. Keep the sample fixed so that results can be compared over time.

What is the most common reason an LLM judge gives misleading scores? In most teams the cause is a vague rubric combined with missing context. When the judge does not see the source record and the criterion asks whether an answer is good, it tends to reward fluent, confident text. Giving the judge the evidence and writing criteria as observable checks removes much of that risk.

Should the judge and the agent use the same model? Using the same model family for both can introduce shared blind spots, because each may accept the same kind of error as reasonable. For high-stakes criteria, use a different model, a stricter code-based check where possible, or a human reviewer for the final decision. Document the choice so that a future team understands why it was made.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.