LLM Judges for Agent Evals: Calibrate Before You Trust Them

An LLM judge is a language model that scores another model's output against a written rubric, such as whether an agent's reply promises only what the policy allows. Judges let a team review far more agent output than people can read, but they are only useful after their labels have been measured against careful human reviews on your own tasks. Use a judge to triage and surface problems, and never as the sole approver of actions that touch money, customer records or commitments made in your company's name.

Key takeaways

• A judge is worth using for volume only after its labels agree with trained human reviewers on your real tasks.
• Write a rubric with observable criteria first, then confirm that two reviewers can apply it the same way.
• Calibrate on a sample that includes boundary cases and planted errors, not only easy outputs.
• Re-check agreement after every prompt change, model update or shift in incoming work.
• Keep a human in the decision for any action that moves money, changes records or sends commitments outside the company.

What an LLM judge is good for, and what it is not

A judge is a second model call that reads an input, an agent's output and a rubric, then returns a label such as pass, fail or needs review. Teams use judges because human review does not scale to every reply, draft or tool call an agent produces. A judge can read a full day of transcripts and flag the ones where the agent offered a discount the policy does not allow, or where an answer makes a claim without the source it was supposed to cite.

What a judge cannot do is know whether it is right. It reads the rubric through its own assumptions, it can be swayed by fluent wording, and a confident explanation placed next to a wrong fact often scores well. That is why the judge should be treated as a measurement instrument with its own error pattern, not as an oracle. Anthropic's engineering guidance on evals for agents makes a similar point: graders must be checked against the outcomes that matter, and the test design should reflect real tasks rather than tidy demonstrations.

Write a rubric a person could also apply

Most weak judges trace back to weak rubrics. 'Is this response helpful?' is not a rubric, because two reviewers will answer it differently and the model will invent its own meaning. A usable rubric names the observable condition for each label. For example, an appointment confirmation passes only if it contains the date, time, location and reschedule link from the approved template, and it makes no commitment outside the service catalog.

A practical rubric usually has these properties:

• One criterion per check, so that a failure has one identifiable cause.
• Evidence the reviewer can point to in the transcript or the record.
• An explicit 'cannot tell' label, because uncertain cases are information, not noise.
• Two or three borderline examples for each label, written down with the reason for the decision.
• A plain-language statement, approved by the business owner, of what counts as a failure.

Test the rubric before you test the judge. Give two people the same twenty outputs and compare their labels. If they disagree often, the rubric is ambiguous, and no model will repair that ambiguity.

How do you calibrate a judge against human reviews?

Calibration means measuring how closely the judge's labels match trusted human labels on the same items, then deciding whether that match is good enough for the specific job. The goal is not a perfect score. The goal is knowing where the judge is reliable, where it is weak, and what happens to the cases it cannot handle.

• Pull a sample of real outputs from recent weeks, stratified so that common cases, rare cases and changes from recent updates are all represented.
• Have two trained reviewers label each item without seeing the judge's output, then settle disagreements in a short session that updates the rubric.
• Run the judge on the same items with the same rubric, and record its label, its reasoning if available, and the prompt and model versions used.
• Build a per-criterion comparison that separates false passes (the judge approved what a reviewer rejected) from false failures (the judge rejected something acceptable).
• Set the tolerance by consequence. A false pass on a refund matters far more than a false failure that sends an item to a review queue.
• Repeat the exercise on a fresh sample before relying on the judge, because the first sample influenced the rubric and may flatter the result.

Pay particular attention to false passes. A harsh judge wastes reviewer time, which is annoying but recoverable. A lenient judge lets mistakes continue downstream unnoticed, which is the outcome the evaluation was built to prevent.

Which failure patterns should you expect?

Judges have recognizable habits. Many favor longer answers, reward confident tone, and prefer outputs that resemble their own writing style. When two outputs are compared side by side, the order in which they appear can shift the verdict. Judges also miss errors that need domain knowledge, such as a part number that is almost right, a tax rule that changed last year, or a date that is valid in format but wrong for the contract.

The countermeasures are simple to describe and easy to skip. Randomize order in pairwise comparisons. Separate the factual check from the style check so that polish cannot hide an error. Include deliberately wrong outputs in the test set, and confirm the judge catches them. A judge that passes every planted error has not been tested hard enough.

There is also a security angle. The text the judge reads is produced by an agent that may have processed customer emails or uploaded documents, and that text can contain instructions aimed at the judge, such as a line telling it to mark the item as approved. The OWASP guidance on prompt injection prevention explains why content from documents and users should be handled as data to examine, not as instructions to follow. Judge prompts should say so explicitly, and the judge should not hold permissions that let its verdict act on its own.

Where should a judge never be the final check?

A judge can triage. It should not be the last word when the outcome is costly, hard to reverse or regulated. Use this decision guide before you allow a judge's label to change anything:

• Money leaves the company, or a refund, credit or payment is approved: a person approves, and the judge only orders the queue.
• A customer record, contract, employment, medical or legal item changes: a named reviewer confirms before the change is committed.
• The agent is about to contact a customer, supplier or public body in the company's name on a topic it has not handled before.
• Agreement with human reviewers has not been measured for this task type, or has dropped since the last model change.
• The output depends on facts the judge cannot see in its input, such as a delivery status that lives in a carrier system the judge never queries.
• A failure would be hard to notice later, because nobody reads the outputs that the judge passes.

Where the judge is allowed to decide, make the decision reversible and logged. Store the judge's label, the rubric version, the model version and the input reference with the action, so a later review can reconstruct why an item was approved. Field design for that record is covered in the superintelligent agent audit trail fields guide.

Illustrative scenario (not a client example): a small training company lets an agent draft replies to course questions, and a judge flags replies that promise certificates the course does not issue. In calibration, the judge catches the obvious promises but passes replies that say 'completion credential', which reviewers treat as a promise. The team adds that phrase and its synonyms to the rubric, re-runs the sample, and keeps human approval for any reply that mentions credentials at all.

How Be Human runs judge calibration in managed operations

In Be Human's managed approach, a judge is introduced only after the first engagement has defined the role, the memory and tools the agent needs, the approval boundaries and an observable success criterion. The calibration sample is drawn from the client's own workflow, labeled by people the client names, and stored with versioned rubrics. Recurring evaluations compare judge labels with fresh human samples, and permission review confirms that the judge cannot approve actions it was never meant to decide. When agreement falls after a model or prompt change, the change is held or rolled back until the rubric and the judge are checked again.

Keep the judge honest after launch

Calibration is not a one-time event. Judges drift for the same reasons agents drift: prompts are edited, the underlying model is updated or retired, the business launches a new product, or the mix of incoming work moves toward cases the rubric never covered. Keep a fixed golden set of items the judge has already been checked on, add a recurring human sample, and review disagreements at least monthly. Where a real outcome exists, compare the judge's label with it, such as whether a reply it passed later produced a complaint. The golden test set approach is a practical way to keep that reference stable across changes.

Treat the judge as a component in your data and approval design, not as a side utility. Its inputs often include customer text, and its scores are read by people who then act on them. The NIST AI Risk Management Framework offers a useful structure for thinking about measurement, accountability and ongoing monitoring of AI systems, and it fits well with a judge that is part of a larger operation.

If you are not yet sure which of your agent's tasks is the right first one to measure with a judge, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Can an LLM judge replace human reviewers? No, not for decisions that carry real cost or risk. A judge can read far more output than a person and flag the items most likely to need attention, which saves reviewer time. Its labels still need to be checked against trained human reviewers on your own tasks before anyone relies on them.

How many examples do I need to calibrate a judge? There is no universal number, and it is better to plan the sample by coverage than by size. Include common cases, rare cases, recent changes and deliberately planted errors, along with the boundary cases where reviewers disagree. If a category is missing from the sample, the judge's behavior on that category is unknown.

Should the judge use the same model as the agent it grades? Using the same model family can share the same blind spots, so the agent's typical errors may look acceptable to the judge. Many teams use a different model for the judge, or at least a stricter and more concrete rubric, and then measure the result. Whatever you choose, the choice should be tested against human labels rather than assumed.

What should I do when the judge and the human reviewers disagree? Start with the rubric, because the disagreement often shows an unclear criterion rather than a faulty judge. Discuss the specific items, update the rubric and examples, and then re-run the comparison on a fresh sample. Only after that should you change the model or prompt used by the judge.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.