Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Brand Voice Evals for Superintelligent Agents: Tone Tests That Work

Score AI agent replies against written brand voice rules with six tone tests, a simple pass rule, and a reviewer log owners can read and act on.

Be Human typographic cover with the title: Brand Voice Evals for Superintelligent Agents: Tone Tests That Work
Conceptual illustration created with AI · Be Human

Brand voice evals for customer agents are small, repeatable tests that score each agent reply against the company's written voice rules. A useful set has five to seven tone tests, each with a plain pass condition, run on a sample of real replies and checked by someone who owns the brand. The goal is not a perfect tone score but a record that shows when replies drift away from how the company wants to sound.

Key takeaways

  • Turn each voice rule into a statement a reviewer can answer yes or no.
  • Start with six tests or fewer, and add new ones only after a real failure shows a gap.
  • Score facts and tone separately so a polite but wrong answer still gets caught.
  • Calibrate every scorer, human or automated, on the same examples before trusting its results.
  • Review a sample each week and log every change to the rules or the agent.

Why voice needs tests, not only a style guide

A style guide tells a writer what the brand sounds like. It does not tell you whether last week's replies actually sounded that way. An agent produces text that varies from one conversation to the next, so a rule that reads clearly in a document can be applied loosely in practice. Voice evals close that gap by turning the rules into questions that a reviewer, or a scoring model checked by a reviewer, can answer on a sample of real replies.

The effort is worth it whenever customer replies carry the company's name and a technically correct answer can still sound careless, pushy or cold. That covers most customer-facing agents. It also covers the period after a prompt change, a knowledge update or a model update, when tone often shifts and nobody notices until a customer does.

Write voice rules as statements a reviewer can check

Most voice guides are full of adjectives such as warm, confident and clear. Those words are a starting point, but no reviewer can score a reply against 'warm' without a definition. Rewrite each adjective as an observable behavior. 'Warm' might become 'names the customer's order or product in the first sentence and never opens with a template apology.' 'Confident' might become 'states the next step in one sentence and does not stack several hedges.'

Keep the rule list short. A handful of specific rules works better than a long list that overlaps itself. Add a short glossary of approved and banned terms, plus two or three sample replies that show the rules in use. Those samples become the calibration set for everything that follows.

Build six tone tests for every customer reply

A practical starting set has six tests. Each one returns a simple yes or no for a single reply, which keeps reviewer disagreement low and makes trends visible over time.

  • Name and greeting: the reply addresses the customer in the form the written rules specify.
  • Plain language: no internal jargon and no phrase from the banned list.
  • Ownership without blame: the reply says what happens next and does not blame the customer or a third party.
  • Honest limits: when the agent cannot act, it says so and names the route to a person.
  • Approved terms: product, policy and sign-off names match the glossary exactly.
  • Facts before tone: a separate check confirms the content is correct, so a polite wrong answer fails.

Keep the facts check separate from the rest. If tone and accuracy are blended into one score, a friendly reply with a wrong delivery date can still pass, and that is the failure you most need to see. Tone tests then only judge replies that are already correct.

A step-by-step method for scoring and setting a pass line

  • Step 1: Pull a sample of recent replies from one intent, such as delivery questions, and include a few that customers escalated.
  • Step 2: Have two reviewers score the same small batch independently against the six tests, and note every disagreement.
  • Step 3: Rewrite any rule where the reviewers disagreed, and add a clear example reply to the calibration set.
  • Step 4: Set the pass line for each test before scoring a new batch, and send any reply that fails the facts check to a person regardless of its tone score.
  • Step 5: Rerun the same checks after each prompt change, model update or new knowledge source, and compare the results with the previous run.
  • Step 6: Log each failed reply, the test it failed and the change that followed, in a form the owner can read without opening the code.

If an automated judge scores tone, check it against the human reviewers first. A judge can reward phrasing that looks polished but does not match your rules. The guide on calibrating LLM judges for business agent evaluations covers that check in more detail.

Illustrative scenario: a delivery reply that passes the facts and fails the voice

Illustrative scenario: a shop that sells kitchen equipment uses an agent to answer delivery questions. A customer asks why an order is late. The reply gives the correct carrier status, but it opens with a stock apology, says 'unfortunately, we cannot speed this up,' and closes with 'please be patient.' The facts check passes. The ownership test and the plain language test fail, so the reply goes into the log with the rule it missed.

The fix is not a rewrite of the whole prompt. The team adds one approved example that states the carrier status, the expected date and what the customer can do if that date passes, then reruns the same sample the following week. The original failure stays in the log, so the owner can confirm that the change addressed a pattern rather than one conversation. Feedback from staff reviewers works the same way, as described in turning staff corrections into agent rules.

How Be Human runs voice checks in a managed operation

Be Human treats voice checks as part of a managed operation rather than a one-time prompt adjustment. The engagement begins by defining the agent's role, the voice rules it must follow and the approval boundaries for replies that touch refunds, complaints or legal questions. The test set is built from conversations the owner's team already accepts, so the standard comes from the business rather than from a generic template. Checks run on a recurring schedule, failed or borderline replies are routed to a named person, and each change to the rules or the agent is written into a readable change log. The approach is the same for a single agent in one language or a team of agents working in English and Portuguese.

Keep the voice tests current and make them an owner's habit

Voice drifts slowly. A new product line, a changed policy or a season of different customer questions can shift the replies an agent produces, even when no one has edited its prompt. Set a recurring review, keep the sample mixed between routine and escalated conversations, and retire any test that never fails because it no longer measures anything useful. Review the log with the person accountable for the brand as well as the operations team, since that person is the one who can say whether a reply still sounds like the company.

Take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

How many tests should a brand voice eval include? Start with five to seven tests that map to rules your team already enforces by hand. Each test needs a plain pass condition that two reviewers would apply the same way. Add a new test only after a real failure shows that the current set misses something.

Can an LLM judge score brand voice automatically? It can, but only after it has been checked against human reviewers on the same replies. Compare its scores with two people on a small batch, study the disagreements, and keep a person in the loop for any reply that touches money, complaints or legal topics.

How often should voice evals run? Run the full set after any prompt change, model update or new knowledge source, and sample recent replies on a regular schedule, weekly at first. Adjust the frequency once you see how often failures appear and how quickly they are fixed.

Who should own the voice rules? The person accountable for the brand, such as a marketing or customer leader, should own the rules and approve changes to them. The operations team runs the tests and records the results. Keep the rules in a document that every change to the agent references, so ownership is clear when a reply goes wrong.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →