Skip to content
be-human.ai
← Back to the journal

Testing & reliability

An AI Agent Observability Dashboard: Five Panels to Read Every Week

An AI agent observability dashboard with five panels to read weekly: failed tool calls, approval waits, cost per completed task and quality.

Be Human typographic cover with the title: An AI Agent Observability Dashboard: Five Panels to Read Every Week
Conceptual illustration created with AI · Be Human

An AI agent observability dashboard is a short set of panels that shows what an agent attempted, what it cost, where it waited for a person and where it failed, so an owner can judge its work without reading every transcript. For a business agent, five panels cover most of what needs attention each week: failed tool calls, approval wait times, cost per completed task, escalations to people and a sample of output quality. The panels only help if they come from events the agent already emits, and if someone reads them on the same day every week.

Key takeaways

  • Five weekly panels cover most of what a business agent needs: failed tool calls, approval waits, cost per completed task, escalations and sampled quality.
  • Measure cost per completed task rather than cost per message, so retries and rework appear in the number.
  • A growing approval queue is a reliability signal, not only a staffing question.
  • Every panel needs a named owner, a threshold and a written action for when the threshold is crossed.
  • Build the events before the dashboard, because no panel can show what the agent never recorded.

Why an owner needs a dashboard, not a transcript archive

Transcripts are useful after something goes wrong. They are poor at answering the questions an operations leader asks on a Monday morning: is the agent still doing its job, is it getting more expensive, and did anything change quietly? A dashboard compresses the week into a few readings that can be compared with the previous week. It does not replace review of individual outputs. It tells you where to look first.

The discipline behind this comes from systems engineering. The OpenTelemetry project describes observability as understanding a system's internal state from the signals it emits, such as logs, metrics and traces. The same idea applies to agents with one important difference: part of an agent's behavior is decided in language. The trace therefore has to record more than timing and errors. It should also show which tool the agent chose, what it was permitted to do and the point at which it stopped to ask a person.

The five panels to read every week

  • Failed tool calls, grouped by tool and by error type
  • Approval wait time, from the moment the agent asks to the moment a person answers
  • Cost per completed task, including retries and reruns
  • Escalations to a human, with a reason code on each one
  • Sampled output quality, scored from a fixed number of finished tasks

Failed tool calls come first because they are the most direct evidence that something is broken. Read the count and the shape together. Repeated failures on one tool with one error type usually point to a narrow problem, such as an expired credential or a renamed field. Failures spread across every tool point upstream, to network trouble or a change in a platform the agent depends on. Grouping by error type turns a raw count into a direction.

Approval wait time matters because agents that need sign-off before a payment, a customer email or a record change will sit idle when nobody answers. Track the typical wait and the longest wait separately, since customers feel the longest waits most. If the queue grows, decide whether the approver is overloaded, the approval rule is too broad, or the agent is asking about things it should resolve on its own.

Cost per completed task is the panel most owners are tempted to skip, and it is the one that tells you whether the agent is worth running. Divide the spend in the period by tasks that reached their intended end state, not by messages or model calls. An agent that retries a failing step several times before it succeeds looks productive on message counts and expensive on this panel. The FinOps Foundation frames unit economics the same way: cost is most useful when it is tied to a unit of business output.

Escalations show where the agent's judgment ends. Each handoff should carry a reason code such as missing data, policy conflict, unavailable tool or a customer in distress. When one reason rises, you know what to fix. Keep this panel honest. An agent that escalates everything is not safe, it is unhelpful, and it moves its cost onto your staff without showing it.

Sampled quality is the panel that checks whether finished work is actually right. Pull the same number of completed tasks each week and score them against a short rubric covering facts, tool use, tone and the quality of the handoff. Keep the sample size fixed. If the sample changes every week, the trend line reflects sampling noise rather than the agent's performance.

How to set thresholds without guessing

Thresholds that are invented on day one tend to be ignored by week three. A simpler method works better:

  • Run the agent for two weeks in its current setup and record each panel without sending alerts.
  • Write down what a normal week looks like for each panel, in plain words and with the values you observed.
  • Set a warning threshold at the point where a change would alter a decision, not where a chart merely moves.
  • Name one owner per panel, the person who reads it and acts on it.
  • Write the action for each threshold in one sentence, for example: pause new customer replies and review the last ten drafts.
  • Review thresholds once a month, and change them only with a short note in the change record.

The last step matters more than it looks. Thresholds that change every week hide drift, and a slow decline in quality is easy to miss when the target keeps moving. Changes should be deliberate and visible to the people who rely on the numbers.

What to log before you build the dashboard

A dashboard is only as reliable as the events underneath it. Each agent action needs a task identifier that survives retries, a tool name, a result status, timestamps for the request and the response, a cost attribute, an approval state and a reason code when the agent hands work to a person. The OpenTelemetry semantic conventions for generative AI define shared attribute names for model activity, which makes it easier to keep the same panels when a model or vendor changes later. The practical list of events is covered in more detail in AI agent observability: what should you actually log?.

Illustrative scenario, not a client case: a small import business uses an agent to draft replies to supplier emails and to create purchase orders in its accounting system after a person approves them. In the first week, the failed tool call panel shows repeated errors on one accounting endpoint, clustered in the afternoon. The approval wait panel shows that orders sit for most of a working day because approvals arrive in batches. Cost per completed task is higher than expected because the agent keeps retrying the endpoint and the retries pile up. The owner does not change the model. They ask the integration owner to check the rate limits, move approvals into two fixed daily windows with a reminder, and cap retries with anything left over going to a human queue. The following week's panels show whether those changes held.

How Be Human approaches this in a managed engagement

Be Human defines the events, the owner for each panel and the approval boundaries before the first reading is taken, so the dashboard reflects the work the agent is actually authorized to do. Managed operations then cover monitoring, recurring evaluations, integration maintenance, permission review and incident response, which keeps the panels read, the thresholds reviewed and every change recorded in a change log the owner can follow. A first engagement scopes one useful role, its memory and tools, and one observable success criterion, which is often the cost per completed task or the approval wait time for that role.

A weekly reading routine that takes less than an hour

Choose one fixed day and one fixed time for the review. Read the panels in the same order each week: failures, waits, cost, escalations, then quality. For each panel, write one of three words in the review notes: unchanged, watch or act. A panel marked act gets a named owner and a deadline. A panel marked watch is read again the following week with the same sample size and the same definitions. Keep the notes short enough that someone can scan the last month in a few minutes. The broader rhythm for checking an agent is described in a weekly maintenance checklist for managed AI agents, and the cost reading is easier to act on when you follow how to measure cost per completed task.

If you are not sure which role deserves this kind of observation first, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.

Frequently asked questions

How many panels does a business agent really need? Five is a practical starting point for most roles, because they cover failure, waiting, cost, handoffs and quality. Adding more panels before the first five have owners and thresholds usually makes the review slower without making decisions better. Add a panel only when a recurring question cannot be answered by the existing five.

Who should read the dashboard each week? The reader should be the person who can act on what they see, which is usually an operations leader or the owner of the process the agent supports. A technical person should prepare the data and explain any anomaly, but the decision to pause, adjust or expand the agent belongs to the business owner. Shared visibility helps, but a dashboard that nobody is accountable for becomes a screenshot in a slide deck.

Can a non-specialist set this up? A non-specialist can define the questions, the owners and the thresholds, which is the part that most often goes missing. The event logging and the integration work usually need someone who can read application logs and work with the systems the agent calls. Many teams split the work this way: the business owner specifies the decisions, and an operator or partner builds and maintains the instrumentation.

What if the panels disagree with what the team reports? Treat the disagreement as information, not as a verdict. Check whether the panel is counting the right thing, for example completed tasks rather than attempted ones, and whether the team is describing the agent's best day or its typical week. Then compare a small sample of tasks by hand. When the panel is right, change the team's expectations. When the team is right, fix the event definition so the next reading means what everyone thinks it means.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →