Harness Health Checks for Superintelligent Agents: Hourly Signals
An hourly checklist of signals showing whether an agent harness, its tools and its model calls are working, so teams spot failures before users do.

An agent harness health check is a small set of signals collected every hour that shows whether the harness, its tools and its model calls are still completing work. The useful signals measure outcomes at each boundary: whether the agent loop is moving, whether each tool call succeeds and returns the fields the agent depends on, and whether model responses arrive on time and pass basic checks. Reviewing these hourly lets an operator notice a failing tool or a slowing model before a customer, colleague or account manager reports the problem.
Key takeaways
- Measure three layers separately: the harness loop, each tool, and each model call.
- Compare every signal with a baseline for the same hour and day, not with a single reading.
- Alert on stalled work, empty results and cost drift, not only on hard errors.
- Give every signal a named owner, a threshold and a written response.
- Tune thresholds from normal runs and failed runs, then review them whenever the harness changes.
What a harness health check covers
A harness is the layer around a model that runs the agent loop, holds memory and context, exposes tools, checks permissions and decides when to stop. A health check asks whether each of those parts is doing its job right now, which is a narrower and more useful question than whether the server is reachable. An agent can be online and still answer from stale memory, call a calendar tool that returns an empty time field, or retry the same request until its budget runs out. Those failures look like normal activity on a basic uptime dashboard, so the check needs signals that describe outcomes.
The vocabulary for this is well established. OpenTelemetry's observability concepts separate traces, metrics and logs, which answer different questions: what happened in one run, how often something happened across many runs, and what detail explains a particular failure. OpenTelemetry also publishes semantic conventions for generative AI, which give shared names to fields such as the model used and the token counts. Consistent names matter, because the same signal should mean the same thing in every report and in every team that reads it.
The hourly signals to collect
Collect these every hour for each agent role. Keep the list short enough that one person can read it in a few minutes.
- Loop progress: runs started, runs completed, runs stopped by a stop condition, and runs waiting for approval longer than the agreed window.
- Work age: the age of the oldest unfinished task and the number of tasks retried more than once.
- Tool success: success rate by tool, grouped by error type, with timeouts shown apart from rejections.
- Tool result quality: responses missing a field the agent needs, or responses cut short before the decisive part.
- Schema rejections: tool calls or model outputs refused by validation before they took effect.
- Model latency: typical and slow-end response times for each model in use.
- Model errors: rate-limit, server and timeout responses, and how often a fallback model handled a request.
- Token use: tokens per run and per task type, compared with the same hour last week.
- Permission denials: actions blocked by the approval gate, split by tool and by agent role.
- Retrieval results: memory or document lookups that returned nothing when results were expected.
- Credentials: connections close to expiry or already rejected by the provider.
Not every signal needs an alert. Most of them exist for trend review, and only a few should wake someone up. The list should follow the agent's purpose: a scheduling agent cares most about tool result quality and work age, while an agent that drafts documents cares most about schema rejections and retrieval results. If a signal never leads to a decision, remove it.
How to set thresholds without guessing
Thresholds chosen by intuition tend to fire constantly or never fire at all. A workable sequence starts with a narrow scope and builds from observed behavior.
- Pick one agent role and its two or three most important tasks, then name the tools each task needs.
- Run the role in a test environment with realistic requests until the signals stop changing shape. That shows what normal looks like.
- Record the baseline for each signal by hour of day and day of week, since load often differs across the week.
- Set a warning level that prompts a review within the working day, and a stop level that pauses the agent automatically.
- Write the response for each level: who checks, which logs they open, and what they may change without a further review.
- Link every alert to the logged run that triggered it, so a reviewer can reproduce the problem. The fields that make this possible are described in replay logs for AI agents.
Revisit the thresholds after the first few weeks of live operation. Early thresholds are a starting point, and the first real failures usually show which signal was missing.
Illustrative scenario: a tool that fails quietly
Illustrative scenario, not a client case: an agent drafts appointment confirmations for a small clinic and reads availability from a calendar tool. One afternoon the tool keeps returning successful responses, so the success rate stays at its usual level. The time zone field, however, begins arriving empty, and the validation check on the agent's output starts rejecting a growing share of confirmations. A health check that watches only success rates would show nothing unusual. A check that watches result quality and schema rejections raises a warning within the hour, the agent stops sending new confirmations, and the reviewer can compare the latest tool response with the last healthy run. The fix belongs to the tool, but the early signal came from the harness.
This is why the most valuable signals sit between layers. Success codes tell you the tool responded. Result quality tells you whether the response is still usable. For more on keeping tool outputs complete, see tool result truncation for AI agents.
Which signal means stop, and which means watch
A decision list lets the on-call person act without a debate about every reading.
- Stop and pause: write tools failing after partial completion, permission denials rising without a related change, or token use per run far above baseline while the same steps repeat.
- Watch within the working day: latency drift on one model, rising retries on one tool, or retrieval returning nothing more often than usual.
- Review at the weekly check: gradual changes in output length, occasional schema warnings that clear on retry, or a timeout count that rises but stays below the warning level.
- Escalate to the change owner: any signal that moved in the same hour as a model, prompt or tool release, since that pattern calls for a rollback decision rather than a wait.
The stop rules themselves should be written down before an incident, not during one. Stop conditions for AI agent loops covers when to halt and escalate, and token and step budgets with alerts shows how cost signals connect to those halts.
How managed operations keep the checks current
Be Human designs and operates personalized AI infrastructure for companies, so health checks are part of the first engagement rather than an afterthought. The engagement defines the agent's role, the memory and tools it needs, its approval boundaries and the success criterion each signal serves. Managed operations then run monitoring, recurring evaluations, permission review and incident response against those signals, and thresholds are updated when a model, prompt or tool changes. The work is delivered remotely, in English or Portuguese, and the signals are reviewed with the people who own the business process, not only with the engineers who built the harness.
Where to start
If you are unsure which of these signals matters most for your first agent, take the free Be Human AI diagnostic to find the first task worth handing to an agent.
Frequently asked questions
How often should an agent harness health check run? Hourly collection suits most business agents because it catches drift within a working day without flooding the team with alerts. Some signals, such as stopped runs and permission denials, may need faster handling, while others are better reviewed weekly. The cadence should match how quickly a failure would cause real harm to a customer, a record or a payment.
Which signal should I set up first? Start with tool success and tool result quality, because a tool can return success while omitting the fields the agent needs to act. Add loop progress and work age next, so you know when the agent has stopped making progress without raising an error. Model latency and token use can follow once the first two signals are reliable.
Do I need a separate monitoring product? Not necessarily. Many teams begin with run logs that record each agent run, tool call and model call with consistent field names, then compute the hourly figures from those logs. A dedicated monitoring tool becomes useful when the volume of runs or the number of agents makes manual review impractical. The tool matters less than whether every signal has a named owner and a written response.
Can a health check replace testing before a change? No. A health check shows what is happening in live operation, while tests and evaluations show whether a change behaves correctly before it ships. Anthropic's guidance on evals for AI agents is a useful reference for building those pre-release tests. Use both: test the change in a staging environment, then watch the hourly signals after release to confirm that real traffic behaves as the tests predicted.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


