Agent Observability for Superintelligence: What to Log From Day One

Agent observability is the practice of recording enough about what an AI agent does, in order and with the context behind each step, that a person can later reconstruct any run. For a business, the minimum set is the run trace, every tool call with its permission and result, every approval request and its outcome, and cost per run and per completed task. Start recording these before the agent touches anything that matters, because evidence cannot be rebuilt from memory once the work is done.

Key takeaways

• Give every run one trace identifier that links the trigger, model calls, tool calls, approvals and final output.
• Record each tool call with the identity and permission it used, not only the final answer.
• Treat approvals as events with a requester, an approver, a decision and a timestamp.
• Measure cost per completed task, so spending changes can be explained by volume, retries or version changes.
• Decide retention, access and redaction before the first production run, not after the first incident.

Why logs matter more as agents act more

When an assistant only drafts text, a person reads the output and judges it. An agent is different. It reads a record, chooses a tool, drafts a request, waits for approval and then acts, and the final message shows only the last of those decisions. As capability grows, an agent can take longer paths, use more tools and handle cases nobody wrote a rule for. The company then needs evidence that does not depend on the agent explaining itself after the fact.

Observability is the discipline of making a system's behavior visible from the records it leaves behind. The OpenTelemetry observability primer separates three familiar signal types: traces, which show the path of one task; metrics, which show trends across many tasks; and logs, which hold the detail behind each step. That vocabulary transfers well to agents. Traces answer what happened in this run, metrics answer whether behavior is drifting, and logs answer why a specific step was taken.

A practical test is simple. Pick any single run and ask: what did the agent do, under which permission, on whose instruction, and at what cost? If the answer requires someone to remember the run or to search several unrelated systems, the agent is not yet observable in any useful sense.

The run trace: what one task must leave behind

Each agent run needs one identifier that ties together the request that started it, the model calls, the tool calls, the approvals and the output. Without that identifier, the company holds fragments that cannot be joined. The trace should record the trigger (which person, system or channel started the run), the agent version, the instruction set and model version in use, the memory items read by identifier, start and end times, final status, and any errors or retries.

Version information deserves special attention. A change to instructions, memory or model is a change in behavior, so it must appear in every trace, not only in a release note. This is the main reason changes to a managed agent should go through a recorded request and approval. The process is described in how to request and approve changes to a managed AI agent.

For shared field names, the OpenTelemetry semantic conventions for generative AI define attributes for the model, the operation and token usage. Using standard names means your traces can be read by people and tools outside the original build team, which matters when the team changes.

Tool calls and permissions: the record of what the agent touched

Tool calls are where an agent changes things, so they need the most careful record. For each call, log the tool name and version, the permission scope under which it ran, the identity the call was made as, the parameters, the result status, a short summary of the result, and whether a human approved it. Sensitive fields should be redacted or stored as hashes, with the redaction rule written down.

Log denied attempts as carefully as successful calls. A refused call shows what the agent tried to do, which is often the first sign that its instructions have drifted or that an input is trying to steer it. The OWASP guidance on authorization treats access decisions as something to be enforced and verified, and a denied call is evidence that the enforcement worked. The OWASP cheat sheet on prompt injection prevention adds a related point: content from outside the company, such as a web page or an incoming email, can carry instructions. Record which external input influenced a tool call, so the trail shows where a surprising action came from.

Illustrative scenario: a support agent may read order records and draft refund requests, but issuing a refund requires a manager's approval. The trace for one customer shows a read call, a draft call, an approval request with the manager's identifier, and a refund call made under the manager's identity after approval. If a refund call ever appears under the agent's own identity, the permission model is wrong, and the log shows exactly where to look.

Approvals: proving who allowed what

An approval is an event, not a checkbox. Each approval record should contain a request identifier, the action in plain language and in its exact parameters, the policy rule that required the approval, the approver's identity, the decision (approved, rejected or modified), the timestamp, the time taken to decide, and the action that executed afterward. Record approvals that expired without a decision too, because a growing backlog of unanswered requests is an operational signal in its own right.

Bind each approval to its parameters. An approval given for a refund of one amount should not silently cover a different amount or a different customer. If the agent proposes a modified version, the modified version needs its own approval. This is the point where many teams discover that their approval step records a yes but not what was yes to.

Approval records are also the evidence for periodic reviews. When someone asks whether the agent still has the right to take an action, the answer should come from these records, not from memory of who once agreed to something.

Cost and usage signals that stay readable

Cost is the signal most likely to surprise a business, and the one most often measured wrongly. Track tokens in and out per model call, the number of model calls and tool calls per run, retries, and cache activity where the provider reports it. Prompt caching, for example, can change the cost of repeated context, so the trace should show whether a cache was used for a given call.

The FinOps Foundation's guidance on unit economics points to the right unit of measure: cost per unit of business output, not total spend. Total spend rises when an agent succeeds more often, which is good, so a rising bill on its own tells you little. Cost per completed task tells you whether the agent is becoming more or less efficient at the work itself.

Segment these figures by agent, by workflow and by version. Alert on a sudden change in cost per run rather than on a monthly total, because a sudden change is usually a bug, a retry loop or a changed instruction. Keep the thresholds in the operating notes so the people reading the alert know what normal looks like.

A day-one logging plan

The following sequence can be applied to any first agent before it handles real work.

• Name the agent, its business owner and the single task it performs, and decide the trace identifier scheme.
• Write the event schema for runs, tool calls, approvals and cost before building the agent, so the instrumentation is designed rather than added later.
• Decide what is stored in full, what is hashed or redacted, and what is never stored, including secrets and unnecessary personal data.
• Set retention and access: who can read traces, how long they are kept, and where they are stored.
• Run a supervised task and reconstruct it using only the logs. If a step cannot be reconstructed, fix the schema before going live.
• Set two or three alerts from the start: a spike in denied calls, an approval backlog, and a change in cost per completed task beyond an agreed band.
• Schedule the first log review with the business owner for a fixed date, and record what was decided.

The NIST AI Risk Management Framework treats this kind of traceability as part of managing AI risk across the system's lifecycle, not as an afterthought for the engineering team.

How Be Human handles this: in a managed engagement, Be Human defines the event schema during the first engagement, instruments the agent so that each run, tool call, approval and cost signal is recorded under one identifier, and keeps those records under the same access rules as the agent's permissions. Monitoring reviews the signals on a recurring basis, permission reviews use the approval records as evidence, and incident response starts from the trace rather than from recollection. The scope stays deliberately small: one role, its memory and tools, and a success criterion the logs can actually measure.

Who reads the logs, and how often

Logs that nobody reads are a storage cost with a compliance label. Name one owner for each agent who reads exceptions every week, reviews cost per completed task every month, and reviews permissions every quarter. The permission review should use the approval and denied-call records as its starting evidence, following the steps in how to run a permission review for your AI agents.

The owner does not need to read every trace. They need to know what a normal run looks like, which signals mean something has changed, and who to call when the signals move. If you are unsure which process should get an agent first, and what its trail should show on day one, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

Do we need a dedicated AI observability product, or can our existing logging tools do the job? Existing logging and tracing tools can carry most of the work if the agent's events are structured with consistent identifiers and fields. The hard part is the schema and the discipline of recording approvals, permissions and versions, which no tool can decide for you. Choose tools after you have written the schema, not before.

Should we store full prompts and model outputs? Store enough to reconstruct a decision, but not everything by default. Full content can contain personal data, secrets or confidential know-how, so decide per field what is stored in full, what is summarized, and what is hashed. Write those rules down and review them with whoever is responsible for data protection.

How long should agent logs be kept? The retention period should follow your legal obligations, your contracts and the time it takes to investigate a complaint or incident in your sector. Keep approval and permission records for at least as long as the business decisions they support. Shorter retention for detailed content and longer retention for summary and approval records is a common and defensible split.

What is the first alert to set up? Start with an alert on denied tool calls, because it reveals both misconfigured permissions and possible attempts to steer the agent. Add an alert on approval requests that stay unanswered past an agreed time. A cost alert based on cost per completed task comes next, once you have a few weeks of normal behavior to compare against.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.