Skip to content
be-human.ai
← Back to the journal

Harness fundamentals

Harness Logging Levels for AI Agents: What to Record and What to Skip

Set three agent harness logging levels, default, debug and incident, to record enough to review decisions while storing as little personal data as possible.

Be Human typographic cover with the title: Harness Logging Levels for AI Agents: What to Record and What to Skip
Conceptual illustration created with AI · Be Human

Agent harness logging levels are a small, fixed set of record types that decide what an AI agent writes down at each stage of its work. A default level keeps the minimum needed to reconstruct routine decisions, a debug level adds detail for a limited window when someone is building or testing, and an incident level captures a narrow, reviewed set of fields around a problem. The aim is to review what the agent did without turning the log into a second copy of every customer message.

Key takeaways

  • Define three levels, default, debug and incident, and write down who may switch each one on and for how long.
  • Record decisions, tool calls, approvals and outcomes by default, and leave full message bodies out unless a level requires them.
  • Redact or hash personal data at the harness boundary, before anything reaches storage.
  • Set a retention rule for each level and delete debug output on a schedule you can name in one sentence.
  • Test the logs with a replay exercise before a live incident forces the question.

Why agent logs need levels at all

A single logging setting usually fails in one of two directions. If it is too quiet, nobody can explain why an agent sent a quote to the wrong contact or skipped a refund check. If it is too loud, the log fills with names, phone numbers and free-text complaints that nobody needed, and that data now sits in a place with its own access rules, its own retention and its own risk. Levels solve this by making the amount of recording a decision instead of a default that nobody revisits.

The useful question is not "what can we log?" but "what would a reviewer need to answer for each kind of event?" A reviewer asking why an agent escalated a case needs the decision and the evidence it relied on. A reviewer chasing a timeout needs timing and the tool that stalled. Neither usually needs the full text of the customer's email. Writing that distinction down is the real work of a logging policy.

The NIST AI Risk Management Framework treats documentation and monitoring of AI systems as part of managing their risks, and the OpenTelemetry concept guide describes telemetry as data collected to understand what a system is doing. Both point toward a deliberate choice of what to collect, not a habit of collecting everything.

What the default level should record

The default level runs on every task, so it has to be cheap, stable and safe to keep. It should answer what happened, in what order, under whose authority, and with what result. For most business agents, that means the following fields.

  • A task identifier and the version of the prompt, tool schemas and model in use at the time of the run.
  • The trigger: who or what started the run, and the time it started and ended.
  • Each tool call by name, with the parameters reduced to non-personal fields such as record identifiers and categories.
  • Each approval request, the approver, and the decision recorded, including rejections.
  • The stop reason, such as completed, escalated, timed out or cancelled by a person.
  • A short outcome summary written by the harness, not free text copied from the model output.

Notice what is missing. Message bodies, attachments, full customer records and raw model reasoning are absent. Those items are usually the most sensitive and the least necessary for a first review. If the default level already answers "what did the agent decide and why was it allowed to," it is doing its job.

When debug logging is allowed, and how to limit it

Debug logging exists for building and fixing agents. It can capture full inputs and outputs, intermediate reasoning traces and raw tool responses. Because it is the most revealing level, it needs the strictest switch. Treat it as a temporary permission, not a second mode the team leaves on.

  • Name one owner who can enable debug logging for one agent, and require a written reason in the change record.
  • Scope it to a test environment or to a filtered slice of traffic, never to all live tasks at once.
  • Set an automatic expiry so the level turns itself off, and make the expiry visible in the harness status.
  • Redact personal fields even at debug level, keeping structure and content categories so the bug can still be reproduced.
  • Delete debug output on the stated schedule and record that the deletion happened.
  • Review at the end of each debug window what was captured and whether the same question could have been answered at the default level.

Teams often find that most debugging questions are answered by a better default level, not by more data. If you find yourself enabling debug logging every week, the default fields are probably missing something you need.

What incident logging captures

Incident logging is narrower and time-bound. It starts when a person declares an incident, and it captures a defined set of fields for the affected agent and the affected tasks. Its purpose is to preserve evidence before it changes, so the team can reconstruct the sequence of decisions and the state of each tool at that moment.

Illustrative scenario: a small operations team runs an agent that drafts replies to supplier questions and proposes purchase order changes. One morning a reviewer notices that several proposed changes used an outdated unit price. The team declares an incident. The harness switches the affected agent to incident logging, which keeps the tool calls that read the price list, the version of the tool schema, the approval records and the stop reasons, while personal contact details stay redacted. The team can then see that the price tool was reading from a cached source, and they pause the agent's write permission until the source is fixed. Nobody needed the full supplier emails to find the cause.

Incident logs should be read-restricted, because they are the evidence the team will rely on later. Record who accessed them, and decide in advance when the incident level closes and who signs off on that closure.

How to keep personal data out of the logs

Personal data in logs is a design problem before it is a legal one. The safest approach is to make sure the harness never writes the field in the first place. Use this checklist when you define each level.

  • List every field the harness can write, and mark each one as structural, categorical, personal or sensitive.
  • Write structural and categorical fields at every level. Hash or drop personal fields at the default level.
  • Redact at the boundary: the harness should clean tool responses before they enter the log, not rely on someone cleaning the log later.
  • Keep identifiers that are internal, such as a record reference, instead of names or contact details where a reviewer can still find the record.
  • Confirm that log access follows the same permission rules as the systems the agent reads from, and review that list when roles change.
  • Include the logs in your data inventory and in the retention rules you already apply to operational records.

The OWASP guidance on authorization makes a related point about access control: a person should see only what their role requires, and that applies to observability tools as much as to the agent itself. A log that anyone in the company can open is a copy of your business data with weaker protection.

How Be Human manages logging levels in managed operations

Be Human designs logging levels as part of the harness for each agent, alongside its memory, tools, permissions and approval boundaries. In managed operations, the team defines the three levels for the specific role, sets the redaction rules at the boundary, writes the retention schedule and the debug expiry, and tests a replay of a sample bad run before going live. Monitoring and permission reviews then check that the levels still match what the agent does, and that debug switches are not left on. Changes to the levels go through the same versioned release process as prompts and tool schemas, so the record of what was captured is itself reviewable.

Set your first logging policy this month

Start with one agent and one page. Write the default fields, the single owner for debug switches, the incident declaration rule and the retention statement, then run one replay of a past task to see whether the default level answers your questions. Once that works, extend the same page to the next agent. Taking the free Be Human AI diagnostic is a practical way to find the first task worth handing to an agent, and the logging policy for that task is a good place to begin.

Frequently asked questions

How many logging levels does an agent harness need? Three is enough for most business agents: default, debug and incident. Default runs on every task and records structure and decisions. Debug is a time-limited permission for building and testing, and incident is a reviewed, narrow capture around a declared problem.

Should we log full customer messages so we can investigate problems? Usually not by default. Most investigations depend on the decision, the tool calls, the approvals and the stop reason, which can be recorded without the message text. If a specific investigation needs message content, enable it at debug or incident level for the affected tasks, with redaction and a fixed expiry.

How long should debug and incident logs be kept? Set a retention period for each level in your own policy, based on your legal obligations and internal review needs, and delete data when that period ends. Debug output should be shorter than default logs because it holds more detail, and incident records should close with a written sign-off that states when they will be removed.

Who should be allowed to change logging levels? One named owner should approve changes, and the change should be recorded with a reason and an expiry where applicable. Access to the logs themselves should follow the same role rules as the source systems, so that people who could not read a customer record directly cannot read it through the agent's logs.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →