AI Harness Requirements: A Checklist for Judging Any Agent Setup
What an AI harness must control, log and allow before a business trusts an agent with real work, plus a checklist to score any agent setup.

An AI harness is the layer that wraps a model and decides what an agent may see, do, record and stop. To judge any agent setup, check that the harness controls identity, memory, tools, permissions, spending and stop conditions, logs every model call and tool action in a form a reviewer can read, and lets people approve, pause, roll back and export work at defined points. The checklist below turns those requirements into questions you can ask a vendor, an internal team or your own draft.
Key takeaways
- A harness is judged by what it controls, records and lets people change, not by which model sits inside it.
- Every tool the agent can call needs a named scope, a read, draft or act level, and an owner who can revoke it.
- Logs must answer four questions for any task: who asked, what the agent saw, what it did, and why it stopped.
- Approval points, pause, resume and rollback should be tested before the first live run, not after the first incident.
- Ask for evidence: a sample trace, a deliberate failure test and a revocation demonstration, not only a diagram.
What a harness must control
Control means the harness decides, not the model. A model can propose the next step, but the harness determines which tools exist, which records are visible, how far an action can go and when the work must stop. If the term is new to your team, what an AI harness is covers the basic layer. When you review a setup, look for these controls:
- Identity: the agent has a fixed name, a stated role and the business it acts for, written in a file people can read and change.
- Memory: the agent's record shows which data it may read, which it may write, and how long its notes persist.
- Tools: there is an explicit list of callable tools, each with input limits and a description of what it does.
- Permissions: read, draft and act are separated for every tool, and act permissions are the smallest the task needs.
- Model choice: each task type maps to a named model, and a change to that mapping requires a named approver.
- Limits: caps on steps, retries and spend per task stop runaway work before it reaches anyone's inbox or ledger.
- Stop conditions: the agent halts and hands off when inputs are missing, a tool fails twice or a request falls outside its role.
A control counts only if it can be changed without rebuilding the agent. Ask who can revoke a tool, shorten a budget or move a task to a fallback model, and how long that takes on a normal working day.
What a harness must log
Logs are the only way a non-engineer can review what an agent did last week. The useful model comes from observability practice: each task produces a trace made of steps, and each step carries its inputs, outputs, timing and errors. The OpenTelemetry observability primer describes this idea in general terms, and it applies to agents as well as to ordinary web services. A setup passes this section when any single task can be reconstructed from start to finish without asking the person who built it. The record should include:
- The request that started the task, and the person or system that sent it.
- The model and version used at each step, so a later change can be traced.
- Each tool call with its arguments, a summary of the result and a timestamp.
- Every memory read and write, including what was removed or overwritten.
- Each approval requested, granted or refused, with the name of the approver.
- Errors, retries and the reason for every stop or handoff.
- The cost of the task in the unit your finance team already uses.
Illustrative scenario: an office manager finds two reminder messages sent to the same customer on the same afternoon. If the log shows one request, one tool call and a retry after a timeout, the fix is a retry rule with a duplicate check. If the log shows two requests from two intake sources, the fix belongs in intake. Without the record, the team argues from memory, and the same error returns the following week.
What a harness must allow
Allowing is the requirement most setups skip. A harness can be tightly controlled and still fail if people cannot intervene quickly or recover cleanly. It should allow the following, and each item should be tested before the agent touches live work:
- Approval at named points, such as sending an external message, changing a customer record or committing money.
- Pausing one task or the whole agent without losing its queue, its notes or its place in the work.
- A stop control that halts actions while keeping logs intact, as explained in the guide to a kill switch for a business agent.
- Resuming unfinished work from a saved point rather than from the first step, which matters for any task that touches external systems. The checkpoints and resume guide describes the pattern.
- Rolling back a prompt, a tool setting or a model change to the last version that passed evaluation.
- Exporting the full record of a task in a form a reviewer can read without special software.
The question to ask is simple: if the agent misbehaves at four in the afternoon, who acts, with what button or command, and what state is left behind?
How to score an agent setup in one review
You can run this review in an afternoon with the people who own the process. You do not need an engineering department to ask every question, but you need someone who can answer for the systems the agent touches. Work through these steps in order:
- Step one: write the role in one sentence and name the business outcome it serves.
- Step two: list every tool and record the agent can reach, and mark each one as read, draft or act.
- Step three: for every act permission, name the human who approves it and the condition that triggers approval.
- Step four: request one sample task trace and confirm that every item in the logging section is visible.
- Step five: trigger a deliberate failure, such as a tool timeout or a missing field, and confirm the agent stops, logs the reason and waits.
- Step six: run a pause, a resume and a rollback while someone watches the log.
- Step seven: write the success criterion in terms a reviewer can check against the trace, not in terms of how the output feels.
A setup that fails step five or step six is not ready for live work, whatever its demonstration looked like. Record the result of each step so the next review starts from evidence.
Where security and data rules fit
Security is a harness requirement, not a separate project that happens later. Prompt injection, where text inside a document, email or web page tries to override the agent's instructions, is covered in the OWASP guidance for LLM applications, and the risk should be assumed for any agent that reads outside content. The OWASP authorization guidance applies directly to tool access: grant the smallest scope, check it on every call, and never accept the model's statement about who is asking as proof of identity. The NIST AI Risk Management Framework gives a useful vocabulary for assigning each risk to an owner and a control. For personal data, check which records the agent may copy into its memory, how long they stay there, and whether the notice and purpose rules that apply to your business are reflected in the harness configuration. Requirements that vary by business type are covered in guardrails for a business agent.
How Be Human approaches the harness
Be Human designs and operates the harness as part of an engagement, not as a one-time installation. A first engagement defines one useful role, the memory and tools it needs, the approval boundaries and an observable success criterion, so the controls in this checklist are specified before anything runs. Managed operations then keep those controls true after launch: monitoring of traces, recurring evaluations, permission review whenever a tool or integration changes, integration maintenance, incident response and controlled improvements that are tested before they reach live work. The goal is that the setup still passes this review months later, when the original team has moved on to other work.
Choosing the first agent to hold to this standard
The first agent should be a task whose inputs, tools and failure cases are already understood by the people who do the work today. A narrow role makes every control easier to test, and it gives you a reference trace to compare later changes against. Teams often start with a task that is repetitive but bounded, and the agent or deterministic workflow decision guide helps separate tasks that need judgment from tasks a fixed process handles better. If you are not yet sure which task deserves to go first, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.
Frequently asked questions
Is a harness the same thing as the model? No. The model produces text or decisions, while the harness controls the environment around it, including which tools the model can call, which records it can read, what must be logged and when a person must approve an action. Two agents can run the same model and still differ sharply in reliability and safety because their harnesses differ.
Do we need to log every single step in full detail? Log what lets a reviewer reconstruct a task: the request, the model used, each tool call with its arguments and result summary, memory changes, approvals, errors and stop reasons. Set retention periods for raw content according to your data rules, and keep the trace readable. A smaller, complete record is more useful than a large dump that nobody can search.
Can a small team meet these requirements without an engineering department? Yes, if the requirements are used as a review checklist and the team accepts that some items need outside help to build. A small team can usually decide the approval points, the success criterion and the stop rules on its own. Configuring traces, scoped credentials and rollback often needs someone who maintains the integrations, which is a common reason to use a managed operations model.
Should the first agent cover a broad area or a narrow one? A narrow one. Choose a single role with a clear success criterion, a short tool list and known failure cases. A narrow first agent lets you test every control in this checklist before widening its permissions, and it produces a reference trace that makes later changes easier to judge.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


