Superintelligent Harness Drift: Catch Unreviewed Agent Changes
Learn how to detect silent changes in agent prompts, tool lists and settings, and require human review before they reach production.

AI harness drift is the gradual, often unannounced change in the parts of an agent setup that shape its behavior: system prompts, tool lists, permissions, model identifiers and runtime settings. It usually surfaces late, when an answer reads differently or an action lands in the wrong system. The fix is to keep a baseline of every harness component, compare each live version against it on a schedule and after every release, and route any change that matters through a named human reviewer before it touches real work.
Key takeaways
- Treat the prompt, tool list, permissions, model pin and runtime settings as one versioned object, not as scattered settings.
- Keep a known-good baseline and compare every deployed harness against it, so silent edits become visible diffs.
- Classify each diff by risk, such as wording, new tools, widened permissions or model changes, instead of by who made it.
- Require a named owner to approve any change that widens tool access, data scope or autonomy, and record the reason.
- Run changes in a test environment against the same evaluation set before release, and keep a tested rollback path.
What counts as harness drift?
Drift is any gap between what your team believes an agent is configured to do and what it is actually configured to do. It rarely comes from one dramatic mistake. It usually comes from small edits: a system prompt copied from a shared document into a dashboard, a tool added during a debugging session and never removed, a permission scope widened so that a test would pass, a model alias that a provider repointed to a newer model, or a step limit raised because a long task kept stopping.
Most of these edits are reasonable on their own. The problem is that nothing records them as changes, so nobody reviews them, and the agent's behavior shifts without a decision anyone can point to. When an incident happens, the team then has to reconstruct what the harness looked like last Tuesday from memory and chat logs.
Build a baseline you can diff
A baseline is a complete, readable record of the harness at a known-good point. If you cannot diff it, you cannot detect drift. At minimum, record these components:
- The system prompt and any injected instructions, stored as exact text.
- The tool list, with each tool's name, description, input schema and the systems it touches.
- Permissions for each tool: read, write, send, delete, and any approval gate in front of them.
- The model identifier, the provider version where it is available, and any routing rules between models.
- Runtime settings: maximum steps, timeouts, retry limits and output length limits.
- Memory sources and the retrieval scope the agent is allowed to query.
- Hooks and stop conditions that decide when the agent checks its work or halts.
Store these as files in a single layout that people can read and review, rather than as values spread across dashboards. A reviewable file structure makes the diff itself meaningful, because a reviewer can see exactly which line changed. The workspace file layout for agents is one practical way to organize this.
A baseline that lives only in someone's head is not a baseline.
How do you detect changes automatically?
Detection works best as a routine, not as a one-time audit. The following plan is simple enough to run with a scheduled job and a shared review queue:
- Step 1: Export the live harness into the same file format as the baseline, on a fixed schedule and after every deployment.
- Step 2: Compare content hashes first. If the hash matches, nothing changed. If it differs, generate a human-readable diff.
- Step 3: Tag each diff by risk: wording only, new or changed tool, changed permission, changed model or routing, or changed limit or stop condition.
- Step 4: Send high-risk diffs to the reviewer queue immediately, and send low-risk wording diffs to a daily digest.
- Step 5: Record who or what made the change, when it happened, and the stated reason. An empty reason is itself a finding.
- Step 6: Attach the harness version to every run log, so that behavior can be traced back to the exact configuration that produced it.
The last step matters more than it looks. Structured run logs that carry a version identifier let you ask which configuration produced a given answer, which is the basic question an observability approach is designed to answer. The concepts in the [OpenTelemetry observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) are a useful general reference for how to think about traces, metrics and logs in this setting.
Hooks are the natural place for some of these checks. A pre-action hook can confirm that the tool list matches the approved version before a run starts, and a post-action hook can confirm that outputs still match the expected schema. The guide to hooks for automatic checks describes how to set them up.
Why a hash check is not enough
A hash tells you that the text changed. It does not tell you that the behavior changed, and some of the most important drift has no text change at all. A provider can update the model behind an identifier, a retrieval index can be rebuilt with different documents, or a tool's backend can start returning results in a new format. In each case the harness files are identical, yet the agent behaves differently.
For that reason, pair the diff with a recurring evaluation. Keep a fixed set of representative tasks, with expected outcomes or scoring criteria, and run it against the live harness on a schedule. When the score moves, or when a specific case fails that passed before, you have a behavioral signal even if no file changed. Evaluation design is a discipline of its own, and the guidance in [Anthropic's notes on demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) is worth reading before you build your own set.
Which changes should require review before production?
Not every change deserves the same scrutiny. A reviewer who must approve a comma will stop reading the queue within a week. Use a short decision guide that ties approval to consequence:
- Wording changes to tone or formatting: the reviewer sees the diff and may approve in a batch.
- New or changed tool descriptions: the reviewer confirms that the description matches what the tool can actually do, and that the agent will not choose it for the wrong task. The tool allowlist approach helps keep this list short.
- Any permission widening, including write, send or delete access: a named owner approves, and the reason is recorded.
- Model or routing changes: the change runs in a test environment against the evaluation set before anything reaches live work.
- Changes to stop conditions, budgets or retry limits: the owner and the operations lead both approve, because these decide when the agent stops and when it keeps going.
Test changes in an environment that mirrors production but cannot send real messages or change real records. The distinction between a test and a live environment is covered in why you need both a staging and a live environment. Prompt text also deserves basic safeguards against manipulated inputs, and the [OWASP prompt injection prevention guidance](https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html) covers the common patterns to review.
An illustrative scenario: a support agent that gained a refund button
The following is an illustrative scenario, not a report of a real client. A support agent is configured to draft refund recommendations for a human to approve. During a late debugging session, an engineer adds a tool that can issue refunds directly, so the test flow can complete without waiting for approval. The change is made in a hurry and the tool description is copied from the vendor documentation.
The next scheduled comparison tags the change as a permission widening, because the new tool writes to a payment system. The diff is held in the reviewer queue. The reviewer finds that the approval gate was removed along with the tool, and that the stated reason was only "testing." The change is rejected, the previous version is restored, and the team adds a rule that a write-capable tool cannot be added outside a staging run. Without the comparison, the first sign of the problem would have been a refund that nobody approved.
Authorization principles apply here as well. The [OWASP authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) describes the general idea of granting only the access a function needs, and an agent's tool list is a form of access grant.
How does Be Human handle drift in managed operations?
Be Human designs and operates AI agents whose identity, memory, channels, tools, permissions and operating context stay under human control. In the managed setting, the baseline for each agent is kept as part of the agent's own files, comparisons run on a schedule and after releases, and changes that widen permissions go to a named approver before they take effect. Monitoring and recurring evaluations cover behavior as well as configuration, and permission reviews are scheduled so that access granted for a test does not quietly remain in production. Controlled improvements are released through the same review path, so the people accountable for the work can see what changed and why.
If you are not sure which of your current agents or workflows would suffer most from an unreviewed change, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.
Frequently asked questions
Do we need engineers to track harness drift? You need someone who can read the baseline files and understand what a tool can do, but the routine itself does not have to be heavy. A scheduled export, a diff, and a short review queue can be run by an operations lead once the baseline is set up. Engineering help is most useful when the diff reveals a tool or permission that needs to be rebuilt.
Is a hash check enough to catch silent changes? A hash check catches any text change, which is a strong first layer, but it cannot see behavior changes that come from a model update or a changed data source. Pair it with a fixed evaluation set that runs on a schedule and reports when outcomes move. Together, the two signals cover both what changed and how the agent now behaves.
Who should approve harness changes? The approver should be the person accountable for the outcome of the work, with a second reviewer for changes that affect tools, permissions or stop conditions. Low-risk wording changes can be approved in batches by the operations lead. The important rule is that every approval is recorded with a reason, so the history explains each change.
Where should the baseline live? Keep the baseline in the same place as the agent's other reviewable files, under version control, so that every change has a history and a previous version can be restored. Avoid keeping the authoritative copy only in a vendor dashboard or a chat thread. If a configuration cannot be exported and compared, treat it as a gap to close before it becomes an incident.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


