AI Agent Versioning: Release Prompts, Tool Schemas and Models Together
Version prompts, tool schemas and model identifiers as one release so every agent run traces back to an exact, reviewable configuration.

AI agent versioning means giving the prompt, the tool schemas and the model identifier one shared release identifier, so that every agent run can be traced to the exact configuration that produced it. Versioning these three parts in separate places leaves gaps where an agent changes behavior without anyone approving the change. The practical fix is a release manifest that lists all three parts, a unique identifier for the release, and a record of that identifier on every run.
Key takeaways
- Treat the prompt, tool schemas, model identifier and related settings as one release with one identifier.
- Keep a release manifest that points to exact versions, not to floating names or the latest file.
- Stamp the release identifier on every run, trace and tool call so an incident can be matched to its configuration.
- Promote releases through a test environment and keep the previous release ready to restore in one step.
- Approve each release with evaluation evidence and a named owner, not with a quick reading of the prompt.
Why separate versioning of prompts, tools and models breaks down
Most teams start in a familiar way. The system prompt lives in a document or a repository file, the tools live in application code, and the model name lives in a configuration setting. Each piece has its own history, and each may be reviewed by a different person, if it is reviewed at all. The problem is that an agent never runs one of these pieces alone. It runs a combination, and its behavior comes from how the pieces fit together.
Consider three changes that look small. A prompt sentence is rewritten to make answers shorter. A tool description is edited to clarify when the tool should be used. The model setting moves to a newer version of the same family. Each change may pass a quick check on its own, yet together they can change which tool the agent calls, what arguments it passes and how it phrases the result. If the changes were recorded in three places with no shared identifier, nobody can later say which combination produced a bad answer.
Model identifiers need particular care. Some providers offer aliases that point to a current version and move over time. An alias is convenient, but the same configuration string can then produce different behavior after the provider changes the underlying model. A pinned identifier in the manifest keeps the record honest, and an upgrade becomes a deliberate release rather than an invisible side effect.
What belongs in one agent release
A release should contain everything that can change how the agent behaves. If a setting can alter an answer, a tool call or a permission decision, it belongs in the manifest. A practical release includes:
- The system prompt and role instructions, stored as a file with a content hash.
- The tool registry: each tool name, its description, its input and output schemas, and the version of the code that implements it.
- The model identifier, pinned to an explicit version where the provider offers one, along with the provider name.
- Model parameters such as output length limits, sampling settings and any reasoning effort setting.
- Approval rules and permission scopes attached to each tool, so permission changes appear in the release history.
- Retrieval and memory settings, including which document collection the agent may read and the version of that index.
- The evaluation suite version and the result of the run that the release passed.
Do not place secrets in the manifest. Store credentials in a secret manager and reference them by name. The manifest should be readable by a reviewer who was not part of the original change. Anthropic's guidance on writing tools for agents is a useful reference when you review tool descriptions and schemas, since those descriptions shape when and how the model calls each tool.
How to build the release manifest step by step
A manifest can be a single structured file kept in the same repository as the agent definition. The exact format matters less than three properties: it is complete, it is machine readable, and it is immutable once released. The steps below suit a small team with one agent and still work when the number of agents grows.
- Step 1: Define a release identifier format, such as the agent name followed by a sequence, and never reuse an identifier after its release is retired.
- Step 2: Move every behavior-changing setting into the manifest, so that nothing important lives only in an environment variable or a dashboard.
- Step 3: Hash the prompt and each tool schema, and record the hashes. A hash confirms that the text a run used is the text the reviewer approved.
- Step 4: Pin the model identifier to a specific version string where possible, and note any provider behavior that matters for your task.
- Step 5: Link the manifest to the evaluation results it passed, including the test cases covered and the pass criteria used.
- Step 6: Record the owner of the release, the person who can order a rollback, and the approval that moved the release into live use.
Once the manifest exists, reject any production change that does not produce a new manifest. That single rule prevents quiet drift. The NIST AI Risk Management Framework treats documentation and change control as part of governing AI systems, and a manifest is a practical way to meet that expectation for a small agent.
How to tie every run to one configuration
A release only helps an investigation if each run records which release it used. Each run should write a small record at its start, and that record should be attached to every model call, tool call and final output. The fields that matter most are:
- The release identifier and the manifest version.
- The model identifier as the provider reports it, when the provider reports one.
- The prompt hash and the tool schema hashes in effect for the run.
- The version of the retrieval index or memory collection the run read from.
- The outcome of each tool call, including error codes and input and output sizes.
- The stop reason, so that a run that ended early can be explained.
The OpenTelemetry semantic conventions for generative AI define attribute names for model and operation details. Using those names makes it easier to read logs from different tools together. Keep personal data out of these records where you can, and set a retention period that matches your own policy instead of keeping everything indefinitely.
An illustrative scenario: rolling back a refund triage agent
This scenario is illustrative and does not describe a real company. A small online shop runs an agent that reads refund requests, calls an order lookup tool and drafts a decision for a staff member to approve. Release 14 changes two things at once: the model identifier moves to a newer version, and the lookup tool renames a field in its output schema from purchase total to total paid. Soon after the release, some drafts describe orders with the wrong amounts.
Because every run records its release identifier, the team filters the bad drafts and finds that all of them came from release 14. Runs from release 13 show the expected amounts. The release 14 manifest lists both changes, so the team does not have to guess which one caused the problem. They restore the release 13 manifest, which returns the agent to the earlier model identifier and the earlier tool schema, and they hold release 14 until the field mapping is corrected and the same test cases pass. Without a shared release record, the team would have to compare a prompt history, a code history and a provider notice before reaching the same conclusion.
How to promote and roll back a release safely
Promotion should move a release through the same stages every time: a test environment built from the same manifest, an evaluation run against cases that reflect real work, a limited rollout, and then full use. Published guidance on evaluations, such as the OpenAI evals guide, treats evaluations as repeatable tests that you run on each change, and that is the habit to build. For the environment side, see why you need both test and live environments.
Rollback is the mirror image. It should restore a previous manifest as a whole, never one file at a time, because a mixed configuration is the very situation versioning is meant to prevent. Use this guide to choose the path for a change:
- If the change is wording in one prompt file, run the evaluation suite and a limited rollout before full use.
- If the change touches a tool schema or a permission, require review by the tool owner and a test run that calls the tool with realistic inputs.
- If the change moves the model identifier, treat it as a new release even when the prompt is unchanged, and compare results on the same cases.
- If quality drops or complaints rise after a release, restore the previous manifest first and investigate second.
For reviewing unexpected changes between releases, see how to catch unreviewed harness changes.
How Be Human handles release control, and where to start
Be Human designs and operates agents with this kind of release record from the start. In managed operations, the team keeps a manifest for each agent, reviews permission and tool changes, runs recurring evaluations after model or integration updates, and uses the release identifier during incident response so that an investigation begins with the configuration that actually ran. Controlled improvements are proposed as new releases, tested first and approved by the business owner before they reach live work. The first engagement defines one useful role, its memory and tools, its approval boundaries and an observable success criterion, and that definition becomes the first entry in its release history. If you are not yet sure which agent task deserves this level of control, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.
Frequently asked questions
What is AI agent versioning? AI agent versioning is the practice of recording the prompt, tool schemas, model identifier and related settings as one numbered release, so each agent run can be matched to the configuration that produced it. It goes further than versioning a prompt file, because a behavior change can come from a tool schema or a model identifier with no prompt edit at all. The release identifier is what connects these pieces during review and incident response.
Do I still need to record the model identifier if my provider handles upgrades? Yes, because a floating alias can change what your agent runs on without any change to your files. Pin an explicit identifier where the provider allows it, and record the identifier the provider reports on each run. An upgrade then becomes a release you can test and, if needed, reverse.
How many old releases should a small team keep? Keep enough history to restore the last working release quickly, and keep every release that reached live work, since an incident may involve any of them. Retire older manifests according to a written policy and never reuse an identifier. Storage is rarely the constraint, but unclear ownership of old releases often is.
Where should the release identifier appear? Put it in the run record, in every trace span for the run, in tool call logs and in any output metadata a reviewer sees. If the identifier appears in only one place, every investigation depends on that single log being complete. Treat the identifier as a required field in the harness, so that a run cannot start without one.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


