Design a First AI Pilot That Compounds as Models Improve

To start with superintelligent AI in a company, do not begin by chasing the most powerful model. Begin with one narrow role and build the parts you keep: its memory, its tests and its permissions. When a stronger model arrives, you run it through what you already have, so the upgrade adds value instead of restarting the work.

Key takeaways

• Treat the model as a replaceable part. The durable assets are memory, tests, permissions and run records.
• Pick one role with a named owner, a bounded set of tools and an outcome a person can check.
• Write the test set before launch, from real work, so a new model can be judged by evidence rather than by impression.
• Grant access by action, not just by system, and keep the permission map short enough to read in one sitting.
• Upgrade by running the same tests on the new model and promoting it only when it clears the same bar.

What does starting with superintelligent AI mean in practice?

Nobody can deliver superintelligence as a finished product that you install once. What a company can do is prepare the place where increasingly capable models will work: a defined role, a body of trusted company knowledge, a set of tools, and clear rules about what is allowed. Models will change several times. That surrounding layer is what you keep. The idea is developed further in the guide to the company superintelligence layer.

A first pilot is therefore not a test of a model. It is the first small version of that layer, narrow enough to finish and real enough to show you where your own processes are vague.

Why do first pilots often get rebuilt when the model changes?

The usual first pilot is assembled quickly. Someone writes a long prompt, pastes in documents, connects a tool using their own login, and judges the answers by feel. It works well enough to impress people. Then a better model appears and the trouble starts, because everything is fused to the quirks of the old one.

Four patterns cause most of the rework:

• Knowledge lives inside prompts instead of a maintained store, so it cannot be moved, corrected in one place or reviewed by its owner.
• Quality is judged by impression, so nobody can show that a change is an improvement or a regression.
• Access was granted to whoever built the pilot, so permissions reflect convenience and not the needs of the role.
• Runs are not recorded in a comparable way, so you cannot place old and new behavior side by side on the same work.

Each of these turns an upgrade into a project. The fix is to turn each one into an asset that exists independently of the model.

Which four assets should the pilot leave behind?

Whatever else the pilot achieves, it should leave these four things in a form you control:

• Memory. The facts, policies, glossary, examples of good output and past decisions the role needs, stored outside the model with an owner and a review habit. When a policy changes, one edit fixes it for every future model.
• Tests. A set of real cases with an agreed answer or grading rule, including awkward cases and cases where the right behavior is to stop and hand over to a person. This is what lets you compare two models fairly.
• Permissions. A short map of what the agent may read, draft, change or send, which actions need human approval, and who granted each one. The agent has its own identity, not a borrowed personal login.
• Tools and run records. Each tool has a clear name, defined inputs and outputs, and the agent's steps are logged so you can see what happened and replay comparable work later.

Notice what is not on the list: the exact wording of prompts. Treat it as a tuning detail for the current model, not as an asset. Keeping the four assets in formats you can export is also the practical heart of avoiding vendor lock-in with business agents. And a permission map that is short and honest is what makes a permission review quick instead of dreaded.

How do you build the pilot, step by step?

This sequence keeps the work reusable. Resist the urge to reorder it so that the exciting part comes first.

• Choose one role. Look for work that repeats, has clear inputs, and has a person who knows what good looks like. Define the success criterion before building, as described in how to define success criteria for an AI agent pilot.
• Write a role card. State the purpose, what the agent may do, what it must hand to a human, who owns it and who is the backup owner.
• Collect real cases. Gather past tasks, including messy ones, with the answer a competent colleague would accept. Remove personal or sensitive data the test does not need.
• Build memory as a store, not a paste. Separate stable material such as policy from fast-changing material such as current status, and give each part an owner.
• Define tools narrowly. Start read-only where possible, then add one action at a time, each with its own permission line and approval rule.
• Run the tests on the first model and keep the results as the baseline. When something fails, fix memory or tools before touching the prompt.
• Run with a small group of users, with human approval on any consequential action. Log every run in the same format.
• Hold a review against the success criterion. Decide to continue, widen or stop, and write the decision down.
• Rehearse an upgrade before you need one. Run the same tests against a second model in a separate setup, and read the differences. This proves the pilot is portable while the stakes are low.

The last step is the one most teams skip, and it is the one that shows whether the first eight built anything reusable.

What does a model upgrade look like when the pilot is built this way?

Illustrative scenario. A distribution company pilots an agent that drafts replies to supplier queries about deliveries. The role card says it may read the order system and draft emails, but never send them or change orders. Memory holds supplier terms and the company's tone guidelines. The test set contains past queries, including a few where the correct behavior is to escalate to a buyer.

Later, a stronger model becomes available. The team points the same agent at it in a separate setup and runs the tests. The new model handles the escalation cases better, but its drafts run longer than the style guide allows. The team adjusts one style instruction in memory, reruns the tests, and promotes the new model. Permissions do not change, because the role has not changed.

Then the owner proposes a second task, since the model seems capable of it. That proposal goes through the same path as everything else: new test cases, a new permission line, an approval. Capability grew, and control grew with it. For how such requests are handled in a managed setup, see how requests are approved in proof of concept, pilot or production.

How does Be Human's managed approach handle this?

Be Human designs, implements and operates personalized AI infrastructure for companies. A first engagement defines one useful role, the memory and tools it needs, the approval boundaries and an observable success criterion. Memory, tests and permissions are kept as documented parts of the setup under the client's control, not as hidden details of one model. Managed operations can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. When a new model is considered, it is evaluated against the same tests before any change goes live. Service is remote, in English and Portuguese.

How can you tell the pilot is compounding?

Check for these signs after the first review:

• A new model can be evaluated in hours, without rewriting instructions from scratch.
• Every real failure becomes a new test case.
• Anyone responsible can read what the agent is allowed to do and who approved it.
• A correction to company knowledge is made once and appears in every later run.
• The owner can state what result would make you pause or stop.

The opposite signs are just as clear: only one person understands the setup, upgrades are discussed with dread, and scope grows because a model looks impressive in a demo. Widen scope only when the tests and the success criterion say the current scope is working.

Frequently asked questions

Should we wait for more capable models before starting? No. The memory, tests and permissions you build now are the parts a better model needs in order to be useful and safe in your company. Waiting gives you no head start on them, and starting small lets you learn where your processes are unclear.

Do we need to fine-tune a model for the pilot? Usually not at the start. Keeping knowledge in a maintained store and measuring with tests is easier to correct and to carry to a new model. Fine-tuning ties work to a particular model version, so consider it later and only if the tests show a gap that memory and tools cannot close.

How large should the test set be? Large enough to cover typical requests, difficult requests and cases where the agent should refuse or escalate, and small enough that someone actually maintains it. It should grow every time a real failure appears. Quality of coverage matters more than volume.

Who should own the pilot? A named business owner who can judge whether the output is good, plus someone accountable for access and approvals. Name a backup for both, so the pilot does not depend on one person. A sponsor who can decide to widen or stop completes the picture.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.