How to Pick an AI Implementation Consultant for Superintelligence Prep

A good AI implementation consultant is judged by what they leave in your hands, not by the model they name. Look for a candidate who makes memory, permissions, evaluations and exit terms explicit before any build starts, and who can show you where each one lives and who can change it. The checklist below turns those four areas into questions you can put to any proposal.

Key takeaways

• Score candidates on who owns the memory, records and configuration, not on which model they favor.
• Require one named role, its tools and its approval boundaries in writing before any build begins.
• Ask for an evaluation plan with a success criterion you can read and check yourself.
• Confirm exit terms: data export, configuration handover and a tested path to another provider.
• Treat permission review, monitoring and incident response as ongoing work with named owners.

Why model choice is the wrong first test

Models change faster than most contracts. A consultant who sells on a model name is selling something you will probably replace. The durable part of an AI agent is the operating context around the model: its instructions, the memory it builds, the tools it can call, and the rules for what it may do without a person approving. The NIST AI Risk Management Framework frames AI risk as something governed and managed across the whole life of a system, which is a more useful lens for a buyer than any single benchmark.

Ask the consultant to describe the agent in those terms. If the answer stays on the model, you have learned how they sell, not how they will run the work.

Who owns the memory the agent builds?

Agents remember. They keep notes from earlier tasks, retrieve company documents and carry context from one conversation to the next. That memory is some of the most valuable material your company will produce around AI, so treat it as business records, not as a byproduct of the software. Before signing, confirm the following in writing:

• Your company owns the memory, prompts, configuration and logs created for your agent.
• The storage location and hosting region are named, and you can request access logs.
• Memory can be exported in open, readable formats at any time and at exit.
• Your material is not used to train or tune models that serve other clients.
• Deletion is possible on request, with the process and timing agreed in the contract.

Each item should appear as a clause, not as a verbal assurance. A consultant who cannot name where the memory is stored is not yet running the system in a way you can audit.

How are permissions scoped and approved?

Every tool an agent can call is a permission. Reading a shared folder, sending a message, updating a record and issuing a refund each carry a different level of risk. OWASP's authorization guidance favors checking access at the point of each action and denying by default, rather than granting broad access once and hoping nothing goes wrong later.

Ask the consultant for a tool list with one line per tool: what it reads, what it writes, who approves its use, and what happens when it fails. Then ask which actions require a human to confirm before they run. A candidate who says the agent will work out what it needs on its own is describing a permission problem, not a capability.

Illustrative scenario: a regional distributor wants an agent that answers internal questions about delivery status. The operations lead and the consultant agree that the agent may read order records but may not change them, and that any answer quoting a delivery date must cite the source record. Sending a customer-facing message requires a person to approve it first. Each boundary is written into the tool list before the first connection is made.

What should the evaluations prove?

An evaluation is a repeatable test: a fixed set of realistic tasks, the result you expect, a way to score it, and a rerun after every meaningful change. Anthropic's engineering guidance on evaluating agents describes the same building blocks, including tasks, graders and recorded transcripts. For a business buyer, the question is simpler. Can the consultant show you the test set before the build, and will they run it again after each update?

Ask for the success criterion in plain language that an operations leader can check without help. Ask how each agent action will be recorded so a failed run can be traced to its cause. OpenTelemetry's observability concepts describe traces, metrics and logs as the basic signals for this kind of tracing, and a consultant should be able to name which of them they will use.

A simple sequence to run with any candidate:

• Step one: agree the success criterion in one sentence that a business owner can verify.
• Step two: build the test set from real past requests, including the awkward and ambiguous ones.
• Step three: record the expected action for each test, including what the agent must refuse or escalate.
• Step four: run the set before go-live and after every change, and keep the results.
• Step five: review failed runs with the consultant and decide whether the fix belongs in memory, tools or instructions.

Which exit terms protect your company?

Exit terms decide whether you can leave without losing the work. Lock-in usually comes from three places: configuration that exists only in the consultant's tools, credentials that only the consultant controls, and memory stored in a format nobody else can read. A useful contract addresses each of them.

Ask for a termination clause with a notice period, a handover of prompts, tool definitions and documentation, revocation of every credential the consultant issued, and a defined date by which all exported data is returned. Then ask for a rehearsal. A restore or migration tested once on a copy tells you more than any clause. Our guide on avoiding AI vendor lock-in covers the commercial side in more depth.

What should a proposal and its team show before you sign?

A proposal should be specific enough that you could hand it to a colleague and get the same agent built. Check that it names the role, the tools, the approval boundaries, the success criterion and the owner of each. Then check the people. Who will do the work, how much of their time is committed, and who covers incidents when the primary contact is unavailable?

Use this decision guide once the proposal is in hand:

• Proceed when memory ownership, tool permissions, evaluations and exit terms are all written and each has a named owner.
• Pause when any of those four areas is described only in general language, and ask for revised wording.
• Decline when the consultant promises outcomes they do not control, or resists written evaluation rights, data export or a handover.

For a fuller picture of the day-to-day role, read what an AI implementation consultant actually does.

What does ongoing operation cover after launch?

The hardest part of an agent is rarely the first build. It is the months after launch, when integrations change, permissions drift and new kinds of requests arrive. Ask who monitors the agent, how often the evaluation set runs, who approves a change to tools or memory, and who gets called when the agent fails. Quarterly permission reviews, with written evidence, are a reasonable baseline; our quarterly AI agent permission review steps show what that evidence can look like.

Be Human designs, implements and operates personalized AI infrastructure, with memory, tools, permissions and operating context kept under the client's control. A first engagement defines one useful role, the memory and tools it needs, its approval boundaries and an observable success criterion. Managed operations can then cover monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements, each recorded so the client can see what changed and why.

If you are not yet sure which task deserves an agent first, the free Be Human AI diagnostic can help you identify it.

Frequently asked questions

Do I need an AI implementation consultant, or can my own team build the agent? Your team can build a small agent, but the decisions in this checklist, such as memory ownership, approval boundaries and exit terms, need an accountable owner inside the business. A consultant can speed up the build and document the operating context, but your company should keep the decision rights. Agree on who holds those rights before the work starts.

Should the consultant use the model my company already pays for? Not necessarily. Ask which model the agent depends on, whether it can be replaced without rebuilding the memory and tools, and how the consultant will test a replacement. Keeping the operating context under your control turns a model change into a managed update rather than a rebuild.

What should I get in writing before the first build? Get the role description, the tool list, the approval boundaries, the success criterion, the evaluation plan, the owners for memory and permissions, and the exit clauses. If any of these exist only in a sales conversation, they are not yet part of the agreement. Written limits protect both sides.

How do I tell a hype-driven consultant from a serious one? Hype sounds like broad promises, a focus on named models and vague limits on what the agent may do. A serious candidate asks detailed questions about your processes, narrows the first role, shows the test set and the records they will keep, and is comfortable writing down what the agent must not do without approval.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.