Skip to content
be-human.ai
← Back to the journal

Personalized AI infrastructure

AI Implementation Partner: Questions to Test Superintelligence Readiness

A buyer's question list for judging whether an AI implementation partner can build a company layer that gets stronger as models improve.

Be Human typographic cover with the title: AI Implementation Partner: Questions to Test Superintelligence Readiness
Conceptual illustration created with AI · Be Human

An AI implementation partner designs, builds and operates the agents, memory, tools and permissions a company relies on once the demo is over. To judge whether one can build a company-level layer that gets stronger as models improve, ask for specific answers on how memory and tools are separated, how improvement is measured, how changes are approved and who runs the system after launch. The questions below are written for the owner or operations leader who has to make that call.

Key takeaways

  • A capable partner describes the layer as separate parts (model, memory, tools, permissions, operating context) so that each part can change without rebuilding the others.
  • Improvement from a better model is only real if you can measure it on your own tasks with a repeatable test.
  • Ask for the evidence format before you sign: traces, evaluation results, permission records and an incident log.
  • Permissions and change approvals should be written down with named owners on your side, not left to the vendor's judgment.
  • The partner's work after launch decides whether the layer keeps getting stronger, so judge monitoring and review as closely as the first build.

Start with what the partner is actually building

Many proposals describe an agent. Fewer describe the layer around it. A company-level layer holds what the agent may know, which systems it can reach, what it is allowed to do and how its work is checked. Superintelligence readiness, in practical terms, means that when a stronger model becomes available, the company can upgrade the reasoning engine without losing its memory, its permission history or its record of past decisions. A partner who cannot describe that boundary is selling a single tool, not a layer.

Ask the partner to draw the design. A one-page version with labeled boundaries is easier to check than a slide deck, and the process is described in Draw Your AI Stack: A One-Page Architecture Map for Leaders. If the partner cannot fit the design on one page, with each box owned by a named party, the design is probably not finished.

Ask how memory and tools stay apart from the model

Memory and tools are where company value sits. Memory is what the agent may retrieve: approved procedures, product rules, customer terms, past decisions. Tools are what the agent may do: open a ticket, read an order status, draft a reply for approval. When memory lives in a store you control and the model only reads from it at the moment of use, you can change the model later without losing the knowledge. Retrieval-based designs follow this pattern, as explained in cloud architecture guidance on retrieval-augmented generation.

Ask these questions directly:

  • Where is the memory stored, and can you export all of it without the partner's help?
  • Which tools does the agent call, and what does each tool accept and return?
  • Can one tool be replaced or disabled without changing the others?
  • Who decides what enters memory, and is each addition logged with a date and an author?

A partner who answers these with a tour of the agent's interface has not answered them. You want file locations, tool definitions and a written process for additions.

Ask how the layer improves when models change

A layer that gets stronger over time needs a fixed baseline. Ask how the partner will know that a new model is better for your work, not just better on public leaderboards. The answer should name a set of tasks taken from your operations, scored against criteria agreed in advance, and run before and after any change. Anthropic's engineering notes on evaluating agents describe this discipline in detail, and the logic applies whatever model vendor you choose.

Illustrative scenario: an equipment distributor (illustrative, not a real case) uses an agent to draft replies to supplier emails. The partner keeps a fixed set of past supplier emails with approved answers. When a newer model appears, the partner runs it against that set. The new model writes more clearly but invents delivery dates in a few of the test emails. The team keeps the current model for any reply that mentions dates, and the partner adds a rule that delivery dates must come from the order system through a tool, not from the model's own text. The improvement was real only because the test existed before the change.

The question to put to any partner is simple: what would make you tell me a model upgrade is a mistake? If the answer is vague, the layer has no way to get stronger on purpose.

  • Figure placeholder after this section.

Ask what evidence proves the agent works

Demonstrations show the best case. Ask for the record of ordinary cases. A capable partner can show a trace of each agent action: what it was asked, which memory it read, which tool it called, what came back and what it produced. The vocabulary of traces, logs and metrics is standard in software operations, and the OpenTelemetry observability primer is a useful reference for the terms. Your partner does not need to use that tooling, but should be able to explain what each record shows.

Request four items before any build begins:

  • A written success criterion tied to one task and one measurable outcome.
  • Sample traces from a test period, including the failed cases and how they were handled.
  • An incident log with the cause, the fix and the change that followed.
  • Evaluation results per model version, so that you can compare the layer over time.

If a partner resists showing failures, treat that as information. Systems that are managed well generate a record of problems, and that record is where the improvement comes from.

Ask who controls permissions, changes and data

Permissions should be the narrowest set the task requires. Actions that spend money, change customer records or send messages in your company's name should pass through an approval step with a named person. Access control guidance from OWASP stresses that authorization must be checked on every action, not only when a session starts, which is a sound test for an agent that calls several tools in one task. The NIST AI Risk Management Framework gives leaders a neutral vocabulary for mapping these risks to owners and controls.

Ask the partner:

  • Who on our side approves each permission, and who approves a change to one?
  • How is access removed, and how long does that take in practice?
  • Which parts of our data leave our systems to reach a model provider, and under which terms?
  • How are changes requested, reviewed and recorded? The mechanics are covered in How to Request and Approve Changes to a Managed AI Agent.

Be Human designs, implements and operates this kind of layer for companies. In managed operations, the permission set, the approval boundaries and the change requests are reviewed on a set schedule with the client's named owner, so that the written record of who can do what stays accurate as the company and its models change. Other parties should be able to offer the same kind of review, and you should ask for it in writing.

Run the partner interview in five steps

The following sequence turns the questions above into a decision you can defend internally. It takes a few weeks of normal business time and does not require technical training from the people in the room.

  • Step one: send the question list ahead of the meeting and ask for written answers before anyone talks about scope.
  • Step two: ask for the one-page architecture map, with memory, tools, permissions and owners labeled on it.
  • Step three: request an anonymized sample trace and an evaluation result, and ask what the partner would change if the result were worse.
  • Step four: ask what happens to memory, permissions and documentation if the working relationship ends. The answer should name what you keep and in what format.
  • Step five: score each answer as specific, partial or vague. Do not move forward until the answers on memory, evaluation and permissions are specific.

Keep the record of these answers. They become the basis of the statement of work, and the scope should follow the same structure as the answers. For the deliverables to expect from a consultant, see What a Superintelligence Consultant Should Deliver to Your Company. If you are not yet sure which task deserves the first agent, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

Do I need an implementation partner if my developers can build the agent themselves? Your developers may be able to build a working agent, but the harder work is the layer around it: memory ownership, permission reviews, evaluation after each model change and incident response. Ask whether your team has time to run those activities every month, not only to build them once. If not, a partner's value lies in operating the layer, and the interview above will show whether that operation is designed or improvised.

How can I tell whether a partner's claims about future models are realistic? Judge the claims by the mechanisms they describe, not by the forecast itself. A realistic partner talks about swapping models, measuring results on your tasks and keeping memory independent of any one vendor. Be cautious when a partner promises that a future model will solve your problem without a test that shows it. Plans that depend on forecasts rather than measurements are hard to audit.

What should be in the written agreement with the partner? The agreement should name the scope, the owners on both sides, the permissions granted, the evaluation method, the change approval path, the incident process and the terms for exporting memory and documentation. The statement of work should match the answers you received during the interview. Any item that exists only in a sales conversation should be written in before you sign.

How often should the partner review the layer after launch? The right rhythm depends on how fast your tools, data and models change, but a layer with live permissions should have a scheduled review of who can do what, plus a review of evaluation results after every model change. Ask the partner to propose the schedule in writing, including the evidence each review produces, and confirm which of those reviews you will attend.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →