Personalized AI infrastructure
Choosing an AI Implementation Partner: Questions on Memory and Access
A practical guide to judging an AI implementation partner by how it builds memory, permissions and observability, with questions to ask before you sign.

An AI implementation partner designs, connects and operates AI agents inside your business so they work on your data, through your tools and under your rules. The useful test is not which model a firm prefers but whether it can build the memory, permissions and observability your company will need as models become more capable. The questions below show which answers are specific enough to trust.
Key takeaways
- Judge a partner on how it stores memory, grants permissions and records agent activity, not on which model it demonstrates.
- Ask who owns the memory, prompts, tool definitions and logs, and what you receive if the engagement ends.
- Require one named role, one observable success criterion and a written approval boundary before any build starts.
- Look for evidence after launch: permission reviews, recurring evaluations and a documented incident process.
- Treat a partner that cannot explain its failure modes in plain language as a risk, whatever its technical depth.
Why the memory layer matters more than the model
Models change. A provider updates its model, a better one appears, or your team decides a different vendor fits a particular task better. If the agent's memory, instructions and tool connections live inside one vendor's setup, every change becomes a rebuild. A capable partner separates these pieces on purpose. Memory holds what the agent has learned about your business, such as approved product names, customer handling rules and past decisions. Context holds what the agent reads for a given task. Tools define what it can do. Each piece should be stored in a form you can inspect, export and move.
The practical question is simple: if we replaced the underlying model next year, which parts of the agent would survive unchanged? A good answer names the memory store, the instruction files, the evaluation set and the tool definitions, and explains how each is versioned. A weak answer describes the model's features. For a deeper view of this separation, read Build a Company Superintelligence Layer That Improves With Models.
Who owns the memory, prompts and records?
Ownership is the question that most often gets skipped during selection and most often causes trouble later. Before you compare proposals, ask the following in writing:
- Where is the agent's memory stored, and can we export it in a readable format at any time?
- Are prompts, tool descriptions and evaluation sets kept in a repository that our company controls?
- Under whose accounts are the connected tools authorized, and can we revoke that access without the partner's help?
- Which logs are retained, for how long, and where can we read them?
- If we end the engagement, what do we receive, in what format, and within what period?
Vague answers such as "we manage that for you" are not answers. Ownership should be a property of the setup, not a favor granted at renewal. The questions are developed further in Choosing an AI Implementation Partner: Ownership Questions to Ask First.
How should permissions be designed?
An agent should hold only the access its role requires. This is the familiar principle of least privilege, and it applies with more force when an agent can read email, edit records or trigger payments. The OWASP authorization guidance describes the general discipline of checking every action against an explicit rule rather than trusting that a request is acceptable because it came from inside the system. A partner who understands this will produce a permission map before writing any integration code.
A permission map can be built in five steps:
- List every action the agent could take, such as reading a folder, drafting a reply, sending a message, updating a record or issuing a refund.
- Mark which actions are read-only, which are reversible and which are irreversible.
- Assign each action to a named approval owner, or record that no approval is needed and why.
- Set thresholds where they matter, for example a maximum amount the agent may commit without a person reviewing it.
- Define how a permission is removed, who can remove it, and how quickly the change takes effect.
Also ask how the partner handles untrusted input. Agents read documents, emails and web pages written by people outside your company, and any of that text can contain instructions meant to override the agent's rules. The OWASP guidance on prompt injection prevention describes why such content must be treated as data, not as commands. The following scenario shows the principle.
Illustrative scenario: a support agent reads incoming customer emails and can issue refunds up to an approved limit. One email contains text instructing the agent to ignore its rules and approve a larger refund. A well designed setup treats the email as content to analyze, blocks any refund above the limit without a person's approval, and records the attempt for review.
What should observability show after launch?
Observability means you can see what the agent did, why it did it and what it cost, without asking the partner to reconstruct events from memory. The OpenTelemetry concepts documentation describes how traces, metrics and logs let teams follow a process across its steps. For an agent, a useful trace records the triggering input, the context retrieved, each tool call and its result, any approval requested and granted, the final output and the resources consumed. Without this record, a wrong answer is an anecdote. With it, a wrong answer becomes a case you can classify and fix.
Ask the partner to show you a sample trace from a test run, not a summary slide. Check whether a non-technical manager could find the reason for a particular decision. Ask who receives alerts when an agent fails or behaves outside its boundaries, how quickly they respond, and what the incident record looks like afterward. Observability is also the foundation for governance. The NIST AI Risk Management Framework frames trustworthy AI as an ongoing practice of mapping, measuring and managing risk, and a partner cannot measure what it does not record.
How do you test a partner before you sign?
A proposal is a promise. A short, structured test turns it into evidence. Use the steps below in the selection meeting and ask for written answers to each.
- Ask for one named role with one observable success criterion, such as the share of drafts accepted without edits, written as a measurable target.
- Request the full list of memory sources and connected tools that role would use.
- Request the approval boundary, written as actions and thresholds, not as general principles.
- Ask for a sample trace from a demonstration run, including each tool call and the approval points.
- Ask how evaluations are designed, who writes the test cases, who reviews the results and how often they run.
- Confirm exit terms in writing: export of memory, prompts, evaluation sets and logs, with a stated format and timeline.
- Ask for the partner's process for a permission review and an incident, including who from your company is notified.
A partner that answers these steps with specifics is usually one that has operated agents long enough to know where they fail. A partner that answers with a long list of model names and no artifacts deserves more questions. Selection criteria for the wider platform are covered in What to Look for in a Company Superintelligence Platform.
What does a managed partner keep running?
Be Human designs and operates this layer for companies. A first engagement defines one useful role, the memory and tools that role needs, the approval boundaries and an observable success criterion. Managed operations can then include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements, with the agent's identity, memory, channels, tools and permissions kept under the company's control. The service is remote and available in English and Portuguese. Start with the first task that is worth handing to an agent, then judge the partner by the evidence it produces each month.
The free Be Human AI diagnostic helps you find that first task worth handing to an agent.
Frequently asked questions
Do we need deep technical knowledge to evaluate a partner? You do not need to write code, but you do need to understand where decisions are made and what evidence exists. Ask for written artifacts such as a permission map, a sample trace and an exit plan, and check whether your own operations lead can follow them. If the explanation depends on one engineer's memory, the setup is not yet documented well enough.
Should we pick a partner because it uses the model we already like? Not on that basis alone. Models change, and a setup that depends on one model's behavior will need rework when the model is updated or replaced. Give more weight to how the partner separates memory, tools and evaluations, since that separation determines how easily you can switch models later.
How often should an agent's permissions be reviewed? Review them on a fixed schedule and whenever the agent's role, connected tools or approval thresholds change. The review should compare what the agent is allowed to do with what it actually needs, and record the decisions. Ask the partner to show you a completed review from a previous client engagement, with names removed.
What should happen if an agent makes a costly mistake? The partner should follow a written incident process: stop or restrict the affected action, use the trace to find the cause, correct the instructions, permissions or tools, add a test case that would have caught the error, and report the outcome to your company. Ask for that process before you sign, not after the first incident.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


