Skip to content
be-human.ai
← Back to the journal

Personalized AI infrastructure

Enterprise AI Agent Platforms: A Buyer Checklist for Keeping Gains

Buyer checklist for an enterprise AI agent platform: memory ownership, export, permissions and observability questions to ask before you sign.

Be Human typographic cover with the title: Enterprise AI Agent Platforms: A Buyer Checklist for Keeping Gains
Conceptual illustration created with AI · Be Human

An enterprise AI agent platform is the infrastructure that runs AI agents with their own memory, tools, channels and permissions, so the company keeps control when the underlying models change. When you compare vendors, the decisive questions are less about which model they use and more about who owns the memory, whether you can export it, how permissions are granted and revoked, and what you can observe when an agent acts. If the answers are clear, your agents can keep what they learn as models improve.

Key takeaways

  • Memory should belong to your company and be stored in a form you can read, export and delete without asking the vendor's console for help.
  • Ask for a demonstrated export of memory, tool definitions, instructions and logs, not a verbal promise that data is portable.
  • Permissions should be scoped per agent and per tool, with human approval thresholds and a recurring review that leaves evidence.
  • Observability should show what the agent saw, what it decided and what it did, with a named person responsible for reading it.
  • Keep the model replaceable. Gains should live in the layer you control, not inside one vendor's black box.

Who owns the memory your agent builds?

Memory is what separates a useful agent from a chatbot that forgets. Over months, an agent accumulates customer context, process exceptions, preferred formats and decisions that people approved. That material is a business asset, and the contract should describe it as one.

Ask the vendor these questions before anything else:

  • Where is memory stored, in what format, and can we read it without the vendor's interface?
  • Is memory scoped to one agent and one business unit, or shared across the platform's customers?
  • Can our conversations or records be used to train or tune models that other customers use?
  • What happens to memory when the contract ends, and how is deletion confirmed in writing?

Retrieval-based designs keep company knowledge in a separate store that the model queries when it needs an answer. That separation makes ownership easier to verify than knowledge absorbed into model weights, where it is hard to inspect or remove. The AWS guidance comparing retrieval with fine-tuning explains the trade-off in plain terms and is useful reading for a non-specialist buyer. For a deeper view of how memory and permissions interact, see Why Smarter AI Models Need Scoped Memory and Explicit Permissions.

Can you export everything the agent depends on?

Export is the test of whether a platform is a tool you rent or a system you own. A complete export should cover memory, tool definitions and their schemas, instructions and prompts, evaluation sets, permission configuration, and logs. Credentials should stay in your own secret store, so they never need to be exported at all.

The most common gap is the description. Teams often export the code behind a tool but not the text that tells the model when and how to use it. Without that text, a rebuilt agent behaves differently even though nothing looks missing. Ask whether tool descriptions and approval rules are included in the export, and ask to see a sample.

The practical test is simple. During evaluation, request an export of a sandbox agent and have your own technical person read it. If the files are readable without the vendor present, you have evidence. If the answer is a promise to help later, you have a dependency.

Illustrative scenario: a regional services company decides, after a year, to change model providers. Its team exports memory, tool definitions and approval rules. Two tools need rewriting because their descriptions existed only inside the vendor's console. The lesson is that export must include the descriptions and rules, not only the code. For the broader pattern, read How to Avoid AI Vendor Lock-In With Business Agents.

How are permissions granted, limited and reviewed?

Permissions decide what an agent can actually change in your business, so they deserve more scrutiny than the interface. Authorization guidance from OWASP stresses checking access on every request, not once at setup, and denying actions by default unless they are explicitly allowed. Those principles translate directly into vendor questions.

  • Are permissions defined per agent and per tool, or is one service account used for everything?
  • Can approval thresholds be set by action type, record type or risk level, such as sending a message versus changing a payment record?
  • Who can grant a new permission, and is that grant logged with a reason?
  • Can we revoke a permission within minutes without redeploying the agent?
  • How is a periodic permission review run, and what evidence does the platform produce for it?

A review is only useful if it happens on a schedule. A quarterly cycle that compares granted permissions against what the agent actually used is a reasonable starting point. The steps are described in Quarterly AI Agent Permission Review: Steps and Evidence.

What can you observe when an agent acts?

Observability answers the question every operations leader eventually asks: what did the agent do yesterday, and why? The OpenTelemetry primer defines the basic signals, which are traces that follow a request through its steps, metrics that summarize behavior over time, and logs that record events. For an agent, a trace should show the request received, the context retrieved, each tool call with its inputs and outputs, every approval requested and granted, and the final response or action.

Ask whether you can see that trace for any single run, how long traces are retained, whether sensitive content can be masked, and who receives alerts when an agent fails or behaves outside its limits. Also ask whether cost and volume per agent are visible, so unusual activity is noticed early.

Observability without an owner produces dashboards that nobody reads. Name the person who reviews the trace summary, and decide what kind of pattern triggers an incident response. Incident handling for agents is covered in On-Call Rotation for AI Agents: Who Gets Paged, and When.

How does the platform handle a better model?

Stronger models arrive regularly, and a business benefits from them only if the agent can be moved to a better model without losing its memory, tools, permissions or track record. The gains stay with you when those assets sit outside the model. The model becomes one component you can test and replace.

Ask the vendor how a model change is made. Can the same evaluation set run against the current model and the candidate before anything changes? Is the switch a logged change with an approver? Can you roll back within a defined time? Evaluation practice for agents is described in Anthropic's guidance on [demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), which is a useful reference for the questions you put to a vendor.

Be Human designs personalized agents so that identity, memory, channels, tools, permissions and operating context stay under the client's control, with the model treated as a replaceable part. In managed operations, a model change goes through recurring evaluations and a controlled change request before it reaches production users, and the evidence is kept with the agent's records. This describes how the work is organized, not a promise of any particular outcome.

A vendor call plan you can run this month

Use the questions above as a structured call rather than an open demo. Send them in writing before the meeting, so the vendor can answer with documentation instead of impressions.

  • Before the call: write down the one business process you would hand to an agent first, and the decisions that must stay with people.
  • During the call: ask for a live demonstration of an export, a permission change and a single run trace, not a slide of each.
  • After the call: score each answer as documented, promised or unclear. Treat unclear answers as open risks.
  • In the contract: require export rights, data deletion terms, permission review support and trace retention in writing.
  • Before any pilot: confirm who owns the agent's memory, who approves permission changes and who reads the weekly trace summary.

The NIST AI Risk Management Framework offers a neutral vocabulary for these governance questions, and it helps when internal reviewers ask why each control exists. Keep the scoring sheet with the contract so that the decision can be explained later.

If you want to find the first task worth handing to an agent before you compare vendors, the free Be Human AI diagnostic is a practical place to begin.

Frequently asked questions

Do I need a full enterprise platform to start with one agent? Not necessarily. A single well-scoped agent still needs owned memory, explicit permissions and readable logs, so the same questions apply at a smaller scale. Starting small makes it easier to test export and review before the agent grows.

What is the single most important question to ask a vendor? Ask whether you can export the agent's memory, tool descriptions, permission settings and logs, and ask to see a real sample during evaluation. The answer shows whether the platform is a system you own or a service you rent. Vague answers should be treated as a dependency.

How often should agent permissions be reviewed? Quarterly is a sensible default for most companies, with an earlier review after any significant change to tools or data sources. Each review should compare granted permissions with actual use and record the decisions. The evidence matters as much as the review itself.

Does a better model automatically make my agent better? Not by itself. A stronger model improves results only if the agent's memory, tools and evaluation sets remain intact and are tested against the new model before it reaches users. Gains are kept when the surrounding layer is owned, documented and reviewed.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →