Questions to Ask an AI Agent Vendor Before You Sign the Contract

The questions to ask an AI agent vendor fall into five groups: who owns what the agent knows and produces, what it can access, who supports it after launch, how you can leave, and what evidence shows it is being supervised. A vendor who can answer each group in writing, with named roles and records you can inspect, is selling an operated system. A vendor who answers with general assurances is selling a demonstration.

Key takeaways

• Ask about ownership first: instructions, memory, outputs and the accounts the agent uses should be clearly yours, or clearly documented as not yours.
• Treat access as a permission list, not a feature list. Every connection needs a scope, an owner and a way for you to revoke it.
• Supervision is shown by records, approval boundaries and recurring evaluations, not by a promise that the agent is monitored.
• Support and exit are the same question seen from two ends: what do you receive, in what form, and who acts when something goes wrong.
• Score every answer by whether it points to something you can inspect. A refusal to put ownership, access or exit in writing is a reason to stop.

Who owns the agent, its memory and its work?

A demo shows what an agent can do on a good day. A contract decides who holds the pieces afterward. An agent is not one object. It is a set of instructions, tool definitions, connected accounts, accumulated memory, and a trail of outputs and logs. Each part can have a different owner, and vendors rarely volunteer which is which.

A practical test: imagine the vendor disappears next month. Which parts could you pick up and hand to someone else? If the answer is unclear, ask these questions before you sign:

• Who owns the instructions, tool definitions and configuration written for our agent?
• Who owns the memory the agent builds about our customers, products and decisions, and in what format can we export it?
• Who owns the outputs: drafts, summaries, records and logs?
• Are our documents and conversations used to train or improve anything outside our own account?
• Which model providers and subprocessors touch our data, and in which regions?
• Are the agent's accounts, numbers and mailboxes registered to us or to you?

Pay particular attention to memory. It is where the agent's working knowledge of your business lives. If it exists only inside the vendor's system, in a form you cannot read, you are renting your own institutional knowledge. Ask to see a sample of what stored memory looks like. A vendor who has designed memory as something a human can read and correct will be able to show you one.

Also ask where the agent's source of truth sits. If the agent answers from documents, someone has to decide which version is current and who can change it. That decision should belong to you.

What can the agent access, and who controls that access?

Most of the risk in an AI agent engagement sits in access. An agent that can only read a shared folder is a different proposition from one that can send email, edit your CRM and move money. A good vendor describes access the way a security reviewer would: per system, per action, per account.

Standard authorization patterns exist for this. OAuth 2.0 lets you grant a tool limited, revocable scopes without sharing a password, and general authorization guidance such as the OWASP Authorization Cheat Sheet recommends denying by default and granting the minimum needed. Your vendor should be comfortable speaking in those terms. The principle is covered in more detail in least privilege for AI agents.

Bring these questions to the call:

• Which systems will the agent connect to, and what level of permission does each connection have: read, draft or send?
• Are connections made through our own accounts with revocable scopes, or through shared or vendor-held credentials?
• Which people at your company can see our data or act through our connections, and where is that access logged?
• Can we revoke any connection ourselves, immediately, without opening a ticket?
• How is our data separated from the data of your other clients?
• What does the agent do when a connection expires or a permission is removed?

The last question is easy to overlook and revealing. An agent that fails quietly, retries blindly or improvises around a missing permission is a liability. The right answer is that it stops, records what it could not do, and tells a person.

If the vendor proposes broad access to start with and narrowing later, ask why. Narrow first and widen with evidence is the safer order.

How will the agent be supervised once it is live?

This is the question that separates a managed agent from a tool you are left to watch yourself. Every vendor says the agent is monitored. Your job is to find out what monitoring produces: which records, who reads them, how often, and what happens when something looks wrong.

Supervision has three parts, and you should ask about each.

The first is records. The general idea of observability, described in the OpenTelemetry primer, is that you can understand a system's behavior from the data it emits. For an agent that means a readable trail of what it received, what sources it used, which tools it called and what it produced. The article on what to log for AI agent observability goes through the specifics.

The second is boundaries. Some actions should never happen without a person approving them. Ask the vendor to show where that line is drawn for your workflow and who can move it. Where your AI agent should stop and ask explains how to decide.

The third is evaluation. An agent that worked at launch can drift as documents, tools and models change. Ask how quality is tested before go-live and on a recurring basis. A frameworks-level reference such as the NIST AI Risk Management Framework treats ongoing measurement and human oversight as part of managing an AI system, not a one-time check.

Use these questions:

• What is recorded for each run: input, sources, tool calls, outputs and approvals?
• Can we read those records ourselves, in plain language, without your help?
• Which actions need human approval, and where is that boundary written down?
• How is the agent evaluated before launch, and how often afterward?
• What is the success criterion for this role, and how is it observed rather than reported?
• Will the agent first run in shadow mode, producing proposals that nobody acts on until we have reviewed them?
• Who reviews the agent's behavior each week, and what do we receive from that review?

On shadow mode, a vendor who has done this before will have an opinion. The reasoning is laid out in running an AI agent in shadow mode. If the vendor wants to give the agent live authority on day one, ask what evidence they would use to justify that.

What does support cover when something goes wrong?

Support for an AI agent is not only a help desk. The agent depends on connected tools that change, documents that go out of date and models that get updated. Someone has to notice and act. The question is whether that someone is you, the vendor, or nobody.

Get specifics on these points:

• Who responds to an incident, in which language and time zone, and through which channel?
• What counts as an incident, and what is the first action: pausing the agent, revoking a connection, or investigating while it keeps running?
• Who maintains integrations when a connected tool changes its interface or permissions?
• How are changes to the agent's behavior proposed, tested and approved before they go live?
• Will we receive a written account after an incident, covering cause, effect and correction?
• Which routine checks happen on a schedule, such as permission review and evaluation reruns?

The difference between reactive and managed support shows up in that last item. A reactive vendor waits for you to report a problem. An operated service has a recurring routine that finds problems first. A sample of such a routine is in a weekly maintenance checklist for managed AI agents. Ask the vendor to show theirs.

Also ask about the language of support. If your team works in Portuguese, English or both, confirm that incident explanations and reports will arrive in the language your decision makers actually read.

How do you leave if the relationship ends?

Exit terms feel pessimistic to discuss while you are excited about a new capability. They are also the clearest signal of how a vendor thinks about your business. A vendor confident in the value of the service has no reason to make leaving hard.

Ask for the exit package in concrete terms:

• What do we receive when we leave: memory, configuration, instructions, logs and documentation?
• In what format, and how quickly after we ask?
• Are all connections and credentials revoked, and do we get that confirmed in writing?
• Is our data deleted on request, and what proof of deletion do we receive?
• Can the agent keep running for a defined handover period while we move to another provider or bring the work in-house?
• Do any contract terms limit our ability to export or to leave early?

A useful habit is to ask for the exit package as a document before you sign, not as a promise. If the vendor can describe the files, the folder structure and the handover steps today, they have thought about it. If the answer is that it would be handled case by case, expect delay later.

Exit also connects to the earlier ownership questions. Everything you could not export in the first section is something you would lose here.

How should you score the answers you get?

A long question list is only useful if you can compare the answers. Rather than adding up points, sort each answer into one of four groups:

• Strong: the answer names a role, a document or a record you can inspect, and the vendor offers to show it.
• Acceptable: the answer describes a process with a clear owner, but no sample exists yet. Ask for one before signing.
• Weak: the answer is a general assurance, such as our platform handles that, or it depends on a later conversation.
• Stop: the vendor will not put ownership, access or exit terms in writing.

Illustrative scenario: a regional distributor is comparing two vendors for an agent that answers order-status questions. Both demos look good. The operations lead sends the same short list to each vendor. One replies with a sample run record, a description of which actions need approval, and a list of what is delivered on exit. The other replies that monitoring and data portability are included in the platform. The operations lead does not need to judge the demos. The first vendor produced something inspectable, and the second produced a sentence.

Send the list ahead of the call, so the vendor can bring the right people and the right documents. Ask for the answers in writing, then use the call to probe the weak ones. If you want to go further, ask the vendor to run a small, defined task before you commit, with an acceptance checklist agreed in advance. An acceptance checklist for your AI agent integration can serve as a starting template.

How Be Human handles this: Be Human designs, implements and operates personalized AI infrastructure, and a first engagement defines one useful role, the memory and tools it needs, the approval boundaries and an observable success criterion. Managed operations can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. The service is remote and available in English and Portuguese. Those are the same areas this list covers, so expect to be asked for the same clarity in return.

Frequently asked questions

How many vendors should I compare before choosing one? Two or three is usually enough if you send each the same written questions. More than that tends to produce comparison fatigue without better information. The quality of the written answers will separate vendors faster than the number of features.

Should a small company ask all of these questions? Yes, but you can shorten the list by focusing on the workflow you are buying. A small company with one agent in one role still needs to know who owns the memory, what the agent can touch and how to leave. The depth of the answers can be proportionate, but the topics are the same.

What if the vendor says some answers are confidential? Some implementation details can reasonably be protected. Ownership, access scopes, approval boundaries, support responsibilities and exit terms concern your business and should be shareable. If a vendor will not describe these under an agreement of confidentiality, treat that as a weak answer.

Do I need technical staff to evaluate the answers? Not for most of them. You need someone who can read a run record and tell whether it is clear, and someone who can approve access to your systems. If neither exists internally, ask the vendor to explain each answer in plain language and to show the evidence. A vendor who cannot explain it plainly has told you something useful.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.