How to Evaluate AI Vendor Stability Before You Commit

To evaluate AI vendor stability, stop trying to predict whether the company will survive and test whether your work would survive without it. Four things can be verified before you sign: documentation that matches the real system, the ability to export what you depend on, support that reaches beyond one person, and an architecture that does not hinge on a single model. A provider that passes these checks is a safer choice whatever happens to its funding or its favorite model.

Key takeaways

• Stability is less about the provider's size and more about whether your agent can keep working if the provider changes, shrinks or disappears.
• Ask for documentation written for the system you would actually run, then have someone who missed the sales calls read it.
• Request a real export of instructions, memory, tool definitions and logs before signing, and open it yourself.
• Support depth means named backups and written escalation, not one friendly contact.
• Ask what happens when the underlying model is retired or changes behavior, and look for a tested fallback.

What does stable actually mean for an AI provider?

Buyers usually start with funding, headcount and years in business. Those are hard to verify and weak predictors. A small, well-run provider can be a safer partner than a large one whose product shifts every quarter and whose knowledge lives in one engineer's head.

A more useful definition is continuity. If the provider raised its fees, retired a model, lost its lead engineer or closed, how much of your agent could you keep running? That question turns a vague worry into something you can inspect, and it maps to four signals: documentation, portability, support depth and dependency on one model.

The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) encourages organizations to map and manage risks that come from third-party components. These four signals are a practical way to apply that idea to a provider relationship without building a large governance program first.

Each signal can be tested with a request rather than an opinion. If a provider hesitates to let you test, that hesitation is information. For a wider method, see how to test an AI vendor's claims before you commit.

Does the documentation describe the system you would actually run?

Good documentation is the cheapest continuity insurance a provider can offer, because it lets someone other than the author operate the system. Marketing pages do not count. You are looking for material that explains how your agent will be built and maintained.

Ask to see, in redacted or template form:

• An architecture description that names the model, the memory store, the connected tools and the channels the agent uses.
• A role definition: what the agent does, what it must not do, and who approves exceptions.
• A runbook for routine work such as updating instructions, rotating credentials and handling a failed integration.
• A change log showing how past updates were recorded and reviewed.
• An incident process: how you are told when something breaks, and what the written follow-up contains.

Then run a simple test. Give the material to a colleague who did not attend any sales meeting and ask them to explain, in their own words, how the agent is configured and who to call when it misbehaves. If they cannot, the documentation is a person, not a document.

Also check freshness. Documentation that describes features the product no longer has, or omits recent changes, suggests no one owns it. The same material should feed directly into your statement of work, so that what is documented is also what is promised.

Can you take your agent with you if you leave?

Portability is the signal that matters most when everything else goes wrong. It answers one question: if the relationship ended tomorrow, what would you walk away with?

Break the agent into parts and ask about each one:

• Instructions and role definition, in a readable format you can open without special software.
• Memory and knowledge sources, including how they are stored and whether they can be exported.
• Tool and integration definitions, including which accounts and credentials they depend on.
• Permissions and approval rules, written down rather than implied by configuration.
• Logs and audit records of what the agent did.

Ownership of accounts deserves its own line. Check whether the model account, the cloud project, the email identity and the integration credentials sit in your company's name or the provider's. An agent that runs entirely on accounts you cannot access is not portable, however good the export file looks.

Open standards help at the tool layer. The [Model Context Protocol](https://modelcontextprotocol.io/docs/getting-started/intro) is one example of a shared way to connect models to tools and data, which makes it easier to move a tool connection between systems than if every connector were proprietary. Ask whether the provider uses standards where they exist and explains where it does not.

The decisive test is simple: request a sample export before signing and have your team open it. If it takes weeks to produce, you have learned how an exit would feel. The full mechanics of leaving are covered in how to migrate AI agents to a new provider, and the lock-in side of the question in how to avoid AI vendor lock-in with business agents.

How deep is the support behind the first contact?

A responsive salesperson says little about what happens in month nine. Support depth is the number of people and written processes between your problem and its resolution.

Put these questions to the provider and ask for answers in writing:

• Who is the named primary contact, and who is the named backup who also knows your setup?
• If the primary contact is unavailable for two weeks, who acts, and with what authority?
• What are the response windows for a broken integration versus a quality complaint, and who decides which is which?
• How are you informed of incidents, and do you receive a written explanation afterward?
• Which time zones does support cover, and what happens outside them?

Pay attention to who does the maintenance work. Models get retired, integrations change their interfaces, and credentials expire. A provider with a routine for these events, rather than a promise to react, is more likely to still be useful a year from now.

Also ask how the provider would behave if your own people changed. A good provider can onboard a new owner on your side from documentation alone. If the relationship only works because one of your managers has a personal rapport with one of theirs, the continuity is social, not structural.

How much does the provider depend on one model?

Most AI agents rely on a model operated by another company. That company can change behavior between versions, retire a version, alter its terms or adjust its limits. A provider fully tied to one model passes that risk straight to you.

The goal is not to use many models. It is to know where the value sits. In a well-designed system, the durable assets are the role definition, the memory, the tool connections and the evaluation set, and the model is a replaceable component. In a fragile one, the behavior you liked is a side effect of one particular model and nobody can say why it works.

Ask the provider:

• What happens, step by step, when the current model is retired?
• Is there a tested alternative, or only a plan to find one?
• Do you keep a set of example tasks with known good answers, so a replacement can be compared against current behavior before it goes live?
• Who approves a model change, and how are you told?

The evaluation set is the strongest evidence. Anthropic's article [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) explains why systematic checks matter once agents are in production, and the same logic applies to swapping a model. A provider with no way to measure the difference cannot safely change anything. For the strategic side, see when more than one AI model pays off.

What checklist can you run before signing?

The signals above fit into a short verification sequence. Treat it as a week of focused requests, not an audit.

• Ask for the documentation set and have a colleague outside the sales process summarize it.
• Ask for a sample export of instructions, memory format, tool definitions and logs, and open each file.
• List every account the agent would run on and mark who owns each one.
• Request the named primary and backup contacts, the escalation path and the response windows in writing.
• Ask for the model retirement plan and the evaluation approach used to compare a replacement.
• Ask how incidents are communicated and see an anonymized example of a written follow-up.
• Ask what the provider does with your data, including whether it is used for training, and get the answer in writing.
• Score each answer as verified, claimed or missing, and decide which gaps you can accept.

Illustrative scenario: a distribution company compares two providers for an agent that drafts replies to supplier emails. Both demonstrate well. Provider A sends a documented role definition, a sample export in plain files, accounts set up in the company's name, and a named backup contact. Provider B promises the same quality but says exports are available on request after launch, and the only contact is the founder. The demonstrations were equal. The continuity was not, and the decision becomes straightforward once the evidence is on the table.

One caution: verified beats claimed. A promise to provide an export is weaker than an export you have already opened.

How does a managed approach handle continuity?

Be Human treats continuity as part of the design rather than an afterthought. Each engagement starts by defining one role, the memory and tools it needs, the approval boundaries and an observable success criterion, and these are written down so that the client holds a readable description of the agent. Identity, memory, channels, tools and permissions are kept under human control, and the model is treated as a component that can be evaluated and replaced. Managed operations include monitoring, recurring evaluations, integration maintenance, permission review and incident response, so model changes and broken connections are handled as routine work. Service is remote, in English and Portuguese.

Frequently asked questions

Should I avoid young AI companies? Not automatically. Age says little about whether your agent will keep running, while documentation, exportable assets and named backup support say a great deal. A young provider that passes those checks can be a safer choice than an established one that fails them.

Is a large provider automatically more stable? Size reduces some risks, such as sudden closure, but adds others, such as product changes you cannot influence and support that treats you as one account among many. Run the same checks on every provider, large or small, and compare the evidence rather than the brand.

What if the model vendor changes its terms or retires a model? Ask the provider for its written plan before it happens. A prepared provider keeps an evaluation set, tests a replacement against it, tells you about the change and records who approved it. An unprepared one will discover the problem when your agent's answers change.

How often should I recheck a provider after signing? Review the signals on a regular rhythm, for example when you review permissions or at each renewal discussion. Confirm that documentation is current, that a fresh export still opens, that contacts are still named and that the model plan has not changed. Continuity decays quietly, so a short recurring check is better than one large review.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.