Shared Responsibility Model for AI Agents: Provider vs Company

A shared responsibility model for AI agents is a written split of duties between the provider that builds and runs the agent and the company that owns the data, the decisions and the risk. In general, the provider is accountable for how the system is built, hosted, monitored and repaired, while the company is accountable for what the agent may know, touch and decide on its behalf. Problems appear where the two overlap, so every duty needs one named owner and one named backup.

Key takeaways

• Split duties across four areas: data, access, quality and incidents. Write the split down before the agent goes live.
• The company decides what the agent should know, do and be allowed to decide. The provider makes sure those decisions are enforced and observable.
• Quality is shared in a specific way: the company defines what good looks like, and the provider measures it on a schedule.
• Most failures happen in the handoffs, such as an unreviewed permission or a stale document nobody owns. Name an owner for each handoff.
• Rehearse the incident path once with a small scenario, so the first real incident is not also the first time anyone reads the plan.

Why does a shared responsibility model matter for AI agents?

Cloud computing taught a useful lesson: the vendor secures the platform, and the customer secures what it puts on the platform. AI agents follow the same logic, with one addition. An agent does not only store and process information. It reads, decides and sometimes acts, so its behavior is part of what must be owned.

Without an explicit split, each side assumes the other has it covered. The provider assumes the company reviewed which systems the agent can reach. The company assumes the provider is watching answer quality. Both assumptions feel reasonable, and both can be wrong at the same time. The agent then runs for weeks with a permission nobody approved or a source nobody maintains.

A shared responsibility model removes that ambiguity. It is not a legal document first. It is an operating agreement that answers a plain question: when this specific thing goes wrong, who notices, who acts and who decides?

It also fits inside broader risk practice. The NIST AI Risk Management Framework asks organizations to govern, map, measure and manage AI risk, and each of those verbs needs a person attached to it. If your company already has a lightweight governance habit, the split described here becomes one of its working documents. If it does not, AI governance for mid-sized companies is a reasonable place to start, but this article stands on its own.

Who is responsible for data?

Data is the area where the split is most intuitive and still most often left vague. A useful rule: the company owns the meaning of the data, and the provider owns the safe handling of it.

The company is responsible for:

• Deciding which sources the agent may use, and which are off limits.
• The accuracy and currency of those sources, such as price lists, policies and procedures.
• Classifying sensitive information, including personal data, and confirming it has a lawful basis to be processed.
• Setting retention expectations, meaning how long conversations and memory should be kept.

The provider is responsible for:

• Storing and transmitting data securely, with separation between one company's data and another's.
• Building the retrieval and memory layers so the agent uses only the approved sources.
• Disclosing where data is processed and which subprocessors are involved.
• Deleting or exporting data on request, and keeping logs that show what the agent accessed.
• Stating in writing whether your data is used to train any model. This point deserves its own clause, not a line in a marketing page.

The grey zone is staleness. Suppose the agent quotes a policy that was replaced last quarter. The error is the company's if nobody updated the document, and it is the provider's if the retrieval layer ignored a newer version that was available. A good split assigns the document to a business owner and assigns the freshness check to the provider, so each side covers one half of the same failure.

Who is responsible for access and permissions?

Access is where an agent can do real damage, because it turns a wrong answer into a wrong action. The split here is about authority and enforcement. The company grants authority. The provider enforces it technically and proves it.

The company decides which systems the agent connects to, what it may read, what it may change and which actions need a human approval first. It also holds the accounts that matter. If the agent's identity and model accounts live inside the provider's tenancy, the company has handed over more than convenience. Who owns the API keys and accounts is a decision to make deliberately, and a dedicated identity for the agent makes every action traceable to the agent rather than to a person's login.

The provider implements those decisions with the narrowest scopes that still let the agent do its job. It stores secrets safely, tests that the agent cannot step outside its boundaries, and documents every connection. The OWASP authorization guidance is a sound reference for what least privilege and deny-by-default look like in practice, and the OWASP Top 10 for LLM applications lists the agent-specific risks, such as excessive agency and prompt injection, that a permission design should anticipate.

Prompt injection is a good example of why this must be shared. The provider can harden the agent against instructions hidden in emails or web pages. The company decides how much damage a fooled agent could do, by limiting what the agent can touch in the first place. Neither side alone is enough.

Who is responsible for quality?

Quality is the duty most likely to fall between the cracks, because nobody feels they own it. A clear split looks like this: the company defines what good looks like, and the provider builds the machinery that measures it.

The company supplies real examples of good and bad outcomes, decides the acceptable error rate for each type of task, and reviews samples of the agent's work. Only the business can say that a reply is technically correct but wrong in tone for a key customer. It also decides which mistakes are tolerable and which are not, since a mislabeled internal ticket and a wrong refund promise carry very different weight.

The provider turns those judgments into repeatable tests, runs them on a schedule, and reports results in plain language. It watches for drift after any change to the underlying model, the instructions or the connected tools. Anthropic's writing on evaluating agents and OpenAI's evals guide both describe the idea: you cannot improve what you do not test, and tests should reflect your real tasks rather than generic benchmarks.

Quality also needs a reporting rhythm. A monthly note that says what was tested, what changed and what the business owner needs to decide is more valuable than a dashboard nobody opens. If you want to define the measures before launch, success criteria for an AI agent pilot walks through how.

Who is responsible when something goes wrong?

Incidents are where an unclear split costs the most, because time matters. Think of an incident as five steps: detect, contain, communicate, correct and learn. Each step has a natural owner.

• Detect: the provider, through monitoring and alerts, plus the company's staff, who report odd behavior.
• Contain: the provider, who must be able to pause the agent or cut a connection quickly, with the company authorized to demand it.
• Communicate: the company decides who is told, such as customers, regulators and leadership, because those relationships and legal duties are its own.
• Correct: the provider fixes the technical cause, and the company fixes any business consequence, such as reversing a wrongly sent message.
• Learn: both, in a short review that ends with a changed rule, test or permission.

If a duty has no named owner, the incident will assign one badly.

Illustrative scenario: a company runs an agent that drafts replies to supplier emails. One supplier message contains hidden text telling the agent to forward the latest contract terms to an outside address. Monitoring flags an unusual outbound request. The provider pauses the sending action within minutes and confirms from logs that nothing left the system. The company decides whether the supplier needs to be contacted and whether the draft that was written needs review. Afterward, the provider adds a test for this pattern, and the company removes the agent's ability to attach contract files at all. Each party did the part that only it could do.

Notice what made this work: the pause authority, the log access and the communication decision were agreed in advance. For the broader question of who is accountable across the whole agent life cycle, a RACI for company AI agents gives you a structure to assign these roles step by step.

How do you draw the line in practice? A working plan

You do not need a long workshop. A single working session with one business owner, one technical contact and the provider's lead is usually enough. Follow these steps.

• List the agent's role in one sentence, and the systems and data it touches.
• For each of data, access, quality and incidents, write three lines: what the company owns, what the provider owns, and what is shared.
• For every shared item, name a single accountable person on each side and a backup.
• Mark each permission as read, draft, or act, and mark which actions need human approval.
• Agree the pause rule: who can stop the agent, how fast, and without needing anyone's permission.
• Set the review rhythm: when quality results, permission reviews and document freshness checks are reported.
• Put the result into the contract or statement of work, so it survives staff changes.
• Rehearse one incident scenario in discussion and fix any step where two people each thought the other would act.

Revisit the document whenever the agent gains a new tool, a new data source or a new channel. Most responsibility gaps open at those moments, because the agent has changed and the agreement has not.

How does Be Human handle this split?

Be Human designs, implements and operates personalized AI infrastructure, so the split is part of the first engagement rather than an afterthought. The engagement defines one useful role for the agent, the memory and tools it needs, the approval boundaries and an observable success criterion. Those four items map directly onto data, access, quality and incident duties.

In managed operations, the provider side of the split includes monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. The company side stays with the company: it keeps authority over what the agent knows, what it may do and what counts as acceptable. The work is done remotely, in English and Portuguese, and the written split is kept as a living document so both sides can see who owns what. For a view of how the recurring work is typically organized, see what managed AI services include.

Frequently asked questions

Is a shared responsibility model the same as a service level agreement? No. A service level agreement usually covers availability, response times and support terms. A shared responsibility model covers who owns each duty, including the ones that do not appear in a technical commitment, such as document accuracy, approval of permissions and the decision to notify customers. Use both, and make sure they do not contradict each other.

Who is liable if an AI agent makes a costly mistake? Liability depends on your contract, your jurisdiction and the facts, so ask a lawyer rather than rely on a general rule. What the responsibility model gives you is clarity about cause: whether the failure came from a source the company maintained, a permission the company approved or a technical defect the provider should have caught. That clarity makes both the fix and any legal conversation faster.

Does a company without an IT team still need to hold its share? Yes, but its share is smaller and more about decisions than technology. The company still decides what data the agent can use, what actions need approval and who is told after an incident, and it can do that without engineers. A single named business owner, with the right to pause the agent, covers most of it.

How often should the split be reviewed? Review it whenever the agent gains a new tool, data source or channel, and on a regular schedule even when nothing changes. A short quarterly check is usually enough for stable setups. The goal is to catch drift between what the document says and what the agent actually does.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.