Skip to content
be-human.ai
← Back to the journal

Human oversight

A One-Page Safety Case for AI Agents: Scope, Checks and Stop Rights

A one-page safety case states what each company AI agent may do, how its work is checked and who can stop it, and stays valid as models change.

Be Human typographic cover with the title: A One-Page Safety Case for AI Agents: Scope, Checks and Stop Rights
Conceptual illustration created with AI · Be Human

A safety case for a business AI agent is a short written argument that the agent may do specific work, that its output is checked, and that a named person can stop it. For most companies, one page holds that argument well enough to decide, launch and review. The page should name the actions an agent may take, the checks that apply to them and the stop rights, and it should be written so the claims still hold when the underlying model is replaced.

Key takeaways

  • Write the case on one page with four blocks: scope, checks, stop rights and review triggers.
  • Describe permitted actions as verbs tied to named tools, not as general goals.
  • Attach each action to a check: a fixed rule, a sampled human review or an approval gate.
  • Name one person who can pause the agent, and test the pause before launch.
  • Tie every claim to something you can observe, and revisit the page when the model, tools or data change.

What belongs on the one-page safety case

A safety case is often pictured as a long technical file, but a business owner needs something shorter that can be read in ten minutes and signed by the person accountable for the process. The page is not a proof that a model is safe in general. It is a statement about one agent, in one role, with the tools and data it has today. That narrower scope is what makes it checkable.

Use these nine lines as the working template:

  • Purpose: one sentence naming the role, the team served and the decision or task the agent supports.
  • Permitted actions: a short list of verbs, each with the tool it uses, such as reading a shared folder or drafting a reply for review.
  • Forbidden actions: what the agent never does, including moving money, deleting records and contacting customers without approval.
  • Data access: which source systems, memory categories and personal data fields the agent may see, and which it may not store.
  • Approval boundaries: the actions that wait for a named person before they run.
  • Checks: how output is verified, which checks run automatically, and how often a person samples the work.
  • Stop rights: who may pause the agent, through which channel, and what happens to queued work during the pause.
  • Evidence: which logs, prompts and decision records exist, where they are kept and who may read them.
  • Review triggers: events that reopen the page, such as a model change, a new tool, a new data source, an incident or a repeated complaint.

Keep each line to something a reviewer can answer yes or no. The sentence 'the agent handles customer issues' fails that test. 'The agent drafts replies to billing questions from the approved policy text and cannot send them' passes it.

Filling in the page in one working session

Start with the role, not the technology. Choose one task that is frequent enough to matter and bounded enough to test. A single queue, a single report or a single intake form is a better starting point than a general assistant with access to everything. A first engagement with Be Human typically defines exactly one such role, together with its memory, tools, approval boundaries and an observable success criterion, so the page has something concrete to describe.

Then work through the page in this order:

  • List every tool the agent can call, including read-only ones. Read access to a customer file is still a data decision.
  • For each tool, write the verbs it allows and the verbs it must never perform. Be explicit about deletion, external messages, payments and changes to permissions.
  • Mark the actions that need a person to approve them before they run. Keep this list short at the start. Widen it deliberately later, rather than quietly narrowing approvals after an incident.
  • Decide the checks for each permitted action. Some checks are fixed rules, such as refusing an amount above a stated limit. Others are human samples.
  • Name the stop owner and a backup. Write down the exact step that pauses the agent, and confirm that someone other than the builder can perform it.
  • Decide what evidence is kept: the inputs, the tool calls, the output and the approval record. Confirm who may read it and how long it stays.

Read the finished page aloud to the person who will carry the accountability. If they cannot say what the agent is allowed to do without looking at the page, the page is not ready.

How do you check the agent's work without reviewing everything?

Checking every output usually defeats the purpose of having an agent. The practical approach is layered. Some checks are automatic and cheap: the agent may only call tools on the permitted list, outputs must match the expected format, and any amount or date outside a stated range is held for review. Other checks are sampled: a person reads a selected set of completed tasks on a regular schedule, chosen to include unusual cases as well as routine ones.

The sampling rate is a judgment about risk, not a fixed rule. An agent that drafts internal summaries can be sampled lightly. An agent that touches customer records or external messages needs a denser sample and a higher approval boundary. The guide to sampling audits for agent work explains how to set that rate and how to record what the reviewer found, which is the evidence a later reviewer will ask for.

Tools deserve particular attention because an agent that reads external text can be steered by that text. The OWASP Top 10 for LLM Applications lists excessive agency as a risk category, meaning an agent doing more than its design intended because it was given broad permissions or trusted content too readily. The safety case should therefore state that tool permissions are the smallest set the task needs, and that content from emails, web pages or uploaded files is treated as data, not as instructions.

Who can stop the agent, and how fast?

Every safety case needs a stop right that a person can use without help from the team that built the agent. Name the role, not only the individual, so the right person is covered during leave or travel. Write down the channel: a switch in the admin console, a message to a named mailbox, or a documented procedure that revokes the agent's tool credentials. Test the stop before launch and record the result on the page.

A stop should be specific about what it halts. Pausing new actions is different from cancelling work already in progress, and deleting logs is almost never the right response. The page should say what happens to queued tasks, whether the agent keeps its memory, and who decides when it restarts. The kill switch guide for business agents separates halting actions from preserving the record.

Illustrative scenario: a support agent drafts replies to billing questions from the approved policy text, and a staff member notices two drafts quoting a refund rule that was retired recently. Under the page, the operations lead pauses the send permission but not the drafting, so the queue is preserved. The team samples the recent batch, finds the stale source, removes it from the agent's memory, and records the incident and the change. The agent restarts only after the operations lead approves the sample. This scenario is illustrative. It shows how the page is meant to work, and it does not describe a recorded event.

Approval gates interact with stop rights. If a person must approve every refund above a limit, that person is also the first control when something looks wrong. The piece on where people should sign describes how to make that signature meaningful rather than automatic.

Keeping the case valid as models change

The page must survive a model change. If it names a particular model, it becomes false the day the vendor changes the default. Write the claims about behavior instead: what the system is permitted to do, which checks apply, which evidence exists, and who can stop it. The model name belongs in an appendix that is updated whenever the model changes, with a note of the date of the change and the evaluation that followed.

The NIST AI Risk Management Framework organizes this work around governing, mapping, measuring and managing AI risk. Those four activities translate well to a single page: the owner governs the scope, the team maps the data and tools, the checks measure behavior, and the stop rights manage the residual risk. Re-running the same evaluation set after each change shows whether the agent still behaves as the page says. An evaluation set is a small, fixed collection of tasks with known good outcomes, kept separate from the live work the agent handles.

Be Human handles this as part of managed operations. After a first engagement defines the role and the page, the service can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. The page is updated when something changes, rather than filed once and left to drift, and each change to tools or permissions is recorded against the same four blocks.

If you are not yet sure which task deserves this treatment, look for work that is repetitive, bounded and reviewable. Take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

Is a one-page safety case enough for a small company? It is enough when the scope is one role with a small set of tools. The page does not replace legal review, a data processing agreement or a formal risk program where those apply. It gives the owner one place to see what the agent may do, who checks it and who can stop it.

Do we need a separate page for every agent? In practice, yes, because each agent has different tools, data and stop owners. Several agents can share a common use policy, but each role needs its own page with its own permitted actions and checks. Keeping pages short makes it realistic to maintain them.

Who should sign the page? The person accountable for the business process the agent supports should sign, not only the technical builder. The builder confirms the tool list and checks, and a security or compliance reviewer can co-sign where the data is sensitive. The signature records that someone with authority accepted the scope and the stop rights.

How often should the page be reviewed? Review it on a fixed schedule, such as quarterly, and also whenever a review trigger occurs, including a model change, a new tool, a new data source or an incident. A short review that confirms the four blocks still match reality is usually enough between formal updates.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →