Skip to content
be-human.ai
← Back to the journal

Testing & reliability

Error Budgets for AI Agents: Setting Acceptable Failure Rates per Task

Learn how to set an AI agent error budget for each task, so you know when to slow autonomy, when to add review, and when to expand what the agent can do.

Be Human typographic cover with the title: Error Budgets for AI Agents: Setting Acceptable Failure Rates per Task
Conceptual illustration created with AI · Be Human

An AI agent error budget is the amount of failure a specific task can absorb over a set period before the team reduces the agent's autonomy on that task. You set it per task, starting from what a wrong action costs the business and how much review that task needs. You then measure failures against the budget every week, and the budget tells you whether to keep going, add a check, or pull authority back.

Key takeaways

  • An error budget belongs to one task, not to the agent as a whole.
  • Start from the cost of a wrong action, not from the model's average quality.
  • Spending the budget triggers a pause in autonomy and a review of the cause, not a search for someone to blame.
  • Expand authority only after a clean run inside a defined window, and change one permission at a time.
  • Keep the budget in the same weekly review as cost, incidents and change history, so it does not drift out of sight.

Why one accuracy number misleads owners

Most teams first ask how accurate an agent is. That question is reasonable, but a single accuracy figure hides the thing an operations leader needs to decide. An agent that drafts internal summaries can be wrong often and still be safe, because a person reads every draft before it leaves the building. An agent that issues refunds, changes a delivery address or sends a message to a customer can be wrong rarely and still cause serious damage, because nobody sees the action before it lands.

An error budget moves the conversation from "how good is the agent" to "how much wrong work can this task tolerate, and what happens when we reach that limit." That framing matches how operations teams already manage service levels for software and staff. The idea is borrowed from site reliability practice, where a target for reliability leaves room for a known amount of failure, and the room is spent deliberately. For an agent, the same logic works if you define failure in business terms and write the limit down before the agent starts working.

The NIST AI Risk Management Framework describes the broader discipline of mapping, measuring and managing AI risk over the life of a system. An error budget is one practical way to make that discipline concrete for a single workflow, because it turns a general commitment to manage risk into a number, a owner and a trigger.

Start from the cost of a wrong action

Before you choose any threshold, write down what a wrong action costs in each task the agent might perform. Cost here is not only money. It includes a customer who receives a wrong answer and calls back, a staff member who has to undo a change, a regulatory exposure, a reputational problem, or a decision made on bad information. Group the tasks into three or four tiers by the worst realistic outcome of one error, not by how often the task runs.

A useful test is to ask what the team would do on Monday morning after one bad action. If the answer is "fix a line of text," the task sits in a low tier. If the answer is "contact the customer, reverse a payment, review the logs and explain the mistake to a manager," the task sits in a high tier. High-tier tasks get a small budget, a short measurement window and a human approval step. Low-tier tasks can carry a larger budget and more autonomy, because the cost of learning from their failures is modest.

This step also protects you from a common mistake: setting one budget for the whole agent. An agent that answers questions, updates records and sends emails should not have a single failure allowance. Each capability has a different consequence, so each capability needs its own line.

Set the budget for each task, step by step

Use this sequence for every task you plan to hand to an agent. It works for a first pilot and for a mature deployment that is being widened.

  • Name the task in one sentence, with the input it receives and the action it takes at the end.
  • Define what counts as a failure for that task. Write it as an observable condition, such as "sends a message with a wrong order status" or "applies a discount outside the approved list."
  • Separate minor failures from serious ones. A typo and a wrong refund are not the same event, and they should not draw from the same budget.
  • Assign the task to a cost tier, and write down the tier's worst realistic outcome in one line.
  • Choose the measurement window. High-tier tasks usually need a shorter window, so that a problem appears in days rather than months.
  • Set the budget as the number of serious failures the team will accept inside that window, and a separate, looser allowance for minor ones.
  • Name one owner who reads the budget each week and has authority to pause the task.
  • Decide in advance what happens at the limit: pause the task, route every action to human approval, or reduce the task to a read-only mode.
  • Record the budget next to the task in your operating notes, with the date it was set and the reason for the number.

The last two steps matter most. A budget that does not trigger an action is a report, not a control. Decide the response before the first failure, because the team will be tempted to make exceptions when the agent is doing useful work.

What a budget looks like in an illustrative week

The following scenario is illustrative and does not describe a real client or result. A small service business uses an agent to answer customer questions about order status. The agent can read the order system and send a reply, but it cannot change an order or issue a credit. The team classifies this as a moderate tier, because a wrong status message causes a phone call and some frustration, but no money moves.

The owner sets a window of one week and writes a budget that allows a small number of minor failures, such as an unclear phrasing or an outdated tracking link, and no serious failures, defined as a status that contradicts the order system. During the first week, the agent produces several minor failures and one serious one, when it reads a canceled order as active because a field was updated after the lookup. The budget is spent.

The response is not to punish the agent or the staff. The team pauses the outbound replies for that order type, routes them to a person for review, and opens an investigation into the timing gap. They add a fresh lookup before sending and a check that compares the status with the latest record. Only after two clean weeks does the owner allow outbound replies again, and the permission change is written in the change log before it takes effect.

When to slow autonomy and when to expand it

A budget is most useful as a decision guide. Each week, the owner should be able to choose one of four responses for each task, without debate, based on the position of the budget.

  • Hold: the task is inside its budget, but the window is not yet long enough to show stability. Keep the current permissions and continue measuring.
  • Slow: the task has used most of its budget or has produced a serious failure. Add human approval, narrow the inputs, or move the task to a read-only mode until the cause is understood.
  • Pause: the budget is exhausted, or a failure has affected a customer, money or a legal obligation. Stop autonomous action for that task and run the incident process.
  • Expand: the task has stayed inside its budget for the full window, the failure types are well understood, and the next permission has a clear, bounded scope. Widen one permission, then measure again.

The discipline here is in the expansion step. Teams often widen authority because the agent feels reliable, or because a stakeholder wants the time savings. A budget gives a neutral answer. If the window is clean and the failure definition still matches what the business cares about, the expansion is justified. If not, it waits. For a deeper look at how quality can slip without a single dramatic failure, see AI Agent Drift: How to Spot Quality Decline Over Time, which covers the slow changes that a budget can miss if nobody reviews the trend.

How Be Human runs error budgets in managed operations

Be Human designs and operates personalized AI infrastructure in which each agent keeps its identity, memory, channels, tools, permissions and operating context under human control. In a first engagement, the work starts by defining one useful role, the memory and tools it needs, the approval boundaries, and an observable success criterion, which is the natural place to set the first error budget. In managed operations, the same budget is reviewed alongside monitoring, recurring evaluations, permission review and incident response, so that a spent budget leads to a specific change in permissions and a logged decision rather than an informal adjustment. Engagements are delivered remotely, in English and Portuguese.

Keep the budget honest, and find the first task to hand over

A budget only stays useful if the failure definitions stay current. Review them when the business changes a policy, when a new channel is added, or when the model behind the agent is updated, because each of those can make an old definition too narrow or too loose. Keep a short record of every change to a budget and every expansion of authority, so that a future reviewer can see why a number was chosen. For a practical way to keep that record readable, see Superintelligent Agents Need a Change Log Owners Can Read. Before any task reaches write access, many teams also run the agent in observation mode first, as described in Run an AI agent in shadow mode before giving it authority.

If you are not yet sure which task deserves a budget first, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

What is an AI agent error budget? It is the amount of serious and minor failure a specific agent task can produce in a defined period before the team changes how much the agent can do on its own. The budget is set per task, based on what a wrong action costs, and it is reviewed on a regular schedule. Its main value is that it turns the question of acceptable risk into a number with a named owner and a predetermined response.

How do I choose the number for the budget? Start with the tier of the task, which depends on the worst realistic outcome of one error, and then choose a small allowance for high-tier tasks and a larger one for low-tier tasks. Use a measurement window long enough to show a pattern but short enough to catch problems early, often a week for active tasks. Write the number and the reason for it next to the task, and revisit it after the first few windows, when you have real data.

How is an error budget different from an accuracy target? An accuracy target describes the share of outputs that meet a standard, and it treats every error as equal. An error budget separates minor and serious failures, ties each class to a business consequence, and specifies what happens when the limit is reached. Accuracy can be a useful input to the budget, but it does not tell you when to pause authority or when to expand it.

When should I give an agent more autonomy? Expand authority only when the task has stayed inside its budget for the full measurement window, the failure types are well understood, and the next permission has a narrow, clearly bounded scope. Widen one permission at a time, log the change, and measure again before widening another. If the budget is close to its limit, or if the business process has changed recently, hold the current level until the picture is clear.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →