Context Budgets for AI Agents: What to Load and What to Skip

A context budget is the amount of information an AI agent may load for one task, decided before the agent runs. To set one, name the decision the task must support, load only the current facts that bear on that decision, and leave out history, drafts and unrelated records unless a stated rule brings them in. The budget turns context from a pile the model sorts for itself into a limit you can inspect, test and adjust.

Key takeaways

• A context budget is a per-task limit on what the agent may load, not a single setting for the whole deployment.
• Start from the decision the agent has to support, then work backward to the facts that would change that decision.
• Stale, duplicated and off-topic material uses the same attention as the facts you need, so it belongs outside the budget.
• Every loaded item should have a reason, a named source and a current version. Anything without those stays out.
• Test the budget with real past tasks, and review what was loaded whenever an answer is wrong.

Why a larger window does not replace a budget

A larger context window lets an agent hold more text at once, but it does not tell the agent what matters. When a task receives everything that might be relevant, the agent must choose among old notes, superseded policy wording, a draft proposal and an email from a different account. The result is often a fluent answer built on the wrong fact. Long inputs also make it harder to keep track of a single line placed in the middle, so more text can hide the one detail that mattered.

A budget forces the question a bigger window avoids: what must the agent know to act correctly on this task, and what would an experienced employee ignore? That question is an operating decision, so it should be answered by the person who owns the process, not left to the model.

Start from the decision, not the document pile

Write one sentence describing the decision or output the agent must produce. Deciding whether a rescheduling request can be confirmed without staff approval is a decision. Answering questions about scheduling is not, because it invites loading the entire calendar. Once the decision is written, list the facts whose change would alter the answer. Facts that would not change the answer belong outside the budget, even when they are accurate and interesting.

This step also reveals which material is missing. If the decision depends on an approval threshold that exists only in a manager's head, the budget cannot include it until someone writes it down. Capturing that knowledge comes before loading it.

A step-by-step plan to set a budget for one task

• Step 1: Write the decision or output in one sentence, and name the person who reviews the result.
• Step 2: List the facts that would change the outcome, and record the source of each one, such as a policy page, a contract field or a system record.
• Step 3: Set a limit for each category: standing instructions, task facts, customer or case history, and examples. Keep the total small enough that a reviewer can read everything loaded in a few minutes.
• Step 4: Give each category a freshness rule, such as the current version only, or the most recent record within a period the owner defines.
• Step 5: Exclude by default: drafts, superseded versions, personal notes, unrelated accounts, and any sensitive data the task does not need.
• Step 6: Run the agent in shadow mode on real past tasks, log what it loaded, what it used and what it ignored, then tighten or widen one category at a time.

Change one category at a time. If you change several limits together, you cannot tell which change improved the answers or caused the errors.

What to load and what to skip

Use this guide when a category is unclear:

• Load the current rule that governs the decision, along with its version or review date.
• Load the record for the specific customer, order, case or asset involved, and nothing about other records.
• Load recent events only when they can change the answer, such as an open complaint or a payment awaiting approval.
• Skip full conversation transcripts when a short, confirmed summary states the same facts.
• Skip drafts, proposals and internal debates unless the task is to review them.
• Skip anything the agent is not permitted to use for this task, even if the system makes it reachable.

Summaries deserve care. A summary keeps the outcome but can drop the reason behind a decision, and the reason is often what the next task needs. The guidance on context compaction without losing decisions covers how to keep those reasons when history must be shortened.

How stale and conflicting material uses the budget

Stale facts are the most expensive loads because they look authoritative. An old fee schedule, a retired process document or a note from a former employee can carry the same visual weight as the current policy, and the agent has no built-in way to know which one replaced the other. Conflicts make this worse. When two items disagree, the agent may blend them into a sentence that neither source supports.

A budget therefore needs two things beyond size limits: a freshness rule for each category and a named owner who can retire material. The practice of keeping a stale context from misleading an agent is mostly a review habit, and the budget is where that habit becomes enforceable. Each loaded fact should also trace back to a source of truth that the business accepts as authoritative.

Illustrative scenario: a quote follow-up agent

Illustrative scenario: a small installation company uses an agent to draft follow-ups on open quotes. The first version loaded the full quote archive, every email thread and the complete price book. Drafts sometimes cited superseded material, and the agent mixed up two customers with similar names because both appeared in the same load.

The revised budget loaded only the open quote, the current version of the price book, the last two messages in that customer's thread, and the rule stating when a discount needs manager approval. Anything else was excluded by default. Staff review of the drafts then shifted from checking whether the facts were right to checking whether the tone fit the relationship, which is the review a person should be doing. Nothing about the company changed except what the agent was allowed to read for each task.

How Be Human's managed approach handles the budget

Be Human designs the context budget during the first engagement, when one useful role is defined along with the memory and tools it needs, its approval boundaries and the success criterion it will be measured against. Managed operations then keep the budget honest over time: reviewing what each task loads, retiring stale sources, checking permissions when tools change, and keeping a log of the facts an agent used so a person can trace an answer back to its source. The work continues because the business and its documents keep changing.

If you are not sure which task to start with, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

How large should a context budget be? There is no universal size. Derive the limit from the decision the task supports, then keep the total small enough that a reviewer can read what was loaded. Adjust it after reviewing real runs rather than setting it once from a guess.

Is a context budget the same as retrieval? No. Retrieval is one method for selecting candidate material, and the budget is the rule that decides how much of that candidate material reaches the agent. You can use retrieval with a budget, and you can also load a fixed set of documents with a budget and no retrieval at all.

Who should own the context budget? The owner of the process should own it, usually a business owner or operations lead, with the agent operator maintaining the technical limits. The owner decides which facts matter and how fresh they must be, because those are business judgments rather than model settings.

How do I know the budget is too small? Watch for the agent asking for information it should already have, or for answers that cite nothing and rely on general knowledge. When that happens, check the missing fact against the decision, add it to the right category and run the same past tasks again to confirm the change helped.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.