Context Budget for AI Agents: What Each Model Call Should Carry
How to decide which facts, tool results and instructions belong in each model call so an AI agent stays accurate as its tasks grow longer.

AI agent context window management means deciding, for every model call, which facts, tool results and instructions go into the window and which stay out. Treat the window as a limited budget: each call should carry what the next decision needs, in a form the model can use, while everything else stays in storage the harness can fetch on request. A practical test: if removing an item would not change the next action, it does not belong in this call.
Key takeaways
- Build each model call from the decision it must support, not from everything the agent has seen so far.
- Keep standing instructions short and stable, and load task-specific procedures only when a task needs them.
- Reduce tool results to the fields that decide the next step, and keep the full output in the run record.
- Define a context budget for each call type, then log what was included and what was dropped.
- Rebuild context at every step of a long task instead of appending to it indefinitely.
Why more context often makes an agent worse
It is tempting to give an agent a larger window and let it sort out the rest. In practice, long windows fill with old tool output, superseded decisions, duplicated instructions and half-finished reasoning. The model then has to find the one constraint that matters among many similar-looking lines, and it may act on a status that was true three steps ago. The result is usually not a clear failure. It is an agent that is slower, more expensive to run, and less consistent from one run to the next.
Anthropic's engineering guidance on context engineering describes context as a finite resource with diminishing returns, which is a useful way to frame the work. The goal is not to maximize what the model can read. The goal is to choose the smallest set of high-signal tokens that makes the next decision reliable.
Start by writing down what each call must decide
Before touching prompts or retrieval, define the decision each call supports. A single agent may make several kinds of calls: choosing a tool, drafting a customer reply, checking whether a record is complete, or deciding whether to escalate to a person. Each of these needs a different slice of information.
- Name the one decision this call supports, in a sentence a manager would accept.
- List the facts that could change that decision, and mark each as required, helpful or irrelevant.
- Write down the constraints that must hold in every case: permission limits, approval rules and the scope of the task.
- Check which items already live in the stable instruction layer and which must be fetched for this run.
- Turn the result into named slots, such as instructions, task state, retrieved facts, tool results and recent turns, each with a maximum size.
Named slots make the budget visible. When a call behaves badly, you can ask which slot held the wrong information, instead of rereading a transcript that runs for pages.
Sort context into four layers
Most business agents work better when context is split by how often it changes and where it comes from. Mixing these layers is a common source of stale or contradictory input.
- Stable identity and rules: the role, the boundaries, the escalation conditions. Keep this short, versioned and reviewed like any other change.
- Task state: what has been completed, what is pending, which outputs were verified, and which questions are still open.
- Retrieved reference material: policies, product details or past records fetched for this call only, with a source reference attached.
- Tool results: compact, typed and limited to the fields the decision uses.
Procedures deserve their own treatment. A long checklist for handling a refund or a records request should not sit in every call. Package it as a skill the harness loads when the task matches. The approach is described in AI Agent Skills: Reusable Procedures Instead of Long Prompts.
Shape tool results before they enter the window
Raw tool output is usually the largest source of waste. An API may return a full event history, a complete customer record or a long log, when the agent needs only the latest status and the one field that controls the next step. Before a result enters the window, the harness should keep the identifiers, status, timestamps, errors and the specific fields the decision needs. The full payload stays outside the window, referenced by an ID, so a later call can retrieve it if a question arises.
The same principle applies to naming. Clear tool names and schemas make it easier to know what a result contains before reading it. For the details, see Tool Result Truncation for AI Agents: Keep the Fields That Decide.
Illustrative scenario: an operations agent handles a delayed supplier shipment. The carrier returns a long event history with dozens of scan records. The call that decides whether to notify the customer needs only the latest status, the promised delivery window and the customer's notification preference. The harness keeps the full history in the run record. If the customer disputes the delay later, a separate call fetches the older events and works from them.
Re-select context at every step
Long tasks break when the agent keeps appending to its own history. Each step adds tool output, partial reasoning and intermediate drafts, and by the middle of the task the window holds more noise than signal. A better pattern is to rebuild the context for each call from the task state, not from the transcript.
In practice, this means a compaction rule that runs between steps. Completed steps become short verified facts, such as "invoice matched to purchase order" with a reference. Superseded tool output is dropped. Open questions stay visible. The agent always knows what is settled and what is still unknown, without rereading everything that led there. Anthropic's guidance on long-running agents makes a similar point about carrying forward a clear, structured state between sessions. Checkpoints that let a task resume from verified state are covered in Checkpoints for Long-Running AI Agents: Resume From Verified State.
Measure what the agent is missing, and treat untrusted content with care
A context budget is only useful if you can see whether it works. Log, for each call, the slots that were filled, their approximate size, and the items that were dropped or truncated. When an agent makes a poor decision, the log should show whether the missing fact was never retrieved, was cut by a limit, or was present but ignored. Evaluation sets help here too: a small collection of realistic tasks, scored against the success criterion for the role, shows whether a change to the budget improved or harmed results. OpenAI's evals guide and Anthropic's writing on evaluating agents both describe this kind of recurring test.
There is also a security side. Retrieved documents, emails and tool outputs can contain text written to manipulate the model. If that text enters the window with the same authority as your instructions, the agent may follow it. Keep untrusted content in clearly labeled slots, do not let it override the instruction layer, and restrict what the agent can do after reading it. The OWASP prompt injection guidance covers the main defenses.
How Be Human handles context budgets in managed operations
In a Be Human engagement, the context budget is part of the first design, not an afterthought. The first engagement defines one useful role, the memory and tools it needs, its approval boundaries and an observable success criterion, and it records which context layers each call type receives. The instruction layer is kept under version control and changed only through review. Ongoing operations include monitoring of context size and dropped fields, recurring evaluations against the success criterion, permission review when a new tool starts supplying data, and controlled improvements when the logs show a repeated gap. If you are unsure which task would benefit most from this discipline, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.
Frequently asked questions
What is a context budget for an AI agent? A context budget is the set of limits that decide what enters each model call, including standing instructions, task state, retrieved facts and tool results. It makes the window a deliberate choice rather than a log of everything that happened. Each call type gets its own budget, sized to the decision it supports.
How do I know which facts to include in a model call? Start from the decision the call must make, then list the facts that could change that decision. Include the required facts, add helpful ones only when space allows, and leave out anything that would not alter the next action. Review the logs over time to see whether the agent lacked information or was distracted by it.
Should I put all my business rules in the system prompt? Keep the rules that govern identity, scope and escalation in the stable instruction layer, but move long procedures into separate, versioned skills that load when needed. Very long instructions dilute attention and become hard to maintain. Short, reviewed rules are easier to test and to change safely.
How often should an agent rebuild its context during a long task? Rebuild it at each step from verified task state rather than appending the full transcript. Summarize completed steps into confirmed facts, drop superseded tool output, and keep open questions visible. This keeps each call focused and makes it easier to resume the task after an interruption.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


