Skip to content
be-human.ai
← Back to the journal

Context & memory

Per-Task Context Loading for Agents: Choosing Memory and Records

A repeatable method to decide which memory, policies and records an AI agent loads for each task, so answers stay focused and traceable.

Be Human typographic cover with the title: Per-Task Context Loading for Agents: Choosing Memory and Records
Conceptual illustration created with AI · Be Human

A context budget is the fixed set of memory, policies and records an agent may load before it answers one task. For each task, decide what the agent needs in order to be correct, what it should not see, and which evidence must travel with the answer. Scoped this way, the agent stays on the question in front of it, and every answer can be traced to the exact material it used.

Key takeaways

  • Treat each task as its own load decision, not as a standing pile of company knowledge.
  • Load the smallest set of memory, policies and records that can produce a correct answer, then widen the set only with a stated reason.
  • Separate what the agent may read from what it may do, and review both for every task.
  • Log which items were loaded for each answer, with versions, so staff can audit it later.
  • Review the budget on a schedule, because policies and integrations change faster than agent setups do.

Why more context can make an agent worse

It is tempting to give an agent everything the company has written down, on the theory that more material means fewer gaps. In practice, a long load mixes current policy with retired policy, a customer's history with an open issue, and a sales playbook with a support rule that contradicts it. The agent then has to choose among material that was never selected for this question, and its answer may rest on a document that was not the governing one at all. The limits of a context window are covered in What Is a Context Window in AI? Why Your Agent Forgets, but the practical point is simpler: capacity is not the same as relevance.

Anthropic's engineering guidance on context engineering treats context as a finite resource that has to be curated, not a store that can be filled without cost. The same idea applies inside a company. Attention is limited, and every extra item competes with the one that matters.

Start from the task, not the document library

Write the task list before touching any file. A useful task has a verb, a decision and an owner: answer a delivery date question for a customer, draft a renewal reply for an account manager, or classify an incoming complaint for the support lead. Each one needs a different set of facts to be correct. A question about order status does not need the brand voice guide, and a complaint classification does not need last quarter's forecast.

The easiest way to build the list is to collect the questions staff already answer by hand. Ask the people who handle them which documents they open, which systems they check and which colleague they call when something is unclear. Those steps are the real budget. Anything the team never touches is a candidate for exclusion.

Keep customer facts apart from general policy. A customer fact is about one party and changes with that party's history. A policy applies to everyone and changes on a known date. They should be loaded through different paths, because a wrong customer fact and a wrong policy cause different kinds of damage and need different owners.

A five-step method for each task

  • Name the task and the decision it supports in one sentence.
  • List the facts the answer depends on, and mark each as stable (policy, process), changing (order status, open ticket) or personal (customer preference, account history).
  • Assign one source of truth and one owner to every fact, so the agent never has to choose between two versions of the same thing.
  • Classify each item as always loaded, loaded when a stated condition matches, or excluded for this task.
  • Set a ceiling on the number of items, and write down what happens at the ceiling: narrow the query or hand the case to a person.
  • Record the identifiers and versions of everything loaded, stored next to the answer.

The method is deliberately plain. It can be run on paper for one task before any software is involved, and most teams find the gaps at step two, when someone asks where a fact actually lives and nobody can say for certain.

Illustrative scenario: a delivery date question

This scenario is illustrative and does not describe a real client. A customer asks whether an order will arrive before a holiday weekend. Under the method, the agent loads the current order record, the shipping policy section that covers cutoff times, and the carrier schedule marked as current. It does not load the full sales playbook, the previous year's shipping policy, or notes from unrelated accounts. The answer cites the policy version and the timestamp of the order status.

If the order record is missing, or older than the carrier schedule, the agent does not guess. It hands the case to the operations queue along with the items it did load, so the person picking it up starts from the same evidence. The difference becomes visible later. When a policy is revised, staff can find every answer that relied on the earlier version instead of searching through conversations by hand.

Separate what an agent may read from what it may do

Loading a record is a read, and reading is not the same as acting. An agent that can see a refund table is not thereby allowed to issue a refund. For each task, keep two lists: the read set, which is the context budget, and the action set, which is governed by approval boundaries written separately. OWASP's authorization guidance makes the broader point that access decisions should be made deliberately, not inherited from wherever a file happens to sit. Role-based limits on what each function can load, as opposed to what the agent can change, are a practical way to enforce that split.

Retrieved text is also an input that can carry instructions. OWASP's prompt injection prevention guidance recommends treating retrieved content as data to be read, not as commands to be obeyed. A budget that includes an uploaded customer file or a web page should state which parts of that material may inform an answer and which may never trigger an action.

Keep a trace of what every answer loaded

A load decision is only useful if it can be checked after the fact. For each answer, store the task identifier, the item identifiers, the version of each item, the time it was loaded and the reason it was loaded. Observability practice in software uses traces and spans to follow one request through several steps, and the same discipline fits an agent: one answer, one trace, with each load visible as its own step. The AI agent memory audit guide shows how to review those records once they exist.

When a customer disputes an answer, the trace turns a vague argument about what the agent knew into a short list of facts to verify. The trace also shows which items were loaded but never used. That pattern is the clearest signal that a budget is too generous and that the excluded list needs to grow.

How Be Human handles context budgets in managed operations

Be Human's first engagement defines one useful role, the memory and tools that role needs, its approval boundaries and an observable success criterion. The context budget is written as part of that definition, task by task, with a named owner for each source. In managed operations, the load rules are checked through recurring evaluations, adjusted when a policy or integration changes, and included in permission reviews. When an answer relied on something stale, the investigation starts from the trace, not from someone's recollection of what the agent had seen.

Review the budget as the work changes

Budgets drift. A policy gets revised, a carrier changes its schedule, or a new product line adds records that nobody thought to exclude. Set a review date for each task, name the owner who signs off, and retire any item that no trace has used for a long time. Treat a budget change like any other change to a process: write it down, state the reason, and keep the previous version so a past answer can still be explained. Where a task depends on documents that change often, the guidance on retrieval scope by role helps decide who should see each revision.

If you are not sure which task should go first, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

How big should an agent's context budget be? No single size fits every task. Start with the smallest set of memory, policies and records that can produce a correct answer, then add an item only when a test or a trace shows a specific gap. Measure the budget by the items that matter, not only by how much text fits into the window.

Should the agent load the whole policy manual for each question? Usually not. Keep the full manual in a versioned store, load the section that governs the decision, and cite that section by version so staff can check it. Loading the entire manual makes the governing rule harder to find and harder to cite afterward.

What should happen when the agent needs more context in the middle of a task? Allow a defined retrieval step with its own log entry, limited to the sources already approved for that task. Add a stop rule as well: if the needed item is missing, out of date or outside the role's access, the agent hands off to a person along with the items it has already loaded.

Who decides what goes into a context budget? The task owner decides which business facts matter, the operations lead decides how they are loaded and logged, and whoever is responsible for data access approves what a role may read. Write these decisions down, review them on a schedule, and keep earlier versions so past answers can still be explained.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →