Context Window vs Persistent Memory: What Belongs Outside the Prompt
A bigger context window fits more text into one task, but it does not replace company memory. Learn which facts belong outside the prompt and who owns them.

A context window is the text a model can read in one request: its instructions, tool descriptions, retrieved passages, the current message and some recent history. Persistent memory is the company record an agent can check across days, channels and staff members, with a named owner, a source and a rule for correction or expiry. A larger context window does not replace persistent memory, because the window is rebuilt for each task and cannot decide which fact is current, who may see it, or who approved the change.
Key takeaways
- The context window is working space for one request. It is assembled again for every task and does not carry company knowledge forward by itself.
- Persistent memory holds facts that need an owner, a source, a version and a rule for correction or expiry.
- A longer window makes it easy to paste too much. Loading more text is not the same as loading the right facts.
- Keep permissions, approval limits and governed company facts in records and tools that the model cannot rewrite from inside a conversation.
- Judge an agent by whether it finishes a defined task with current facts, not by how much text it can hold.
Why a bigger window still forgets
Most teams first meet this limit as forgetting. The agent answers a question well on one day, then a few days later, in a different channel, it cannot find the exception a manager agreed to. The natural reaction is to raise the window size or paste the whole policy folder into the instructions. That helps with one long document, but it does not solve the underlying problem, for three reasons.
First, the window starts empty for each task. Nothing in it survives unless something writes it there again, so continuity depends entirely on what your system loads. Second, a window holds text, not authority. A policy pasted from a file last quarter looks exactly like the policy published today, and the model has no built-in way to know which one should win. Third, more text dilutes attention. Anthropic's guidance on context engineering treats the window as a limited resource to be curated, not as a storage room. A useful background on how retrieved content enters a request is in what a context window is and why your agent forgets.
Which facts should live outside the prompt
Facts that outlive a single task belong in governed memory or in the source system the agent reads from. In practice this includes the agent's role and boundaries, company policies with a version and an owner, customer and job records with their history, approval thresholds, the tools the agent may call and under what conditions, decisions with their reasons, and the glossary of internal terms that staff use every day.
Some material should stay in the instructions because it is short and needed on almost every task. Tone rules, the required structure of a reply, and the instruction to escalate when a record is missing are good examples. The test is simple. If a rule is short, stable and needed everywhere, keep it in the instructions. If it changes, has an owner, or applies only to certain customers or teams, it does not belong there.
A decision guide for each fact
Use this list before you write any fact into a prompt:
- If the agent needs it on nearly every task and it rarely changes, keep a short version in the instructions.
- If it describes a customer, job, asset or agreement that should outlive one conversation, store it as a governed memory record with a source and an owner.
- If it is an authoritative document, such as a signed policy or an approved service description, keep the document in its system of record and let the agent retrieve the current version by identifier.
- If it is a permission, approval limit or spending boundary, enforce it in the tool layer and access rules. Memory can inform the agent, but it should never be the only control.
- If it is one staff member's working preference, scope it to that person or team rather than the whole company.
- If nobody can say who owns it, do not store it yet. Log it as an open question first.
Illustrative scenario: a return question after a policy change
The following is an illustrative scenario, not a report of a real case. A customer support agent for a furniture retailer is asked whether a chair bought recently can still be returned. The policy team changed the return window last week. If someone pasted an older policy document into the instructions, the agent will answer confidently from an outdated version, and nothing in the conversation will reveal the mistake.
The governed approach works differently. The return policy is stored as a document with an identifier, a published version and a named owner. The agent retrieves the current version, quotes that identifier in its reply, and escalates when the purchase falls into an exception category that the record does not yet cover. A staff member can see exactly which version was used and check it in seconds. The model did not become smarter. The fact simply had a home outside the prompt, with an owner who can change it. The practice of naming the exact document version is described in how to cite the exact document version.
A plan to separate working context from memory
Follow this sequence for one role before you expand to others:
- Write the task the agent must finish in one sentence, and list the decisions it makes along the way.
- List the facts each decision needs and mark each one as instruction, governed memory, source document or live system value.
- Give every governed fact an owner, a source, a date of last review and a rule for correction.
- Define who may create, correct or expire each record. The agent may propose changes, but a named person or a reviewed process should approve them.
- Set a loading rule: for each task, load only the records the decision requires, and log which ones were loaded.
- Test the role with questions whose answers changed recently, and check that the reply uses the current record and says so.
The loading rule matters most. Teams that skip it end up loading everything again and rediscover the same dilution problem. A deeper treatment of choosing what to load for each task is in context budgets for AI agents.
How Be Human handles this in managed operations
Be Human treats the memory layer as part of the first engagement, not as an afterthought. A first engagement defines one useful role, the memory and tools that role needs, its approval boundaries and an observable success criterion. Company facts are separated into instructions, governed records and source documents, each with an owner. Permissions live in the tool layer, so the agent's access can be reviewed without reading its conversations. In managed operations, the same structure is maintained over time through monitoring of what the agent loads and answers, recurring evaluations against the defined criterion, permission reviews, integration maintenance and controlled changes when a policy or a system moves. The goal is that the company can point to where each fact came from and who is allowed to change it. The NIST AI Risk Management Framework is a useful reference when assigning those responsibilities.
What the window should never decide
Text that enters the window from outside, such as an email, a web page or a customer message, can contain instructions. Treat that text as data, not as authority. OWASP's guidance on prompt injection explains why instructions hidden in retrieved content can pull an agent away from its intended behavior, which is why permissions and approvals belong in systems the model cannot edit. Before you trust a long-context agent with real work, check the following:
- Can the agent change its own permissions, owner rules or memory records from inside a conversation?
- Does every retrieved passage carry an identifier, so a reply can name its source?
- Are approval limits enforced by the tool itself, not only described in the prompt?
- Is there a named person who reviews corrections and expiries?
- Do your evaluations include questions where the right answer changed recently?
These checks separate a large window from a governed memory. If you are not sure which of your facts belong in which group, take the free Be Human AI diagnostic to find the first task worth handing to an agent.
Frequently asked questions
Does a larger context window mean my agent no longer needs persistent memory? No. A larger window lets the agent read more text during one task, but it still starts empty for the next task and cannot tell which document is current or who approved it. Persistent memory provides continuity, ownership and version control, which a window does not provide on its own.
Can I paste all of our company documents into the prompt and skip the rest? You can, but the agent will then face a large mix of current, outdated and contradictory text with no clear priority. Load only the documents a given decision needs, and keep authoritative documents in their source system so the agent retrieves the current version with its identifier.
Which facts should I move out of the prompt first? Start with facts that change, have an owner and affect real decisions, such as policy versions, customer records, approval limits and staff corrections. Keep short, stable operating rules in the instructions, and enforce permissions in the tool layer rather than in text.
How do I know whether the agent used the right fact? Ask it to name the source and version it relied on, and test it with questions whose answers changed recently. If the reply cites an outdated record or cannot name a source, the memory or the loading rule needs review before the agent takes on more work.
Your next step
See where an AI agent fits in your company first.
Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.
Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.


