What Is a Context Window in AI? Why Your Agent Forgets

A context window is the amount of text an AI model can read and consider at one time while it produces a response. Think of it as a desk: everything on the desk is available to the model, and anything that is not on it does not exist for that response. An agent "forgets" when needed information is dropped, shortened or never placed on the desk, not because it lost something it once learned.

Key takeaways

• The context window is working space for a single response, not a storage system. Anything the agent must know next week has to live somewhere else.
• Agents forget for ordinary reasons: the window fills, old material is cut or summarized, a session ends, or a key instruction is buried under noise.
• A larger window helps but does not replace memory or retrieval. It can raise cost, slow responses and bury the important fact among irrelevant ones.
• Use the window for the current task, memory for durable facts about people and decisions, and retrieval for large or changing reference material.
• Forgetting is a design problem you can test for, assign to an owner and fix, instead of something to accept.

What is actually inside a context window?

Every time an AI model answers, it receives a bundle of text and replies to that bundle. The bundle is the context. It usually contains the standing instructions for the agent, the conversation so far, the results of any tools the agent called, any documents that were fetched, and the new request. The model reads all of it together, then writes the next piece of output.

The size limit is counted in tokens, which are small fragments of words. You do not need to count them yourself. What matters in business terms is that the limit is finite and shared. Instructions, history, documents and the answer itself all compete for the same space.

The second point matters more: the model does not keep anything between calls on its own. If a conversation feels continuous, it is because the surrounding system sends the earlier text back in each time. When the system stops sending something, the model has no trace of it. That is the mechanism behind almost every complaint of the form "we told it this yesterday."

Why does an agent forget in the middle of a job?

Forgetting rarely has a single cause. In practice, a handful of patterns account for most cases:

• The window fills up. A long thread, a large attachment or a verbose tool result pushes earlier material out or forces the system to cut it.
• The history is compacted. To make room, the system summarizes older turns, and the summary drops a detail that later turns out to matter, such as an exception a client asked for.
• The session ends. A new conversation starts with an empty desk unless something deliberately carries information across.
• The instruction is buried. A rule stated once near the start of a long exchange can carry less weight than a recent, more prominent message.
• The information was never stored. A person said it in chat, everyone assumed it was remembered, and no one wrote it anywhere durable.

The compaction case deserves attention because it looks like memory but behaves like editing. If you run long tasks, read AI context compaction without losing important decisions to see how to keep decisions and constraints intact when history is shortened.

Does a bigger window fix the problem?

A larger window gives you more room, and for some tasks that is exactly what you need, such as reading one long contract in a single pass. But as a general answer to forgetting, it falls short, for four reasons.

First, more text is not the same as more attention. When a window is crowded with loosely related material, the model has more to sort through, and the fact that matters can be treated as one among many. Anthropic's engineering guidance on context describes context as a finite resource that should be curated, not simply filled (see the sources below).

Second, cost and speed. Every call carries the whole bundle, so a habit of stuffing everything in makes each response heavier. Some providers offer ways to reuse repeated portions of a prompt, which helps, but it does not change what belongs in the bundle.

Third, staleness. If you paste an old policy into the window, the model will read it as current. A big window can hold outdated and current versions side by side with no signal about which one wins.

Fourth, boundaries. Putting more data in the window also means putting more data in front of the model for that request. If some of it belongs to a different client or a different permission level, the window is the wrong place for it.

The discipline of choosing what goes in, in what order and in what form is called context engineering. A practical overview is in Context engineering for business AI agents.

When should memory take over, and when should retrieval?

The useful question is not "how do we make the window bigger?" but "where should this piece of information live, and how does it reach the window when needed?" Use this decision guide, one kind of information at a time.

• Needed only for the current task, and short: keep it in the window. Examples: the request itself, the file being edited, the last few steps taken.
• A durable fact about a person, client or team that should shape future work: store it as memory. Examples: a client's preferred channel, a standing approval rule, a decision made last month and why.
• Large reference material that changes or is too big to include whole: use retrieval. Examples: policy documents, product manuals, procedures, past project notes. The system searches, then places only the relevant passages in the window.
• A fact that lives in a system of record and must be exact: look it up live from that system through a tool instead of copying it. Examples: order status, stock level, account balance, appointment times.
• A fixed way of working that should apply every time: put it in the agent's standing instructions or a reusable procedure, not in conversation history.

Two distinctions prevent most mistakes. Memory is for things the agent should carry as its own knowledge of the people and decisions it works with. Retrieval is for looking things up in a body of documents. If you mix them, you get an agent that either "remembers" a document that was replaced or re-reads a preference every time without ever applying it. For the first distinction, see AI memory vs chat history. For exact facts, see How to give an AI agent a reliable source of truth.

Retrieval has its own quality bar. It works by searching your content and supplying the best matches to the model, as AWS explains in its overview of retrieval-augmented generation. If the search returns the wrong passage, the agent answers confidently from the wrong text, which looks exactly like forgetting.

What does this look like in a real workflow?

Here is an illustrative scenario, not a real case.

An operations manager at a services company asks an agent to prepare a monthly summary for a client. In the first week, the manager mentions that this client wants figures grouped by region and must never receive draft numbers. Three weeks later, a new conversation begins and the agent produces a summary grouped by product, with a draft figure included.

Nothing was wrong with the model. The preference was said in a chat that has since ended, and no memory was written. The grouping rule and the draft-numbers rule were never stored in a place the agent consults.

The fix has three parts. The grouping preference and the "no drafts" rule become a short client memory entry with an owner and a date. The monthly figures come from the reporting system through a read-only lookup, not from pasted text. The reporting procedure becomes a standing instruction that the agent follows each month. After that, the window only needs to hold the current request and the few items fetched for it.

How do you diagnose forgetting in your own agent?

When someone says the agent forgot, resist the urge to say it is a model limitation. Work through these steps instead.

• Step one: reproduce the failure and write down exactly what the agent should have known and when it was told.
• Step two: check whether that information was ever stored anywhere durable, or only existed in a chat.
• Step three: if it was stored, check whether it was fetched for the request that failed. Look at what was actually placed in the window.
• Step four: if it was fetched, check whether something newer or conflicting appeared beside it, or whether it was cut or summarized.
• Step five: decide the right home for the fact using the decision guide above: window, memory, retrieval or live lookup.
• Step six: add the failing case to a small set of recurring checks, so the same gap is caught early next time.

If the culprit is retrieval, test it directly with How to test retrieval quality in an AI knowledge base. If the culprit is old or contradictory content, the problem is in the knowledge, not the window, and it needs an owner who maintains it.

How does Be Human handle context and memory?

Be Human treats the context window as one layer of an agent's infrastructure, not as the place where knowledge lives. In a first engagement, we define the role the agent plays, then decide for each kind of information whether it belongs in the window, in memory, in retrieved documents or in a live lookup from a system of record. Memory entries have scope and an owner, and the agent's permissions and approval boundaries stay under the client's control. In managed operations, we monitor what the agent actually retrieved and remembered, run recurring checks for known failure cases, and make controlled improvements when something is missing or stale.

Frequently asked questions

What is a context window in AI? It is the limit on how much text a model can take in and consider while generating one response. It includes instructions, conversation history, tool results, retrieved documents and the new request. It is working space, not long-term storage.

Why does my AI agent forget what I told it earlier? Usually the earlier text is no longer being sent to the model, either because the session ended, the history was cut or summarized, or the information was never saved anywhere durable. The model itself does not keep a private copy between calls. Saving the fact as memory, or placing it in a document the agent can retrieve, solves most cases.

Is memory the same as a longer context window? No. A longer window lets the model read more at once for a single response, while memory is stored information that the system brings back into the window when it is relevant. Memory persists across sessions and can be reviewed, corrected or removed, which a window cannot do on its own.

When should I use retrieval instead of memory? Use retrieval for large or changing reference material, such as policies, manuals and past project notes, where the agent should find the relevant passages on demand. Use memory for compact, durable facts about people and decisions that should shape future work. For exact values held in a business system, a live lookup is safer than either.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.