Skip to content
be-human.ai
← Back to the journal

Context & memory

Chunk Size for AI Knowledge Bases: Keep Policies and Procedures Whole

Learn how to choose chunk size for policies, procedures and job documents so AI retrieval returns complete answers, not fragments, with a practical checklist.

Be Human typographic cover with the title: Chunk Size for AI Knowledge Bases: Keep Policies and Procedures Whole
Conceptual illustration created with AI · Be Human

Chunk size for a knowledge base AI is the length of each piece of text the system stores and returns as one unit. For policies, procedures and job documents, the right size is the smallest unit that still holds a complete rule, its condition, its exception and the document it belongs to. Start from the document's own structure, such as a section under one heading, then test whether retrieval returns something a staff member could act on without guessing.

Key takeaways

  • Split by meaning and structure first, then apply a maximum length. Reversing that order cuts rules in the wrong place.
  • A chunk should hold one complete rule with its condition, exception and owner, so the agent never answers with half of a policy.
  • Keep the heading path and document version attached to every chunk, so the agent knows which rule and which edition it is reading.
  • Test retrieval with the real questions staff ask before you tune any setting.
  • Set boundary rules per document type, and review them whenever the document's layout changes.

Why fixed-size chunks cut policies in the wrong place

Many teams begin with a fixed rule such as "split every document into blocks of the same length, with some overlap." It is easy to build and easy to explain, and it works poorly for business documents. A refund policy does not know that its exception clause starts three lines below the place where the splitter stopped. The customer asks whether a deposit is refundable after a cancellation, the system returns the opening paragraph that says deposits are refundable, and the exception that removes the right for custom orders sits in the next chunk, which never reaches the answer.

The failure is quiet. The agent sounds confident, the cited text is real, and the answer is still wrong. This is why chunk size should be judged by whether an answer is complete, not by whether the system returned something relevant. If you are new to how retrieval selects content, What Is a Context Window in AI? Why Your Agent Forgets explains why the amount of text an agent can read at once also limits how much retrieved material it can use.

How to decide the right boundary for each document

A practical rule is to ask what a reader must see together to act safely. Use that answer as the unit, then check the length. The table below is a decision guide written as a list, so you can apply it to any document you have.

  • If a sentence only makes sense with the sentence before it, keep them in the same chunk.
  • If a rule has a condition, an exception or a deadline, keep all three together.
  • If a procedure has ordered steps, keep the whole sequence as one chunk unless it exceeds what the agent can read comfortably at once.
  • If a document lists many unrelated items, such as a price list or a set of service areas, split by item and repeat the list title in each chunk.
  • If a section refers to another document by name, store the reference explicitly so the agent can retrieve the linked source too.
  • If the answer depends on who is asking, keep the eligibility rule in the same chunk as the benefit it controls.

The length cap is the last check, not the first. Choose a ceiling that your retrieval setup handles well in tests, then let structure decide the exact cut points inside it. Chunks that are too small return fragments. Chunks that are too large bring in unrelated material and dilute the match, so the agent receives noise along with the answer.

What to do with policies, procedures and job documents

Each document type has a different shape, so each needs its own boundary rule. Policies usually break into principles, scope, rules and exceptions. The safe unit is a rule with its scope and exception, plus the policy title and version. Procedures are sequences. Keep a complete procedure together when it fits, and when it does not, split only at step boundaries and repeat the prerequisites at the top of each part. Job documents, such as a technician's work order template, a shift handover form or a role description, mix fields with instructions. Keep each field with its label and its instruction, and never separate a field from the rule that says when it is required.

A chunk that answers the question is worth more than a chunk that matches the words.

Illustrative scenario: a property management company stores a tenant policy on late rent. The policy has a grace period, a notice rule, and a note that the grace period does not apply to leases signed under a specific older form. A fixed-size splitter places the grace period in one chunk and the older-form exception in another. A structure-based splitter keeps the three together under the heading "Late payments," with the policy version attached. A tenant asking about the grace period now receives the exception as well, so the agent can say that it depends on the lease form instead of promising a deadline that does not apply.

Use overlap, headings and metadata so context survives

Overlap means repeating a small amount of text between neighboring chunks. It helps when a boundary falls inside a sentence that continues onto the next block, but it is a patch, not a strategy. Overlap that is too large duplicates text and makes results repetitive. Use it narrowly, and rely on structural boundaries first.

Headings matter more than most teams expect. A chunk that says "Notify the client within the stated window" means nothing alone. The same sentence under "Warranty claims, commercial contracts, version 4" is specific. Store the heading path, the document title, the version, the owner and the effective status with each chunk. Those fields let the agent cite the source correctly and let staff check it quickly. For the citation side of this, see Cite the Exact Document Version: Sources Staff Can Check in Seconds.

The Google Cloud reference architecture for retrieval-augmented generation describes how a retrieval step feeds source passages to a model, which is the setting where these boundary decisions are made. Read it as a map of the pipeline, not as a recommendation for your exact sizes.

A step-by-step plan to set chunk size and test it

Use this sequence before you tune anything numerical.

  • Inventory the documents the agent will answer from, grouped by type: policy, procedure, job document, reference table.
  • For each type, write the boundary rule in one sentence, such as "one rule with its exception and heading path."
  • Run a sample of documents through the splitter and read the output as an employee would. Mark every chunk that would lead to an incomplete answer.
  • Collect fifteen to thirty real questions from the people who answer them today, including the awkward ones about exceptions.
  • Run those questions against the knowledge base and score each answer as complete, partial or wrong, with the reason written next to it.
  • Adjust the boundary rule where partial answers cluster, then rerun the same questions. Do not change five settings at once.
  • Set an owner for each document type and a date to review the rule when the source document changes.

Keep the test set. It becomes your regression check, so a later change to the model, the splitter or the source files cannot silently break an answer that worked before. For a fuller method, see How to test retrieval quality in an AI knowledge base. Documents also need to be clean before they are split, because a scanned file with broken headings gives the splitter nothing to work with. How to prepare company documents for an AI agent covers that preparation step.

OpenAI's evals guide describes the general practice of writing repeatable checks for model behavior, which is the same habit this test set applies to retrieval.

How Be Human handles chunk size in managed operations

Be Human treats chunking as part of the agent's memory design, not as a one-time setup. In a first engagement, the team defines the useful role, the documents it needs, and the boundary rule for each document type, then writes the success criterion as a set of real questions with expected complete answers. Managed operations then keep those checks running. When a policy is revised, the affected chunks are rebuilt, the test questions are rerun, and the owner receives a short record of what changed. Permission boundaries stay in place, so the agent sees only the documents its role is allowed to read. The same approach applies to the source files behind the knowledge base, which is why it is worth keeping those files under a clear owner and review routine, as described in AI Knowledge Base Maintenance: Owners and a Review Routine. If you are unsure which of your documents should become the agent's first task, the free Be Human AI diagnostic will help you find the first task worth handing to an agent.

Frequently asked questions

What is the best chunk size for a knowledge base AI? There is no universal number that fits every business. Choose the boundary from the document's structure, so each chunk holds one complete rule, step or field, then apply a length cap that your retrieval tests handle well. Confirm the choice with real questions before you rely on it.

Should I use overlap between chunks? Use a small overlap only where a boundary falls inside a sentence or a rule that continues into the next block. Overlap that is large creates repeated results and dilutes the match. Structural boundaries and retained headings do more work than overlap in most policy and procedure documents.

How do I know if my chunks are too small or too large? Read the chunks that the system returns for your test questions. If the answer is missing its condition or exception, the chunks are probably too small or split at the wrong place. If the answer is correct but buried in unrelated text, the chunks are probably too large or mixed across topics.

Do I need to change chunk size every time a policy changes? Not always, but the affected chunks must be rebuilt and retested. Keep the document version and effective date attached to each chunk, and rerun the same test questions after any revision. That routine catches stale answers before staff or customers see them.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →