Chunking strategy for retrieval-augmented generation
How chunk size, overlap, document structure and metadata shape retrieval quality in production RAG systems — and how to measure your choices.
Retrieval-augmented generation (RAG) lets a language model answer questions from your own documents. When RAG systems disappoint, the cause is rarely the model — it is usually retrieval. And the single biggest lever on retrieval quality is how you split documents into chunks before embedding them.
This guide covers the trade-offs behind chunk size, overlap and structure, the metadata that makes chunks findable, and how to test your choices instead of guessing.
Why chunking matters so much
An embedding compresses a passage into a single vector. If a chunk is too large, that vector averages several topics together and matches none of them precisely. If it is too small, it loses the context needed to be understood — "the fee is waived in this case" means nothing without the case. Chunking decides what a "unit of meaning" is for your retriever.
Choosing a chunk size
There is no universal right size, but there are reliable patterns:
| Content type | Typical starting point | Reason |
|---|---|---|
| FAQs, support articles | One question-answer pair per chunk | Natural self-contained units |
| Policies, contracts, regulations | One clause or sub-section, roughly 200–500 tokens | Answers usually live in a single provision |
| Long-form reports | Section-aware chunks of 400–800 tokens | Ideas develop across paragraphs |
| Tables and structured data | Row groups with repeated headers | Values are meaningless without column names |
Treat these as hypotheses to test against your own questions, not as rules.
Overlap: a safety net, not a strategy
Overlapping consecutive chunks by 10–20% reduces the chance that an answer is split across a boundary. But overlap inflates your index, produces near-duplicate results, and cannot rescue chunks that cut through the middle of a table or list. Use modest overlap with fixed-size chunking; prefer structure-aware splitting where documents have structure.
Respect document structure
The best chunk boundaries are the ones authors already created. Structure-aware chunking splits on headings, sections, list items and table boundaries, and only falls back to token counts when a section is too long.
- Carry the heading path into every chunk — "Refund Policy › International orders › Exceptions" — so a short passage keeps its context.
- Keep lists and tables intact where possible; split long tables by rows and repeat the header.
- Strip boilerplate such as navigation, footers and cookie banners before chunking web pages.
Metadata makes chunks findable
Pure vector similarity is rarely enough in production. Attach metadata to every chunk and use it to filter, boost or explain results:
- source document, URL and section heading;
- document type, product, region or business unit;
- effective date and version, so superseded policies do not outrank current ones;
- access level, so users only retrieve what they are allowed to see.
Hybrid retrievalCombining vector search with keyword search (such as BM25) catches exact terms — product codes, section numbers, names — that embeddings can miss. A re-ranking step on the combined results often gives the biggest single quality gain.
Advanced patterns worth knowing
- Small-to-big retrieval: embed small, precise chunks for matching, then pass the larger parent section to the model for context.
- Contextual headers: prepend a one-line summary of the document to each chunk before embedding.
- Semantic chunking: split where embedding similarity between adjacent sentences drops, rather than at fixed sizes.
- Question generation: generate likely questions for each chunk and index them alongside the text.
Measure, don't guess
Build an evaluation set of 50–200 real questions with known source passages. For each chunking configuration, measure:
- Recall@k — is the right chunk in the top k results?
- Mean reciprocal rank — how high does it appear?
- Answer faithfulness — does the generated answer stay grounded in the retrieved text?
- Latency and cost — larger indexes and longer contexts are not free.
Change one variable at a time — size, overlap, structure, metadata, re-ranking — and keep the configuration that wins on your questions. Retrieval quality is an empirical property of your documents, and the teams that measure it consistently ship the RAG systems people actually trust.
