How to choose a chunk size and overlap for RAG

Short answer: start with structure-aware recursive splitting at about 200–500 tokens per chunk with 10–20% overlap, then measure. Write 20–50 real questions with the documents that answer them, compare two or three chunk settings on hit rate and recall, and keep the one that retrieves the right passage most often. There's no universal best size: it depends on your documents, your embedding model and the questions people ask.

Chunking is the first decision in a retrieval-augmented generation (RAG) pipeline, and it's easy to get wrong without noticing. The model still answers, just from the wrong context. This guide covers what chunk size and overlap change, the problems to look for, and how to compare settings on your own data. The examples use DevSnip Pro's local RAG tools in VS Code, but the method works with any stack.

What chunk size changes

Each chunk becomes one embedding, and retrieval returns whole chunks.

• Smaller chunks (around 100–250 tokens) are precise: the embedding is about one idea, so a matching question finds it. But an answer that spans two paragraphs may be split across chunks, and the model sees only half.
• Larger chunks (around 500–1,000 tokens) carry more context, but mix several topics into one embedding. The match gets weaker, and each retrieved chunk uses more of the context window.
• Very large chunks don't fit: most embedding models truncate input beyond their limit, often a few thousand tokens. Text past the limit is simply not represented in the vector.

Questions about specific facts ("What's the rate limit on the search API?") usually prefer smaller chunks. Questions that need explanation ("How does billing work?") prefer larger ones.

What overlap does

Overlap repeats the end of one chunk at the start of the next, so a sentence that falls on a boundary appears whole in at least one chunk. 10–20% of the chunk size is a common starting point.

More is not better. Above about 30%, you mostly pay to embed and store the same text twice, and retrieval returns near-duplicate chunks that crowd out other results. Some studies find little benefit from overlap at all, so include 0% in your comparison.

Split on structure first

How you split matters as much as the size:

• Recursive splitting tries paragraph breaks first, then lines, then sentences, and only cuts mid-sentence as a last resort. It's a good default for prose.
• Markdown section splitting keeps a heading with its content and records the heading trail (for example Install > Configure) as metadata, which you can show in citations or use as a filter.
• Fixed-size splitting ignores structure. It's simple but cuts through sentences, code and tables.

Keep code blocks and tables in one piece where you can. Half a table embeds poorly.

Look at the chunks before you embed

Most chunking problems are visible if you look. In VS Code, open a typical document and run DevSnip Pro: Chunking Tester. Choose a strategy (recursive, Markdown sections, sentences or fixed size), a unit (tokens or characters), a size and an overlap. You get every chunk with its size and section, plus warnings for:

• chunks much smaller than the target, such as a heading on its own, which retrieve poorly;
• duplicate chunks, which should be removed before embedding;
• overlap above 30%;
• chunks over 8K tokens, which most embedding models truncate.

It exports chunks.jsonl with an id, the text and metadata for each chunk:

{"id":"docs_install.md#1","text":"## Configure\n\nSet PORT in .env. The default is 3000.","metadata":{"source":"docs/install.md","chunk_index":1,"headings":"Install > Configure","tokens":16}}

Before indexing a whole corpus, Near-Duplicate Chunk Finder catches repeated boilerplate (footers, cookie notices, navigation), and Chunk Metadata Validator checks that every record has the fields your retriever filters on.

Measure on your own questions

Intuition about chunk size is unreliable. A small labelled set settles it:

1. Write 20–50 questions real users ask, and for each one note the id of the document or chunk that answers it.
2. Index your documents with setting A, run each question through your retriever, and record the ranked ids it returns.
3. Repeat with setting B (and C).
4. Compare the scores.

Retrieval Evaluation in DevSnip Pro takes one line per question, as query | relevant ids | retrieved ids in rank order, or as JSON Lines:

reset password  | d3     | d3, d7, d1
refund policy   | d9, d2 | d4, d2, d9
export to csv   | d5     | d8, d6, d1
api rate limits | d11    | d11, d12, d4

At k = 3 this gives a hit rate of 0.75 (three of the four questions found a relevant document), recall 0.75, MRR 0.625 and nDCG 0.67, and it points out that "export to csv" found nothing relevant. That miss is the useful part: look at the chunk that should have matched, and you'll usually see why. It might be split in the wrong place, buried in a bigger chunk, or worded differently from the question. Keyword-heavy questions like this one often do better with hybrid search, which Hybrid Search Fusion (RRF) helps you test.

The metrics, briefly:

For RAG, hit rate and recall matter most: if the right passage isn't retrieved, the model can't use it.

Check the cost of a setting

Smaller chunks mean more vectors. Chunk & Index Size Calculator turns corpus size, chunk size and overlap into the number of chunks, embedding cost, monthly cost, vector storage and HNSW index memory, so you can see what a setting costs before you choose it. Context Window Budget shows how many retrieved chunks of that size fit next to your system prompt, chat history, question and answer.

A reasonable process

1. Start with recursive or Markdown splitting at about 300 tokens and 10–15% overlap.
2. Inspect a few documents in the Chunking Tester and fix obvious problems (tiny chunks, broken tables, boilerplate).
3. Build a 20–50 question test set.
4. Compare your starting point against a smaller and a larger size, and 0% overlap.
5. Keep the winner, and rerun the test set whenever you change the embedding model, the splitter or the documents.