Comparing chunking strategies quantitatively for RAG

The choice of chunking strategy largely determines the success or failure of a Retrieval-Augmented Generation (RAG) pipeline. When source documents are split into fragments that are too large, irrelevant text floods the language model's context window. If, on the other hand, you make fragments too small, coherence is lost and necessary preconditions for factually answering a question are missing. In practice, this decision is too often made based on rules of thumb or arbitrary defaults such as 500 tokens with 10% overlap.

A well-considered architecture requires a quantitative evaluation: how does a specific splitting method perform on your corpus in terms of accuracy, signal-to-noise ratio, and eventual answer quality? In this article, we formulate a reproducible measurement setup with which different splitting strategies are objectively compared with each other. For a broader overview of the interaction between the retrieval and generation steps, we refer to the foundation in the article on RAG evaluation, which dissects the complete chain from search step to model response.

The trade-offs between fixed, semantic, and structural splitting

Before calculating measurement values, it is necessary to classify the fundamental splitting mechanisms. There are roughly four dominant chunking approaches, each with specific assumptions about text structure and semantic density:

• Fixed windows (fixed-size chunking): The text is rigidly split at a predefined number of tokens or characters (for example, 256 or 512 tokens), often with a sliding overlap window of 10% to 20%. This is computationally cheap, but arbitrarily cuts sentences or lines of reasoning in half.
• Recursive structural splitting (recursive character chunking): An attempt is made to split along natural boundaries, working through a hierarchy of separators (paragraphs \n\n, line breaks \n, sentences . and spaces) until each block fits within the target size. This preserves grammatical units considerably better than fixed windows.
• Document-specific parsing (structure-aware / Markdown chunking): The parser leverages semantic metadata such as HTML headings, Markdown sections, table boundaries, or JSON structures. Fragments represent self-contained subsections in which headings are often included as a contextual prefix.
• Semantic chunking (embedding-based clustering): Sentences are analyzed sequentially. A cut is made at positions where the cosine similarity between consecutive sentence embeddings drops below a certain threshold, indicating a topic shift.

When documents are extremely long and narrative in nature, a chunking strategy can even compete with reading documents directly via huge context windows. Anyone considering offering documents in their entirety without an intermediate vector index would do well to consult the overview of models for summarizing long documents to understand the costs and processing limits of unsplit contexts.

Isolating the search phase from the generation phase

A common methodological mistake is judging chunking based solely on the final model answers. If a model fails, this could be due to a poor chunk (retrieval error), but equally to a reasoning error by the generator or hallucination despite perfect context. To purely quantify a chunking strategy, we split the evaluation into two isolated phases:

1. Retrieval quality (before the LLM): Are the correct passages retrieved from the database? Do the retrieved vectors contain the source data needed to answer the question?

2. Generation efficiency (through the LLM): How much noise do the retrieved chunks contain, and does the fragmented context lead to loss of coherence or unnecessary token costs?

Because the retrieval layer's accuracy is directly tied to the chosen vector representation, the influence of chunk size must always be evaluated in combination with the embedding model used. See also the guide for evaluating embedding models to verify how the effective dimensions and context capacities of embedding vectors relate to your chunk length.

Quantitative retrieval statistics: recall, MRR, and precision

To measure retrieval objectively, a curated evaluation set with pairs of questions and ground-truth document passages is required. We define a query $q$, a set of relevant document IDs $R_q$, and an ordered list of $k$ retrieved chunks $K_q = [c_1, c_2, \dots, c_k]$.

Small chunks (e.g., 128 tokens) often score high on Context Precision and Context Token Efficiency, but can yield lower Recall@k if an argument is spread across multiple sentences. Large chunks (e.g., 1024 tokens) easily achieve high Recall@k, but significantly reduce Context Token Efficiency because hundreds of irrelevant tokens are injected.

Context fragmentation and measuring faithfulness

When a chunking algorithm splits a document, the surrounding context is cut away. This introduces the risk of context fragmentation: a chunk might contain the answer "In this case, the notice period is three months," but the condition "if the contract was signed before 2021" is in the preceding paragraph, which was not retrieved.

To evaluate how sensitive a chunking strategy is to fragmentation, we measure the Faithfulness and Answer Relevance of the generating model under controlled conditions. A low faithfulness score indicates that the generative model has to invent facts because preconditions are missing from the supplied chunk text.

The assessment of faithfulness and factual consistency can be automated using calibrated prompt evaluators. To ensure the reliability of such assessment models, we refer to the systematics in the article on setting up LLM-as-a-Judge, which discusses methods for correcting judge bias and position preferences.

Pitfalls in the Dutch language area

When benchmarking chunking on Dutch-language corpora, specific linguistic complications arise that remain invisible in standard English-language evaluation runs:

• Long compounds: Dutch has long compound words written as one (such as aansprakelijkheidsverzekeringsmaatschappij). Character-based splitters that cut blindly based on length can break words in the middle if the tokenization boundary is not respected.
• Legal and official sentence constructions: Many Dutch-language documents (government policy, contracts, collective labor agreement texts) use discontinuous constructions in which the main idea is spread over dozens of words. Splitting based on a too-short fixed window (e.g., 150 tokens) cuts subordinate clauses off from their modal verbs, leading to a reversal of meaning.
• Abbreviations and punctuation: Sentence detection based on simple regex patterns (such as a period followed by a space) frequently fails on common Dutch abbreviations such as m.b.t., t.a.v., art. and d.w.z. This causes unintended micro-chunks.

Token costs, indexing time, and storage quantification

A quantitative evaluation is not complete without weighing in the operational cost side. Smaller chunks with a lot of overlap lead to an explosion in the total number of vectors in the database, which increases storage costs and the computation time for building the index. Moreover, the total chunk size directly affects the number of input tokens per LLM call.

For a methodological breakdown of how token counts translate into operational budgets, see the methodology for managing evaluation costs. In the calculation below, we lay out how data density and API calls scale per splitting variant.

# Voorbeeld: Formule voor vectorindex-omvang en API-overhead
# N = aantal brondocumenten
# W = gemiddeld aantal tokens per document
# C = chunk-grootte in tokens
# O = overlap in tokens

aantal_chunks = (W - O) / (C - O)
totale_index_vectoren = N * aantal_chunks
totale_invoer_tokens_per_rag_call = k * C

If a corpus of 10,000 documents of 2,000 tokens each is split with $C=256$ and $O=50$, this results in approximately 94,660 chunks. If you choose $C=1024$ with $O=100$, this drops to approximately 20,550 chunks. That is a factor of 4.6 difference in database size and initial embedding costs.

A reproducible experimental test protocol

To run a benchmark yourself that is resistant to random fluctuations, we use a standardized step-by-step plan. All variables outside the chunking method are kept strictly constant.

1. Determine the sample size

Ensure a representative set of at least 100 to 250 test questions with verified source passages. To calculate whether your sample offers statistically sufficient discriminating power, you can consult the statistical framework for test size to correctly interpret confidence intervals around your recall scores.

2. Fix all remaining parameters

Keep the following factors identical across all test variants:

• The embedding model and the distance metric (e.g., cosine distance).
• The vector index configuration (HNSW parameters such as efSearch and M).
• The number of retrieved results ($k$).
• The generation model including temperature=0.0 and a fixed seed.
• The prompt structure in which the chunks are presented to the LLM.

3. Run the comparison script

Below is a reference setup in Python for logging retrieval statistics across different chunking configurations:

from dataclasses import dataclass
from typing import List, Set

@dataclass
class RetrievalResult:
  query_id: str
  retrieved_chunk_ids: List[str]
  ground_truth_chunk_ids: Set[str]

def calculate_metrics(results: List[RetrievalResult], k: int = 3):
  total_recall = 0.0
  total_mrr = 0.0

  for res in results:
    top_k = res.retrieved_chunk_ids[:k]
    hits = [cid for cid in top_k if cid in res.ground_truth_chunk_ids]
    
    # Recall@k
    recall = len(hits) / max(1, len(res.ground_truth_chunk_ids))
    total_recall += recall
    
    # Reciprocal Rank
    rr = 0.0
    for rank, cid in enumerate(top_k, start=1):
      if cid in res.ground_truth_chunk_ids:
        rr = 1.0 / rank
        break
    total_mrr += rr

  count = max(1, len(results))
  return {
    f"Recall@{k}": round(total_recall / count, 4),
    "MRR": round(total_mrr / count, 4)
  }

Decision tree for chunking selection

Once the quantitative measurement results have been collected, the final architecture choice can be determined based on the decision rules below:

• Structured Markdown or document frames: Always choose a structural parser if source documents contain clear subheadings and tables. The measurable gain in Context Precision far outweighs the slightly more complex parsing step.
• Small sentence chunks with parent context (parent-child / hierarchical): When questions require very specific facts but the answer needs broad context, split into micro-chunks (100 tokens) for indexing, but send the parent paragraph (500 tokens) along to the prompt. This combines a high MRR with a high faithfulness score.
• Recursive with dynamic threshold: For unstructured prose, a recursive splitter with 15% overlap consistently performs more stably than rigid fixed windows, without the computational slowness of semantic clustering.

By no longer setting splitting parameters based on intuition but structurally measuring via Recall@k, Context Token Efficiency, and faithfulness scores, you transform the chunking step from an uncertain guess into an optimized engineering choice.