Text Chunker in Machine Learning

The Mismatch Problem

Here's the fundamental issue: embedding models and retrieval systems impose practical constraints that raw documents simply don't satisfy.

Transformer-based encoders have finite context windows. BERT variants typically handle 512 tokens, while modern long-context models extend to 8K-32K tokens -- but even they can't ingest an entire book or codebase in one pass. And even when it's technically feasible to squeeze a full document into one embedding, you lose granularity: the resulting vector averages over all content, making it nearly impossible to retrieve specific facts or passages.

Retrieval at Scale

Retrieval systems suffer similar problems at scale. When a user at, say, an Indian fintech company asks "What are the RBI guidelines for KYC verification?" your vector store should return the most relevant passage -- not an entire 120-page regulatory document. Downstream LLMs have their own context limits too. Feeding retrieved text into GPT-4 or Claude means staying within the prompt budget, which often means 2-4 pages of context maximum.

The Solution

The text chunker emerged as the bridge across this mismatch. By splitting documents into semantically meaningful units of appropriate size, it enables:

1. Embedding models to produce focused, high-quality representations
2. Retrieval systems to return precise passages
3. LLMs to consume relevant context without exceeding token limits

The alternative -- retrieving whole documents or arbitrary character ranges -- produces poor recall and wasteful context that dilutes answer quality. That's not a system anyone wants to ship to production.

Without chunking, your RAG pipeline is a search engine that returns entire books when you asked for a paragraph.

The Central Tension: Granularity vs. Context

The core challenge in chunking is balancing two competing forces: granularity and context.

Small chunks maximize retrieval precision by isolating specific facts. BUT they risk losing surrounding context that gives meaning to the isolated passage. Large chunks preserve context but dilute the semantic focus of the embedding, reducing retrieval accuracy.

The Library Analogy

Let me give you a mental model. Imagine searching a library:

• If books are the retrieval unit (large chunks), you get tons of context but must scan entire volumes to find your answer.
• If sentences are the unit (tiny chunks), you find exact quotes fast but lose the surrounding explanation that makes them meaningful.
• Paragraphs or subsections (medium chunks) often strike the right balance.

A 10,000-word document split into 512-token chunks gives us roughly 40 chunks. That's 40 focused vectors in your embedding space, each one retrievable independently. But what happens when a critical piece of context gets split across two chunks? That's precisely the problem overlap and advanced strategies try to solve.

How Strategies Differ

Chunking strategies differ in how they define boundaries:

• Fixed-size methods prioritize simplicity and uniform memory footprint.
• Semantic methods attempt to detect topic shifts and break at natural boundaries -- the end of a paragraph, a heading, or a sentence that signals a new subject.
• Hierarchical methods maintain both: small chunks for precise retrieval with links to larger parent chunks that provide context to the LLM.

It's okay if these distinctions feel blurry right now -- we'll walk through each one in detail soon.

The chunker doesn't interpret meaning. It applies heuristics (character counts, sentence delimiters, embedding similarity) to approximate semantic coherence. The quality ceiling is set by how well these heuristics align with the true information structure of your corpus.

The Math Behind Chunking

Let's formalize what a text chunker actually does. Don't worry -- I'll explain the intuition before dropping any formulas.

A text chunker is simply a function that takes a document and produces an ordered list of text segments. More formally, it implements a function $f: D \rightarrow C$ where $D$ is the set of documents and $C$ is the set of chunk sequences.

Given a document $d \in D$ with text content $T(d)$ of length $n$ characters, the chunker produces a sequence of chunks:

$C(d) = [c_1, c_2, \ldots, c_m]$

where each chunk $c_i$ is a substring of $T(d)$ satisfying:

1. Length constraint: $|c_i| \leq \text{max\_chunk\_size}$ (often 512-2048 characters or 128-512 tokens)
2. Overlap constraint: $c_i$ and $c_{i+1}$ may share an overlap region of length $o$ (typically 10-20% of chunk size) to preserve context across boundaries
3. Coverage: $\bigcup_i c_i \approx T(d)$, meaning the union of chunks approximates the original document (with possible small gaps or overlaps)

Strategy Breakdown

Chunking strategies differ in how they select chunk boundaries. Let's walk through each:

Fixed-size chunking: Split every $\text{max\_chunk\_size}$ characters (or tokens), optionally with overlap. Boundaries are arbitrary and may split sentences or words. That was pretty simple, wasn't it?

Recursive character splitting: Recursively attempt to split on paragraph delimiters ("\n\n"), then sentence delimiters (".", "!", "?"), then word boundaries, falling back to character-level splits only if necessary. This is the workhorse of most RAG applications.

Semantic chunking: Embed consecutive sentences or small text units and compute pairwise cosine similarity:

$\text{sim}(c_i, c_{i+1}) = \frac{\mathbf{e}_i \cdot \mathbf{e}_{i+1}}{\|\mathbf{e}_i\| \cdot \|\mathbf{e}_{i+1}\|}$

Insert a chunk boundary wherever similarity drops below a threshold $\tau$, indicating a topic shift.

Sentence-window chunking: Treat each sentence as a retrieval unit but embed it with surrounding sentences (e.g., $\pm 1$ sentence) to provide context. At retrieval time, the window is expanded for the LLM.

Late chunking: Embed the entire document with a long-context model, then split the contextualized embeddings at sentence or paragraph boundaries, preserving cross-chunk context in the embeddings themselves (Gunther et al., 2024). This is the newest approach and arguably the most elegant -- but also the most expensive.

Key Takeaways

• A text chunker splits long documents into smaller, semantically coherent segments suitable for embedding and retrieval in RAG pipelines. It defines the retrieval granularity for the entire system -- get it right and everything downstream benefits.

• The core tradeoff is precision vs. context: small chunks (128-256 tokens) maximize retrieval precision but risk losing surrounding explanation; large chunks (1024-2048 tokens) preserve context but dilute embedding focus.

• Chunk size should be token-based (not character-based) to respect embedding model context windows. A starting point of 512-1024 tokens with 10-20% overlap works for most corpora.

• Recursive character splitting (splitting on paragraphs, then sentences, then words) balances simplicity with respect for document structure. It's the default for most applications -- start here.

• Semantic chunking detects topic boundaries using sentence embeddings but adds embedding cost at chunking time. Late chunking preserves cross-chunk context using long-context models but is the most expensive approach.

The text chunker defines the retrieval unit for every downstream component. Its design determines retrieval precision, context quality, and ultimately the ceiling on answer quality in your RAG system. As we've seen, careful tuning of strategy, size, and overlap is critical to production performance. Moving on to the next block -- the embedding model -- we'll see how chunk quality directly shapes vector representations.