Chunking Strategies

This cover sets up the post as a tour of a ladder rather than a single technique. The strategies climb from blunt to context-aware, and the practical skill is choosing the lowest rung that actually solves your retrieval problem, since each step up adds cost or complexity.

Framing them as a ladder also implies a workflow: start simple, measure, and climb only when the data demands it. That discipline keeps systems simpler than the 'use the fanciest splitter' instinct would.

Fixed-size splitting is the baseline because it's trivial to reason about and implement: step through the text, cut every N units, add overlap. Its virtues are speed and predictability — you know exactly how big every chunk will be.

Its flaw is that it's blind to meaning and structure, so it slices mid-sentence and mid-idea. That's acceptable as a starting point and a benchmark, but if retrieval comes back fuzzy, the blindness is usually the first thing to fix by climbing to a structure-aware splitter.

The stack diagram presents the four strategies as rungs, each a smarter answer to 'where does the boundary go?' Fixed cuts by count, recursive respects separators, semantic cuts on meaning shifts, and hierarchical maintains two granularities at once.

Reading it bottom-to-top as increasing sophistication helps you calibrate effort. You rarely need the top rung for simple prose; you reach for it when documents are long, interlinked, or mixed-topic and the lower rungs leave answers fragmented.

Recursive splitting is the workhorse for prose because it degrades gracefully. It tries the largest natural separator first — blank lines between paragraphs — and only if a piece still exceeds the size limit does it recurse to finer separators: single newlines, then sentence ends, then words.

The result is chunks that respect the most natural boundary available while still obeying your size cap. This combination of 'natural where possible, forced only when necessary' is why it's the sensible default and why most frameworks ship it as the recommended splitter.

This snippet shows the recursive splitter in real LangChain code. The `separators` list encodes the priority order — paragraphs, then lines, then sentences, then words, then characters — and the splitter walks down it as needed to keep chunks under `chunk_size` with `chunk_overlap` between them.

The ordering of separators is the part worth tuning. For Markdown you might prepend heading markers; for code you'd use language-aware separators. The defaults are tuned for English prose, which is exactly why you should change them when your content isn't prose.

This slide corrects a subtle but costly habit: sizing chunks by characters when the real constraints are in tokens. Embedding models have token windows, prompts have token budgets, and the character-to-token ratio swings widely between English, other languages, and code.

Sizing by tokens — using the model's own tokenizer — makes chunk sizes reliable across all of that. A 256-token cap means 256 tokens whether the text is English, Japanese, or JSON, so chunks dependably fit the embedder and your prompt budget instead of occasionally overflowing.

Semantic chunking moves the boundary decision from structure to meaning. You break the text into sentences, embed each one, then walk through comparing consecutive sentences. A large drop in similarity signals a topic shift and becomes a boundary.

The appeal is that chunk length follows content: a long passage about one topic stays together, while a place where the subject changes gets cut even if it's mid-paragraph. The cost is extra embedding work at indexing time, which is why it's reserved for documents where topic boundaries don't line up with structural ones.

This trace visualizes the boundary-detection logic. Consecutive sentences with high similarity (0.91, 0.88) stay in the same chunk; when similarity drops sharply (0.41) the algorithm places a break, and the next high-similarity run starts a fresh chunk.

The concrete numbers make the threshold idea tangible: you're looking for the dips, not the absolute values. Choosing where to set that threshold is the main tuning knob of semantic chunking — too low and everything merges, too high and you over-fragment.

Hierarchical (parent-child) chunking is the answer to wanting precision and context at the same time. You embed small child chunks for sharp matching, but each child stores a pointer to a larger parent chunk. You match on the precise child, then hand the model the parent for full surrounding context.

This sidesteps the central trade-off rather than just picking a point on it: matching is done at small granularity, context is delivered at large granularity. The price is a more complex store and retrieval logic, which is why you adopt it when both precision and context genuinely matter.

This flow diagram traces the parent-child retrieval path: the query matches a small child chunk precisely, that hit resolves through a stored pointer to its parent, and the larger parent is what actually gets sent to the model.

The visual clarifies that two different chunk granularities serve two different jobs — children for matching, parents for answering. Keeping those roles distinct in your head is the key to implementing hierarchical retrieval correctly rather than conflating the two sizes.

The picking guide turns the ladder into decisions. Start fixed-size and measure so you have a baseline. Use recursive as the default for ordinary prose. Reach for semantic when documents mix many topics and structure doesn't mark the shifts. Adopt hierarchical when you need precise matching and full context simultaneously.

The meta-advice — start simple, measure, climb — matters as much as the specifics. Most teams over-engineer chunking before they've measured whether the simple option was good enough.

The CTA hands off to the code-heavy build post, where these strategies stop being descriptions and become a runnable pipeline. Having the conceptual ladder in hand makes that code read as choices you understand rather than magic incantations.

It closes the 'how it works' arc and sets up 'here's how to actually do it,' keeping the series' build toward a working RAG system on track.