Chunking Strategies

This cover frames chunking as the unglamorous step that everyone skips past on the way to embeddings and vector databases — and exactly where most RAG quality is won or lost. Before any retrieval can happen, a document has to be cut into pieces, and the geometry of those cuts is decided here.

The goal of this post is to install the right mental model and vocabulary. By the end you should think of a chunk not as 'a piece of text' but as the atomic unit your system can ever retrieve and show the model.

The definition deliberately ties chunking to retrieval rather than to storage. People often imagine chunking as a formatting chore — making text fit somewhere. The truth is sharper: the retriever returns whole chunks, never partial ones and never raw documents, so whatever you decide a chunk is becomes the indivisible currency of your entire system.

That reframing has teeth. It means a question whose answer doesn't sit cleanly inside one chunk is a question your system is structurally bad at answering. Everything later in the pipeline operates on the chunks you defined here.

The reason whole-document embeddings fail is worth internalizing because it motivates every strategy that follows. An embedding model maps text of any length to a single fixed-size vector. To do that for a long document it must average away detail — the vector becomes a blurry summary of dozens of distinct ideas.

A blurry vector is bad at matching. It sits in the middle of everything and is strongly close to nothing, so a precise query about one paragraph gets a weak, ambiguous match. Splitting the document so each vector represents one focused idea is what makes similarity search actually discriminating.

This flow diagram is the skeleton of indexing: a document is split into focused chunks, and each chunk gets its own embedding. The visual point is the fan-out — one document becomes many vectors — which is precisely what gives the retriever something specific to match against.

Keeping this picture in mind prevents a classic confusion: thinking the model 'reads the document.' It never does. It only ever sees the handful of chunks the retriever fetched, so the split step upstream is doing more work than it appears.

Chunk size is the first knob and the one people reach for first. Measuring it in tokens rather than characters matters because tokens are the unit embedding models and LLMs actually count, and the ratio of characters to tokens varies with language and content type.

The 200-800 token range is a practical starting band, not a law. Smaller favors precision, larger favors context, and the right point depends on your documents and queries. The post's later strategies are largely about being smarter than a single global size.

Overlap is the cheapest insurance in chunking. By repeating a slice of text across a boundary, you ensure that an idea straddling two chunks survives intact in at least one of them. Without it, a fact split exactly at a boundary can become unretrievable as a coherent unit.

The cost is real but usually acceptable: duplicated text inflates your index size and can return near-duplicate chunks. A typical 10-20% overlap buys most of the safety without much waste, which is why nearly every production splitter ships with overlap on by default.

Boundaries are where chunking stops being arithmetic and starts being about meaning. A naive splitter cuts at a fixed offset and will happily slice through the middle of a word or a number. A boundary-aware splitter prefers natural break points — the end of a sentence, a paragraph, a heading.

The payoff is coherence: each chunk reads as a self-contained passage rather than a fragment. Coherent chunks embed more cleanly (the vector represents a complete thought) and read better when the model finally sees them, so boundary quality improves both retrieval and generation.

The 'unit of retrieval' idea is the thesis of the whole day. Because top-k retrieval returns whole chunks, the set of questions your system can answer well is essentially the set of questions whose answer fits inside the chunks you created.

This is why two engineers with the same documents, embedder, and database can get very different answer quality: they chunked differently. When an answer is spread across two chunks, retrieval must be lucky enough to grab both, and the model must reconcile them — both fragile. Designing boundaries so answers stay whole is the highest-leverage move available.

This snippet is intentionally minimal so the mechanics are unmistakable: walk the string, take a window of `size`, then step forward by `size - overlap` so consecutive windows share `overlap` characters. It captures fixed-size-with-overlap in eight lines.

It also makes the weaknesses obvious. It counts characters, not tokens, and it splits with zero regard for words or sentences. That's deliberate — it's the baseline the rest of the series improves on. Run it once on a real document and you'll feel exactly why structure-aware splitting exists.

The mindmap lays out the four families you'll meet across the next posts. Fixed splits by raw count; structural splits respect sentences, paragraphs, and headings; semantic splits cut where meaning shifts; hierarchical keeps a small chunk for matching and a large one for context.

Seeing them as a progression rather than a menu helps: each family is mostly a smarter answer to 'where should the boundary go?' You generally start simple and climb only as far as your retrieval metrics demand.

The summary compresses the post into four load-bearing claims. The first — a chunk is the smallest retrievable thing — is the mental model everything else hangs on. The second explains why whole-doc embedding fails. The third names the three knobs you'll tune. The fourth is the strategic punchline.

If a reader remembers only the last bullet — that boundaries decide which questions you can answer — they'll already chunk more thoughtfully than most teams.

The CTA hands off to the next post, which makes the business case: why this preprocessing step deserves real attention. Having the mental model from this post is the prerequisite for appreciating just how much downstream quality rides on it.

The series continues toward embeddings and vector search, so this foundation in chunking sets up everything that follows.