Chunking Strategies

This cover names an uncomfortable truth: a large share of broken RAG demos fail at chunking, not at the model or the database — the components teams instinctively blame and swap. The post is positioned as a field guide to those silent failures.

The word 'quietly' is the theme. Chunking mistakes rarely throw errors; they degrade answers in ways that look like model problems, which is exactly why they go undiagnosed for so long.

The first mistake is applying one global chunk size to every document type. A 500-token recursive split tuned for blog prose will mangle source code, chat transcripts, and tables, because each of those has a different natural unit of meaning.

The fix is to route by content type: detect or tag what kind of document you're chunking and apply a strategy suited to it. This is more work than a single global config, but it's the difference between chunks that respect their content and chunks that fight it.

The second mistake — no overlap — is the most common silent killer. With zero overlap, any idea that spans a chunk boundary is cleanly severed, leaving each half non-retrievable as a coherent unit and producing exactly the split-answer bug from the why-it-matters post.

The fix is cheap: a 10-20% overlap of chunk size keeps cross-boundary context intact in at least one chunk. The only cost is mild index duplication. Given how silent and damaging the failure is, overlap is almost always worth its small price.

The third mistake is treating every document as flat text and splitting by raw count, which shreds anything with structure. Tables become meaningless streams of numbers, headings get orphaned from the sections they introduce, and code blocks are torn mid-function.

The fix is to parse structure before sizing: extract tables, list items, and code blocks as whole units, and split within structural sections rather than across them. Structure-aware splitting preserves the semantics that flat splitting destroys, and it's the single biggest quality jump for technical or tabular documents.

This comparison contrasts bad and good boundaries line by line, making the abstract advice concrete. Bad splits cut mid-sentence, orphan headers, tear table rows, and skip overlap. Good splits break at paragraph ends, keep headers attached to their sections, preserve tables whole, and include sensible overlap.

Used as a checklist, the right column is essentially a definition of healthy chunking. If your splitter violates any item in the left column on real documents, you've found a concrete thing to fix.

The fourth mistake is the false comfort of huge chunks: the reasoning is 'if I make chunks big enough, the answer must be in there.' But oversized chunks produce blurry embeddings that match weakly, and even when retrieved they bury the one relevant sentence in paragraphs of noise the model must wade through.

The corrective is the inverted-U from the why post: bigger is not safer, it's a different failure mode. Right-sized chunks match precisely and present the answer cleanly. Resist the instinct to oversize as insurance.

The fifth mistake is dropping metadata, which forecloses two important capabilities. Without a source, you can't cite where an answer came from — fatal for trust in regulated or high-stakes settings. Without a section or document id, you can't filter retrieval to a scope or reconstruct order.

The fix is to always carry source id, section, and chunk index alongside the text from the moment of chunking. Metadata is nearly free to attach and expensive to retrofit, so the discipline is to never create a chunk without it.

This snippet shows the canonical fix for both the structure and metadata mistakes at once. First a MarkdownHeaderTextSplitter breaks the document on headings so sections stay intact and heading context is captured as metadata. Then a token-aware recursive splitter sizes within each section while preserving that heading metadata on every resulting chunk.

The two-stage pattern — structural split, then size split inside — is the workhorse recipe for real documents. Printing `chunks[0].metadata` confirms the heading trail rides along with each chunk, which is what makes scoped retrieval and citations possible downstream.

The sixth and deepest mistake is shipping a guessed chunk size and never checking it. Chunking is empirical: the only way to know your configuration works is to measure whether the right chunk lands in the top-k for a set of real questions with known answers.

The fix is a small evaluation set and a recall@k metric. Build a handful of representative questions, record which chunk should answer each, and measure how often retrieval surfaces it. Tuning chunk size and overlap against that number replaces vibes with evidence — and it's how every other mistake here gets caught.

This cycle diagram captures the empirical loop the previous slide argues for: choose a chunking strategy, index the corpus, evaluate recall@k on your Q&A set, adjust size and overlap, and repeat. It frames chunking as iterative tuning rather than a one-time setting.

The loop is the antidote to every mistake in the post — each anti-pattern shows up as poor recall, and each fix is validated by the number going up. Adopting this cycle is what turns chunking from guesswork into engineering.

The checklist consolidates the post into five actions: vary strategy by document type, always add 10-20% overlap, respect tables and headings and code blocks, keep chunks tight while carrying metadata, and measure recall@k before scaling. Together they cover every failure mode the post diagnosed.

Treat it as a pre-flight check before you scale a RAG system. Most chunking disasters are some combination of these five items being skipped, so running down the list catches problems while they're still cheap to fix.

The CTA closes the day and points forward to embeddings and vector search — the stage that turns these clean, well-bounded chunks into fast, accurate retrieval. With chunking handled properly, the next link in the pipeline can actually deliver on its promise.

It completes Day 69's arc from concept through stakes, mechanics, code, and pitfalls, leaving the reader with both the understanding and the checklist to chunk well before moving on.