Chunking Strategies

This cover signals a shift from theory to hands-on building. The promise is concrete: by the end you'll have a function that turns any document into clean, token-sized, structure-aware chunks with metadata, ready to embed. Every snippet in the post is real and runs as written.

The post deliberately walks up the ladder from the previous post in code — naive, then recursive, then semantic — so you feel each improvement rather than just being told about it.

Step zero gets the environment ready and loads a sample document. The packages cover the three approaches the post demonstrates: LangChain's splitters for recursive chunking, tiktoken for token-accurate sizing, and sentence-transformers plus numpy for the semantic chunker.

Reading the file with explicit UTF-8 encoding avoids a frequent Windows pitfall where the default encoding mangles non-ASCII characters. Printing the character count is a trivial sanity check that the document actually loaded before you start splitting it.

This naive baseline is here to be felt, not shipped. It reproduces the eight-line fixed-size chunker so you can run it on a real document and observe the damage: printing the last 60 characters of the first chunk usually reveals a word sliced clean through.

Seeing that mid-word cut firsthand is more convincing than any explanation of why boundary-aware splitting matters. It motivates every improvement that follows in the post, and it establishes the baseline you'll compare the better splitters against.

Step two introduces the recursive splitter built from the tiktoken encoder, which is the single most important upgrade in the post. Two things change at once: sizing is now in tokens (`chunk_size=256` means 256 tokens, reliably) and boundaries follow the separator priority — paragraphs, lines, sentences, then words.

The `from_tiktoken_encoder` constructor is the detail people miss; without it the splitter would size by characters and your token counts would drift. With it, chunks dependably fit the embedder's window and your prompt budget regardless of the content's language or character density.

Step three attaches metadata, which transforms anonymous text into citable, filterable records. Each chunk gets a stable id, its source filename, and its position in the document via `chunk_index`.

This is small code with outsized payoff. The source lets you cite where an answer came from; the index lets you fetch neighboring chunks or reconstruct order; the id gives you a stable handle for updates and deletes. Skipping this step is one of the mistakes the next post calls out, so building it in here is intentional.

Step four implements semantic chunking from scratch so the mechanism is fully transparent. It splits on sentence-ending punctuation, embeds every sentence with a small fast model, then walks through comparing each sentence to its predecessor via a normalized dot product (cosine similarity).

When similarity drops below the threshold (0.6 here), the meaning has shifted and the current chunk is closed. Because the embeddings are normalized, the dot product is cosine similarity directly, which keeps the comparison cheap. The threshold is the knob to tune: lower merges more aggressively, higher fragments more.

This pipeline diagram zooms out from the individual snippets to the shape of the whole flow: read the raw document, split it token-aware, tag each chunk with metadata, and store the vectors. It's the map the code has been filling in step by step.

Keeping this four-stage shape in mind helps when you adapt the code to your own stack — whatever splitter or store you swap in, the stages stay the same, and each snippet in the post maps to exactly one box here.

Step five embeds and stores the chunks in Chroma, an in-process vector database that needs no server, making it ideal for a runnable example. The `add` call passes ids, the chunk texts as documents, and the metadata dictionaries in parallel lists.

Chroma embeds the documents for you with its default model, so this single call handles both vectorization and storage. In production you'd likely supply your own embedding function to match the model you chose, but the structure — ids, documents, metadatas — stays identical.

Step six closes the loop with a retrieval sanity check, which is the step too many builds skip. Querying with a real natural-language question and printing the top three chunks (with their indices) tells you immediately whether your chunking actually surfaces the right passage.

This is your fastest feedback signal. If the expected chunk isn't in the top results, you adjust chunk size, overlap, or strategy and re-run — long before building any of the generation layer on top. Treat this snippet as the seed of your evaluation harness.

The production checklist turns the demo into something you can trust at scale. Logging average and maximum chunk token counts catches chunks that silently exceed the embedder's window. Verifying the fit prevents truncation that quietly drops text from the index.

Storing source and index makes citations possible, and re-chunking when documents change shape acknowledges that a chunking config tuned for one document set can degrade as the corpus evolves. These four habits separate a notebook demo from a maintainable retrieval system.

The CTA points to the mistakes post, which catalogs the failure modes this clean pipeline is designed to avoid. Having built the happy path, you're primed to recognize each anti-pattern and the specific code that prevents it.

It also keeps the series moving toward embeddings and vector search, the next link in turning these chunks into fast, accurate retrieval.