Chroma for Local RAG

This closing post is about the failures that don't announce themselves. Chroma rarely throws a stack trace on these issues; instead it returns irrelevant chunks, lets the LLM hallucinate from weak context, or quietly comes up empty after a restart. The cover frames the theme: most Chroma pain is a mismatch or a missing step, not a bug — between client types, between embedding functions, between chunk size and content, between the metadata you stored and the queries you later wish you could run.

The five traps that follow each come with a concrete fix, because knowing a failure mode without its remedy just makes you anxious. The goal is to internalize these before your demo, not during the post-mortem.

The first and most common trap is using the in-memory client. chromadb.Client() and EphemeralClient keep everything in RAM and discard it the moment the process exits. People prototype happily, restart their script, and find the collection empty — then assume Chroma lost their data, when in fact it was never persisted. This is the single most reported confusion for newcomers.

The fix is to use PersistentClient with a path for anything you want to keep. The in-memory client is genuinely useful for unit tests and quick throwaway experiments, but the moment you care about the data surviving a restart, you need persistence. Make PersistentClient your default and reach for the in-memory client only deliberately.

This code slide shows the persistence fix directly, with the wrong call called out in a comment so the contrast is unmissable. chromadb.Client() — the ephemeral one — vanishes on exit, while chromadb.PersistentClient(path='./chroma_db') writes to disk. After this change, your collection and its HNSW index live in the named directory and reload automatically next run.

get_or_create_collection is paired with it for the same reason it appeared in the code post: it makes re-running the script idempotent. Together these two lines give you a store that both survives restarts and doesn't error when you run your setup code twice. This is the boring infrastructure that prevents the most embarrassing class of 'where did my data go' bugs.

The second trap is mismatched embedding functions, and it's insidious because it produces no error — just bad results. If you add documents using the default MiniLM model but then query a collection configured for OpenAI embeddings, or if you change the embedding function between runs, the query vector and the stored vectors live in unrelated coordinate spaces. Their distances are noise, and ranking is effectively random.

The rule is one embedding function per collection, pinned and documented. If you must change the embedding model, you have to re-embed every document in the collection, not just new ones — which in practice means creating a fresh collection. Treat the embedding function as a permanent property of the collection, decided at creation time.

The third trap is poor chunk sizing, which silently caps the quality of every answer. Chunks that are too large dilute relevance: the one sentence that answers the question is buried in a page of unrelated text, so its embedding is an average of many ideas and the LLM receives mostly noise. Chunks that are too small lose surrounding context: you retrieve a fragment that's missing the information needed to interpret it.

Both degrade answers without any error. The fix is to target focused passages — commonly in the 200-500 token range — with a small overlap so boundary content stays intact. The exact numbers depend on your content and embedding model, so tune them empirically against real questions rather than trusting a default.

This bar diagram makes the chunk-size tradeoff visceral. Too-small chunks score low because they lose the context needed to be useful. Too-big chunks also score lower because relevant content gets diluted among irrelevant text. The middle — focused but complete passages — scores highest. The numbers are illustrative, but the inverted-U shape is exactly what teams observe in practice.

The practical method this implies: don't guess chunk size once and move on. Sweep a few sizes and overlaps, run a fixed set of representative questions, and measure whether the right source chunk is retrieved. Pick the configuration that maximizes retrieval accuracy on your own data. Chunking is tuning, not a constant.

The fourth trap is failing to store metadata, which costs you both filtering and citations. If you only add raw text, you have no way to restrict retrieval to a particular source, date range, or section — every query searches the entire corpus indiscriminately. Worse, when the LLM produces an answer, you can't tell the user which document or page it came from, which undermines trust in any serious application.

The fix is to attach a metadata dict — source, page, url, section — at add() time, while you still have that context from the original document. Retrofitting metadata later means re-importing everything, because metadata is stored per record at write time. Treat metadata as mandatory, not optional: it's nearly free to add up front and expensive to add later.

This code slide shows the metadata fix in practice. The add() call attaches a metadata dict to each chunk capturing its source, page, and section. With that in place, two valuable capabilities unlock: you can scope retrieval with where={'source': 'handbook'} to search only the relevant subset, and you can surface real citations by reading each result's metadata alongside its text.

The key timing point is that this metadata must be assembled at ingestion, when you still know which page and section each chunk came from. That's why chunking and metadata extraction usually happen together in the ingestion code. Building this in from the first version costs almost nothing; bolting it on after you've indexed a large corpus means a full re-import.

The fifth trap is trusting retrieval without ever evaluating it. A pipeline that runs end to end without errors feels finished, but running is not the same as correct. If retrieval surfaces the wrong chunks, the LLM will confidently synthesize an answer from irrelevant or contradictory context — and because the answer is fluent, you may not notice it's wrong until a user does.

The fix has two parts. First, inspect the retrieved documents and their distances during development — actually read what came back for representative queries. Second, build a small evaluation set of question-to-expected-source pairs and check whether retrieval returns the right source. Evaluating retrieval separately from generation tells you which half of the system to fix when answers go wrong.

This checklist distills the whole post into a pre-flight you can run before shipping. Use PersistentClient with a real path so data survives. Pin one embedding function per collection and write it down so nobody mixes models. Tune chunk size and overlap toward focused 200-500 token passages rather than accepting a default. Store source and page metadata from the very first ingestion. And evaluate retrieval itself, not just the final answer, with a small set of known question-source pairs.

Running this checklist converts the five silent failure modes into deliberate, verified decisions. None of these checks is expensive, and all of them are far cheaper than debugging a demo that confidently returns wrong answers in front of an audience.

This mindmap groups the traps by the part of the system they live in, which is how you'll actually reason about them. The Persistence branch covers the client type and using a real path. The Embeddings branch covers pinning one function and re-embedding on change. The Chunking branch covers right-sizing and overlap. The Quality branch covers metadata and evaluating retrieval.

Organizing the mistakes this way gives you a diagnostic map: when something's wrong, ask which subsystem owns the symptom — vanished data points at persistence, irrelevant results point at embeddings or chunking, missing citations point at metadata, and confidently-wrong answers point at unevaluated retrieval. The likely cause and fix follow directly from naming the right subsystem.

The teaser closes out the Chroma day. With the concept, the motivation, the internals, a working code pipeline, and the common mistakes all covered, you have an end-to-end working knowledge of Chroma for local RAG. Day 90 moves on to a new topic in the series, building on the vector-database and retrieval foundation laid here.