Chroma for Local RAG

This post is the engineering deep-dive. Chroma's ergonomics hide a familiar stack, and once you can trace what physically happens on add() and query(), tuning stops being guesswork and becomes reasoning about components. The cover promises a tour of the moving parts: the two lifecycles, the default embedding function, the HNSW index that powers similarity search, metadata filtering with where clauses, and how persistence actually lays data out on disk.

The theme throughout is that a collection is really two cooperating stores — a vector index and a document/metadata store — over a shared persistence layer, and almost every performance or correctness question reduces to how those interact.

The add() lifecycle is the first half of the machinery. When you call add() with documents but no precomputed vectors, Chroma runs the collection's embedding function on each document to produce a vector. It then stores four things: the embedding goes into the HNSW index, while the document text, the metadata dict, and the id go into the document/metadata store.

The key insight is that embedding happens at write time, transparently, using whatever embedding function the collection was configured with. This is why the embedding function is a property of the collection, not of each call — every record must be embedded the same way to be comparable. Understanding this also explains why the first add() can be slow: the embedding model may need to download and load.

The pipeline diagram lays the add() path out linearly so the order is unambiguous: text comes in, it's embedded, the document/metadata/id are stored, and the vector is inserted into the HNSW index. The icons are mnemonic — a note for raw text, numbers for embedding, a cabinet for storage, a web for the graph index.

Seeing it as a pipeline clarifies where time goes during a bulk load. Embedding is usually the expensive stage, especially with an API-based embedding function, which is why batching many documents into one add() call matters. The storage and indexing stages are comparatively cheap. If imports are slow, the embedding stage is almost always the culprit.

The query() lifecycle is the second half, and the most important detail is highlighted deliberately: the query text is embedded with the SAME embedding function used at add time. This is non-negotiable — comparing a query vector from one model against document vectors from another produces meaningless distances. Chroma enforces this by tying the embedding function to the collection.

After embedding, the query vector is compared against the HNSW index to find nearest neighbors. If you supplied a where filter, metadata constrains which records are eligible. Finally Chroma returns a parallel set of results — documents, distances, metadatas, and ids — so you get both the content and the information needed to rank, cite, and debug it.

The default embedding function deserves its own slide because it's why Chroma works out of the box. It uses all-MiniLM-L6-v2 through the sentence-transformers library: a small, fast model that maps text into 384-dimensional vectors. It downloads once on first use, then runs locally on CPU (or GPU if available), costing nothing per call.

MiniLM is not the highest-quality embedding model available, but it is genuinely good enough to prototype and to validate whether retrieval helps your task. When you need better recall — especially for domain-specific or multilingual text — you swap in a stronger embedding function (OpenAI, Cohere, a larger sentence-transformer) on the collection. The default's job is to get you to a working retriever with zero decisions.

HNSW — Hierarchical Navigable Small World — is the algorithm that makes similarity search fast enough to be practical. A brute-force search would compute the distance from the query vector to every stored vector, which is linear in corpus size and too slow at scale. HNSW instead builds a multi-layer graph connecting each vector to nearby vectors.

Search enters at a sparse top layer where each hop covers a large distance, greedily moving toward the query, then descends into denser layers to refine the result. This yields roughly logarithmic search time. It is approximate — it can occasionally miss the true nearest neighbor — which is the tradeoff you accept for speed, and which the index's parameters (M and ef) let you tune toward recall or latency.

This comparison crystallizes the two-store architecture inside a single collection. On one side is the vector index: an HNSW graph that answers similarity questions approximately and fast, returning distances. On the other is the metadata-and-document store, backed by SQLite, which holds the text and ids and answers exact where-filter questions.

Understanding that there are two distinct stores explains a lot of behavior. Vector similarity and metadata filtering are different operations over different structures, joined when Chroma intersects them for a filtered query. It also explains the cost profile: the vector index is memory-and-compute heavy, while metadata filtering is a more conventional lookup. One collection, two lenses on the same records.

Metadata filtering is what lets Chroma do precise, structured retrieval alongside fuzzy similarity. Every document can carry a metadata dict, and at query time a where clause restricts results to records whose metadata matches — for example {'source': 'faq'} returns only FAQ documents. A separate where_document clause filters by substring within the text itself.

Crucially, Chroma combines this with vector similarity in one query call: it finds semantically similar documents and applies your filters together, returning only records that are both relevant and matching. This is what enables real-world retrieval like 'find the most relevant passage from the 2024 handbook' — similarity for relevance, metadata for the constraint.

This code slide ties the query mechanics together. The query passes a question, asks for the top 3 results, and applies two filters: a metadata where clause limiting to source 'faq', and a where_document clause requiring the text to contain 'refund'. Chroma embeds the query, searches the HNSW index, applies both filters, and returns parallel lists.

The loop reads documents and distances together, and the comment states the rule that trips up newcomers: lower distance means closer, more relevant. With the default cosine space, distance is effectively 1 minus cosine similarity, so 0 is identical and larger values are less similar. Always interpret distance relative to the collection's configured space — comparing raw distances across spaces is meaningless.

Persistence is what separates a throwaway test from a durable corpus, and it's simpler than people expect. PersistentClient(path=...) tells Chroma to write to a directory: a SQLite database file holds documents, metadata, and ids, while binary HNSW index files hold the vector graphs, typically one set per collection. There's no separate database server — it's all files in your folder.

On startup, Chroma loads these files back into memory, reconstructing your collections and indexes with no rebuild step. This is why your data and even your tuned index survive restarts transparently. It also means backing up or moving your vector store is as simple as copying that directory — a genuinely useful property for reproducible experiments.

These knobs and gotchas catch the things that quietly cause trouble. The distance space defaults to cosine but can be set per collection via metadata (hnsw:space) — and it must match what your embedding model expects. The single most important rule recurs: use the same embedding function for add() and query(), or distances are meaningless. HNSW parameters like M (graph degree) and ef (search breadth) are configurable per collection to trade recall against speed and memory.

The last gotcha is operational: the first time you use the default embedding function, the model downloads and loads, which can cause a noticeable pause or an offline failure. Knowing this prevents you from mistaking a one-time download for a hang.

The teaser points to Day 90's Code Example post. Having seen the machinery — embedding, HNSW, filtering, and persistence — the next post puts hands on the keyboard: building a complete local RAG pipeline that chunks a document, stores it in Chroma, retrieves the relevant pieces, and feeds them to an LLM, turning these internal concepts into working code.