Vector Search Basics

This post reframes a frustration nearly everyone hits: the LLM gives bad answers and gets blamed, when the real culprit is retrieval. The cover line makes that diagnosis explicit, shifting the reader's attention from the generation step to the search step where most production failures actually originate.

The emphasis on quiet failure is the throughline. Vector search rarely throws an error; it returns something plausible-looking, so problems hide until users complain. Naming this upfront primes the reader to treat each of the following mistakes as a thing to actively check for, not assume away.

The mismatched-model mistake leads because it is both the most common and the most catastrophic. Embeddings are only comparable within the space produced by a single model. Index your documents with one model and embed queries with another, and the two vector spaces have no shared meaning — similarity scores become essentially random noise dressed up as results.

What makes this insidious is that nothing crashes. The search returns ids and scores that look normal, so teams ship it and wonder why relevance is terrible. The fix is a discipline, not a feature: pin one embedding model and use it for everything, queries and documents alike.

This code slide contrasts the broken and correct patterns side by side so the rule is unmissable. The wrong version encodes documents with model_a and queries with model_b; even if both output 384 dimensions, the dimensions mean different things, so distances are meaningless. The right version uses a single shared model for both.

Showing it as runnable code rather than prose makes the mistake concrete and memorable. Engineers reviewing a codebase can now scan for two different model variables feeding the same index and immediately recognize the bug, which is the practical outcome this slide is designed to produce.

Chunk sizing is the second mistake because it silently degrades relevance even when everything else is correct. Chunks that are too large pack multiple topics into one vector, so the embedding becomes an average that matches everything weakly and nothing strongly. Chunks that are too small lose the surrounding context needed to interpret them.

The guidance — roughly 200 to 500 tokens with overlap, split on semantic boundaries like paragraphs or sections rather than arbitrary byte counts — gives a concrete starting point. Overlap matters because it prevents a relevant sentence from being orphaned at a chunk boundary where neither neighboring chunk captures it fully.

This comparison makes the chunking tradeoff visual by listing the failure modes of each extreme. Too-large chunks mix topics, dilute similarity, and waste the LLM's context window with irrelevant surrounding text. Too-small chunks lose context, create dangling references that no longer make sense alone, and multiply the number of vectors you must search.

Seeing both failure modes side by side reinforces that chunk size is a balance, not a 'bigger is safer' or 'smaller is more precise' decision. The right size depends on your content, which is why experimentation and overlap are the practical levers rather than a single universal number.

Ignoring metadata filters is framed as both an accuracy and a security mistake because it is genuinely both. Pure similarity has no notion of validity — it will cheerfully return a deleted policy, an expired price, or, in a multi-tenant system, another customer's data, simply because the meaning is close.

The correct pattern is filter-then-rank: constrain by date, tenant, status, or permissions first, then rank the survivors by similarity. Treating filtering as optional is how vector search becomes a data-leak vector. Stating it plainly here ensures readers building real systems do not learn this the hard way in production.

Trusting raw scores is the fourth trap, and it stems from a structural property of kNN search: there is always a top result, even when nothing relevant exists in the index. The search cannot return 'nothing'; it returns the least-bad option, which may still be useless.

The defense is to interrogate the scores: is the best score actually high in absolute terms, and does it clearly separate from the runner-up? A cluster of mediocre scores usually means no good match exists. Without this check, the system confidently feeds irrelevant context to the LLM, which then produces a confident wrong answer.

This code slide turns the previous lesson into an enforceable guard. After searching, we filter the hits by a minimum similarity threshold and, crucially, return an explicit 'no relevant context found' when nothing clears the bar rather than passing weak matches downstream.

The pattern of refusing to guess is what separates robust RAG from fragile demos. An honest 'I don't have that information' is almost always better than a fluent fabrication built on irrelevant context. The specific threshold value depends on your model and data and should be tuned against real queries, but the structure of the guard is universal.

Forgetting to re-embed after a model change is a subtle, time-delayed mistake. Upgrading your embedding model produces vectors in a new space; the old vectors in your index now live in an incompatible space. If you only embed new documents with the new model, your index becomes a silent mixture of two incompatible vintages.

The consequence is gradually degrading relevance that is hard to diagnose because there is no error, just slowly worse results. The discipline is to re-embed the entire corpus on any model change and to version your embeddings so you always know which model produced which vectors. This is an operational habit, not a one-time fix.

The hybrid-and-rerank slide addresses the ceiling of pure vector search. Embeddings excel at meaning but can miss exact identifiers — product codes, SKUs, person names, error numbers — where a literal match is what the user actually wants. This is precisely where keyword search remains strong.

The two-stage remedy is standard in mature systems: fuse vector and keyword (BM25) results to get both meaning and exact matches, then rerank the combined top candidates with a cross-encoder that scores query-document pairs directly for higher precision. The mantra 'retrieve wide cheaply, then rerank narrow' captures the efficient division of labor.

This flow diagram assembles the lessons into a single robust retrieval pipeline: filter by metadata first, run hybrid vector-plus-keyword search, rerank the top candidates with a cross-encoder, then apply a similarity threshold to drop weak hits before anything reaches the LLM. Each stage maps to a mistake the post just covered.

Presenting it as an ordered flow shows that these are not isolated tips but a coherent architecture. The ordering is deliberate — filtering before ranking for correctness and security, reranking before thresholding so the threshold judges the best available scores — and following it turns the listed mistakes into designed-in safeguards.

The checklist slide distills the entire post into six imperatives the reader can run through before shipping: same model on both sides, smart chunking with overlap, filter before ranking, threshold the scores, re-embed on upgrades, and add hybrid plus rerank when precision matters. It is intentionally scannable so it functions as a pre-flight check.

Closing with 'retrieval stops being your bottleneck' ties back to the why-it-matters post's point that retrieval quality caps answer quality. Nail these six and the most common causes of bad RAG answers are eliminated, which is the highest-leverage thing a practitioner can do.

The closing card looks ahead to building full RAG pipelines on top of the vector-search foundation laid across this day. Having covered concept, value, mechanics, code, and pitfalls, the reader is now equipped to assemble these pieces into a complete retrieval-augmented system.

The save prompt fits a checklist-heavy post: readers will want to return to these six rules every time they build or debug a retrieval system, making this a natural reference to keep.