LlamaIndex Crash Course

Post 5 closes the crash course with the most practically valuable content: the mistakes that quietly destroy answer quality. The framing on the cover is the key insight — when a LlamaIndex app gives bad answers, the code is almost always fine. The failure is in retrieval quality: the right passage never reaches the model, so even the best LLM can't answer.

Every mistake in this post is a retrieval or data-handling error, not a syntax error. They produce systems that run without crashing and still return wrong or vague answers, which makes them harder to catch than ordinary bugs. Knowing the catalog in advance is the fastest way to avoid weeks of confused debugging.

Chunk size is the most common silent killer. The intuition: an embedding vector summarizes a whole chunk into one point in space. A huge chunk crams many topics into that single point, so it matches queries fuzzily and retrieves imprecisely. A tiny chunk captures a topic sharply but strips away the surrounding context the model needs to actually answer.

The default (~1024 tokens) is a reasonable starting point, not a universal answer. Dense, fact-packed content like legal clauses or API docs benefits from smaller chunks so each one is a single idea. Flowing narrative tolerates larger chunks. The only real way to set it is to try a couple of values and inspect what gets retrieved.

This fix shows how to take control of chunking with SentenceSplitter, set globally through Settings.node_parser. chunk_size=512 produces smaller, sharper chunks than the default; chunk_overlap=64 means consecutive chunks share some text so an idea that straddles a boundary isn't split in a way that loses its meaning.

Overlap is the underappreciated parameter. Without it, a sentence cut exactly at a chunk boundary can leave both chunks half-answering and neither retrievable. A modest overlap (roughly 10-20% of chunk size) cheaply guards against that. Set the parser before building the index, since chunking happens at ingestion.

Wrong top-k is the second silent killer and it cuts both ways. The historical default of 2 is often too few: the synthesizer gets one or two chunks and answers from a narrow slice, missing context spread across other passages. But the naive fix — crank top-k to 20 — backfires by flooding the prompt with marginally relevant Nodes that add cost, latency, and distraction.

The right move is deliberate tuning: enough Nodes to cover the answer, few enough to stay focused, usually a small single-digit number. When you genuinely need to cast a wide net and still stay precise, retrieve more and add a re-ranker that reorders by true relevance before the top few reach the LLM.

The bar chart visualizes the top-k tradeoff as a curve, not a 'bigger is better' line. At k=1 quality is poor because context is missing. Around k=3 quality typically peaks — enough coverage, little noise. By k=8 some irrelevant Nodes creep in and quality dips. At k=20 the prompt is bloated with noise and quality falls further while cost rises.

The exact peak depends on your content and chunk size, but the shape is general: there's a sweet spot, and both under- and over-retrieving hurt. The takeaway is to treat top-k as a tunable dial you test empirically, not a number to set once and forget.

Throwing away metadata is a subtler mistake that limits the system in ways you won't notice until a query goes wrong. If you ingest raw text and drop the source, date, section, or author, retrieval can only match on semantic similarity — it can't respect structure. A question about the '2024 policy' will happily retrieve a semantically similar 2019 Node, and you'll get a confidently outdated answer.

Attaching metadata at ingestion costs almost nothing and unlocks two things: metadata filters that constrain retrieval to the right subset, and citations that tell users exactly which document and section an answer came from. Treat metadata as part of the data, not an optional extra.

This fix shows metadata filtering in action. MetadataFilters with an ExactMatchFilter on year='2024' tells the retriever to consider only Nodes tagged with that year before doing similarity search. Combined with similarity_top_k=4, you get the closest matches from the correct subset of your data rather than from everything.

This is how you make retrieval respect structure instead of relying on semantics alone. It assumes you attached the year metadata at ingestion — filters can only act on fields that exist on the Nodes. The pattern generalizes to any field: department, document type, language, access level. Filtering plus similarity is far more precise than similarity alone.

Re-embedding on every run is a cost-and-time mistake rooted in misunderstanding the two pipelines from post 3. Calling from_documents() at every startup re-runs the expensive ingestion pipeline — re-chunking and re-embedding the entire corpus — even when nothing changed. On a large corpus that's real money and minutes of latency per launch, for zero benefit.

The fix is the persistence pattern from post 4: build and persist once, reload thereafter. When documents do change, update incrementally — insert or refresh only the affected Nodes — rather than rebuilding the whole index. Recognizing that embedding belongs to the occasional ingestion path, not the per-run startup path, is the mental shift that prevents this.

Trusting answers blindly is the mistake that ships bugs to users. An LLM produces fluent prose regardless of whether the retrieved context actually supported it, so a wrong answer can read just as confidently as a right one. The only defense during development is to inspect resp.source_nodes and confirm the cited Nodes genuinely contain the answer.

This check also localizes blame. If the source Nodes contain the answer but the response is still wrong, the problem is in synthesis or the prompt. If the source Nodes don't contain the answer at all, retrieval failed — and you should fix chunking or top-k rather than fiddling with the prompt. Separating 'did we retrieve it' from 'did we phrase it right' is the core debugging discipline.

The decision tree operationalizes everything in the post into a debugging flow. Start by asking whether the right Nodes were retrieved. If yes but the answer is still wrong, fix the prompt or synthesis mode. If yes and the answer is right, you're done. If the right Nodes weren't retrieved, ask whether the chunks are reasonable — if they are, raise top-k or add a re-ranker; if they aren't, fix chunk size and overlap.

This is the single most useful artifact in the crash course for day-to-day work. It converts the vague complaint 'LlamaIndex gives bad answers' into a sequence of concrete checks that each point to a specific fix, so you stop guessing and start diagnosing.

The checklist gathers the five fixes into one reference: tune chunk_size and overlap to your content; set similarity_top_k deliberately instead of leaving the default; attach and filter on metadata; persist the index instead of re-embedding each run; and always inspect source_nodes before trusting an answer. Each maps to a mistake covered above.

Keep this slide as the pre-flight checklist for any LlamaIndex project — running through it catches the large majority of real-world RAG quality problems before they reach users.

That completes the LlamaIndex crash course: what it is, why it matters, how it works, a full code example, and the mistakes to avoid. You now have both the mental model and the practical checklist to build and debug RAG over your own data. Tomorrow brings the next AI tool, decoded the same way. Save this troubleshooting post — it's the one you'll reach for most.