LlamaIndex Crash Course

Post 3 is the engineering core of the crash course. Posts 1 and 2 told you what LlamaIndex is and why it's worth using; this one shows how it actually works so the framework stops feeling like magic and starts feeling like a system you can reason about and debug.

The organizing idea on the cover is that there are exactly two pipelines. Almost every feature, option, and failure mode in LlamaIndex lives in one of them. Once you can place a behavior into either ingestion or query, you know when it runs, what it costs, and what to change when it misbehaves.

The two-pipelines framing is the single most useful thing to internalize. The ingestion pipeline transforms raw data into a searchable index; it runs once, or whenever your data changes, and it's where the expensive embedding work happens. The query pipeline answers a question against that index; it runs on every request and should be fast and cheap by comparison.

Keeping them mentally separate explains a lot of the framework's design. You persist the index precisely so ingestion doesn't re-run needlessly. You tune top-k and synthesis on the query side without touching how data was chunked. Confusing the two is the root of mistakes like re-embedding on every startup, covered in post 5.

The ingestion pipeline diagram lays out four stages. Load uses a reader to turn sources into Documents. Parse splits Documents into Nodes with a node parser. Embed sends each Node's text to an embedding model and attaches the resulting vector. Store writes everything into the Index.

Each stage is a configuration point. You pick the reader for your source, the splitter and its chunk size, the embedding model, and the vector store. The defaults wire these together automatically, which is why from_documents() can do all four in one call — but knowing the stages exist is what lets you intervene when defaults fall short.

Understanding what a Node holds demystifies a lot of LlamaIndex behavior. Beyond the text chunk, a Node carries metadata — source filename, page, section, any custom fields you attach — an embedding vector once it's computed, and relationships linking it to neighboring Nodes (previous, next, parent, child).

Those extra fields aren't decoration. Metadata powers filtered retrieval ('only 2024 docs') and source citation in answers. Relationships enable advanced retrieval where pulling one Node can bring in its neighbors for context. When post 5 warns against discarding metadata, this is why it matters: a Node stripped to bare text loses half of what makes retrieval precise.

The query pipeline runs on every question and has its own four steps. The question is embedded into the same vector space the Nodes live in — this shared space is what makes similarity meaningful. The Retriever finds the top-k nearest Nodes. An optional re-ranker reorders them for relevance. Then the response synthesizer takes those Nodes plus the question, builds a prompt, and calls the LLM.

The critical detail is 'same vector space.' The question and the Nodes must be embedded by the same model, or their vectors aren't comparable and retrieval returns garbage. This is also why changing your embedding model means re-embedding everything — old vectors no longer live in the new space.

This diagram traces a single question through the query pipeline so the flow is concrete: user text, embed the query, retrieve top-k Nodes, synthesize with the LLM, return an answer plus its source Nodes. The returned source Nodes are what enable citation and debugging — they're the receipts for the answer.

Reading this alongside the ingestion diagram shows how the two pipelines meet at the Index: ingestion writes Nodes in, querying reads relevant Nodes out. The Index is the shared artifact, which is exactly why persisting it (post 4) decouples the cheap, frequent query path from the expensive, occasional ingestion path.

Retrieval being similarity search is worth understanding rather than treating as a black box. Embeddings map text into a high-dimensional space where semantic closeness becomes geometric closeness — passages about refunds cluster near other refund text. Retrieval embeds the query and finds the Nodes whose vectors are nearest, typically by cosine similarity.

The knob that matters here is similarity_top_k: how many nearest Nodes to return. Set it too low and you miss context the answer needs; set it too high and you pad the prompt with marginally relevant noise that costs money and can distract the model. Tuning this single number is one of the highest-leverage things you can do, which is why post 5 dedicates a slide to it.

Response synthesis is the stage beginners overlook, yet it shapes both answer quality and cost. Once Nodes are retrieved, the synthesizer decides how to feed them to the LLM. 'compact' (the default) packs as many Nodes as fit into a single prompt — fast and cheap. 'refine' answers from the first Node, then iteratively refines the answer against each remaining Node — more thorough, more LLM calls. 'tree_summarize' summarizes Nodes bottom-up, good for broad summary questions.

The mode is a deliberate tradeoff between latency, cost, and how completely the answer reflects all retrieved context. For a focused factual lookup, compact is fine. For 'summarize everything we know about X,' tree_summarize earns its extra calls.

This snippet exposes the retrieval stage on its own, which is the best debugging habit in LlamaIndex. Instead of calling a query engine and trusting the answer, you call as_retriever() and inspect the raw Nodes it returns — their similarity scores, source filenames, and text previews.

Doing this answers the most important question when something's wrong: did retrieval even surface the right passage? If the printed Nodes contain the answer, any error is in synthesis or the prompt. If they don't, the problem is upstream in chunking or top-k. Separating retrieval from generation like this turns vague 'it gives bad answers' complaints into a precise diagnosis, which post 5 builds into a decision tree.

The recap compresses the engine into five lines: ingestion is load-parse-embed-store; query is embed-retrieve-synthesize; a Node is text plus metadata plus a vector plus links; retrieval is nearest-vector similarity search; and the synthesis mode trades cost against how much context the answer uses.

These five facts are enough to reason about almost any LlamaIndex behavior. When something is slow, expensive, or wrong, you can now ask which pipeline it's in and which stage is responsible — the foundation post 5 relies on for its troubleshooting.

Post 3 explained the mechanics. Post 4 puts them to work with a complete, runnable example: indexing a folder, persisting the index, querying with cited sources, and swapping models. Theory becomes a script you can paste and run. Save this so the pipeline diagrams are right there when you're reading the code.