Generators & Iterators

This is the hands-on post, and it's deliberately code-heavy. We build a realistic log-processing pipeline as a chain of generators, each a small stage. The aim is to move from understanding generators in the abstract to feeling how a real lazy pipeline behaves — one item flowing through all stages, constant memory, early exit for free.

Log processing is the perfect vehicle because the input is plausibly huge, the transformations are clear, and the streaming win is dramatic. Everything here is runnable; point it at a real log and watch it work.

Stage one streams the file. The key detail is that a file object is already an iterator — iterating it yields lines lazily, one disk read at a time, never loading the whole file. Wrapping it in our own generator lets us strip the trailing newline and gives us a named stage we can compose.

The with statement ensures the file closes when the generator is exhausted or garbage-collected. Crucially, nothing is read at the moment read_lines() is *called* — it returns a paused generator. The first line is read only when the first value is pulled, which is what makes the whole downstream chain lazy.

Stage two is a filter, and it shows the fundamental pipeline shape: take an iterable, yield a subset. only_errors loops the incoming lines and re-yields only those containing ERROR. It neither knows nor cares whether its input is a file generator, a list, or another pipeline stage — it just consumes an iterable.

This interchangeability is the heart of composable pipelines. Each stage is a pure transformation over a stream, with no assumptions about its source or sink. That decoupling is what lets you snap stages together in any order and reuse them across different pipelines.

Stage three parses each line into a structured dict. str.partition splits on the first occurrence of " ERROR ", returning the part before, the separator, and the part after — a clean way to peel a timestamp off the front and keep the message. We yield a dict per line.

Notice the pattern is identical to the filter stage: loop the input, yield a transformed item. A transform stage and a filter stage differ only in whether they always yield (transform) or conditionally yield (filter). Once you see that symmetry, you can write any stage you need from the same template.

Stage four wires the chain together and consumes it. The first three lines just create paused generators — assigning lines, errors, and records triggers zero work. The data only begins to flow at the sum() call, which is the consumer that drives the whole pipeline.

This is the most important behavioral point in the post: building the pipeline and running it are separate steps. You can compose an arbitrarily long chain of stages and it costs nothing until a consumer starts pulling. The sum here counts records by pulling each one through all four stages, one at a time.

The pipeline diagram visualizes the single-item flow. When sum asks for one value, parse asks only_errors for one line, which asks read_lines for one line off disk. That one item travels the full length of the chain, gets counted, and only then does the next item start its journey.

This pull-based, one-at-a-time execution is why memory stays flat regardless of file size. There's never a buffer of intermediate results between stages — just the single item currently in flight. The diagram is worth keeping as the canonical picture of how a generator pipeline actually executes.

The pull-not-push framing names the control flow that makes generators counterintuitive at first. In an eager pipeline, each stage finishes completely and pushes a full result to the next. In a lazy pipeline, the *consumer* pulls, and that pull propagates backward up the chain, drawing exactly one item through.

This inversion is why early exit is free: if the consumer stops pulling — via break or by only taking the first N — the upstream stages simply stop being asked and never do the remaining work. The producer never runs ahead of demand. Internalizing pull semantics is the difference between using pipelines and truly understanding them.

yield from is the delegation tool that keeps pipelines flat. read_many iterates a list of paths and, for each, delegates to read_lines with yield from, which transparently yields every line from that sub-generator before moving to the next path. The result is one continuous lazy stream across many files.

Without yield from you'd write a nested for loop that re-yields each item — which works but is noisier and, for generators that receive sent values or exceptions, semantically incomplete. yield from also correctly forwards those, making it the right tool whenever one generator needs to splice in the full output of another.

These rules of thumb generalize the pattern beyond this example. Every stage takes an iterable and yields an iterable, so stages are interchangeable. Keep them small and single-purpose so they compose and test easily. Remember nothing runs until a consumer pulls, so building the chain is cheap. Use yield from to flatten nested generators. And recognize that the final consumer — sum, list, a for loop — is what actually drives execution.

Follow these and your data code reads like a clean sequence of transformations while executing as an efficient single streaming pass. It's a genuinely powerful architectural style for anything I/O- or data-heavy.

The comparison contrasts the generator pipeline with the naive list-per-step approach. The eager version loads the whole file, then builds a filtered list, then a parsed list — each step materializing a full intermediate collection and doing all its work up front. Memory grows with every stage and early exit is impossible.

The generator pipeline makes one pass, holds one item at a time, can stop early on a break, and composes freely. For large inputs the difference is the gap between code that scales and code that exhausts memory. This is the concrete embodiment of every benefit claimed back in post two.

The full script ties everything into one runnable program. It defines all three stages — note only_errors here uses yield from over a generator expression, a compact idiom worth recognizing — composes them in a single expression, and consumes the result in a for loop that prints each parsed record.

This is the copy-paste-and-run artifact of the post. Drop it next to any log file named app.log, adjust the ERROR marker to your format, and it streams through the file printing structured error records in constant memory. It's a small but genuinely useful tool that demonstrates the entire day's concepts working together.

The mistake slide warns about the most common pipeline bug in practice: consuming the generator twice. After the for loop drains records, the generator is exhausted; a second loop silently iterates nothing. There's no error to tip you off — just missing output or a zero count.

The fix depends on intent. If you genuinely need two passes, either rebuild the pipeline from the source or materialize once with list() and iterate that list repeatedly — accepting the memory cost. The trap is assuming a generator behaves like a list; remembering its single-use nature, established back in post one, is what keeps you out of it.

The cover and CTA position this as the practical centerpiece. With a working pipeline in hand, the final post turns to the failure modes: exhaustion, late-binding closure bugs, unbounded infinite generators, and the other traps that turn elegant generator code into 2am debugging sessions. Knowing the patterns is half the battle; knowing how they break is the other half.