Generators & Iterators

This closing post is the failure-mode catalog. Generators are elegant, but their laziness and single-use nature create a specific family of bugs that are notorious precisely because they're silent — empty results, vanished exceptions, infinite hangs. Every trap here comes from real code, and each has a clean fix once you understand the underlying behavior.

The theme is that none of these are mysterious once you remember the mechanics from the earlier posts. They're the predictable consequences of laziness and single-use semantics meeting expectations built around lists.

Exhaustion after one pass is the defining generator gotcha. Because a generator is its own iterator, draining it once leaves it permanently empty — and the second traversal raises no error, it just yields nothing. Code that loops over the same generator twice, or computes a count and then tries to process the items, fails silently.

The fix is a deliberate choice: if you need the data more than once, either materialize it into a list (paying the memory cost) or rebuild the generator from its source. The danger is the silence — there's no exception to alert you, just downstream logic quietly operating on emptiness. Always ask "will anything iterate this twice?" before relying on a bare generator.

This code makes exhaustion undeniable. The first list(nums) returns [0, 1, 2] as expected. The second returns [] — not an error, just an empty list, because the generator has been fully consumed and sits in its exhausted terminal state.

The fix shown is the simplest: if you know you need multiple passes, store a list from the start. The broader habit is to be suspicious any time a generator is referenced after it's been iterated. When a second loop or aggregation mysteriously produces nothing, exhaustion is almost always the culprit, and recognizing the pattern saves enormous debugging time.

The late-binding closure trap catches even experienced developers. Closures — including generator bodies and lambdas — capture variables by reference, not by value. Because a generator's body doesn't execute until iteration, by the time it runs the enclosing loop has often finished and the loop variable holds its final value.

The result is generators or functions that all see the same last value instead of the per-iteration value you intended. The fix is to bind the value at creation time, typically via a default argument (lambda i=i: i) or an inner factory function. The root cause — deferred execution plus reference capture — ties directly back to the laziness theme of this whole series.

This code separates two related behaviors. The list comprehension of generators behaves correctly because each comprehension iteration creates a fresh binding of i, so each inner generator sees its own value — output [[0,0],[0,2],[0,4]]. This is the modern, well-behaved case.

The second block is the classic trap: a plain for loop building lambdas. All three closures capture the *same* i, and since they're called after the loop ends, all three return 2. The contrast is instructive — comprehensions give each iteration a new scope, but a bare loop does not. When closures and loops mix, always bind explicitly rather than trusting your intuition.

The infinite-generator hang is the scariest gotcha because it doesn't fail fast — it freezes and consumes memory until the process is killed. A while True generator produces values forever, so handing it to a consumer that wants *all* of them — list(), sorted(), sum() over no termination, max() — never returns.

The rule is absolute: an unbounded generator must always meet a bounding consumer. itertools.islice takes a fixed count, a break exits a loop early, and next() called N times pulls a fixed prefix. Any function that exhausts its input is forbidden on an infinite generator. Building this reflex prevents a class of production incidents that are painful to diagnose.

This code contrasts the fatal and the safe. list(naturals()) is commented out with a warning because it would hang forever, attempting to build an infinite list. The correct pattern wraps the infinite generator in islice, which lazily takes exactly five values and then stops, letting list() terminate normally with [0, 1, 2, 3, 4].

itertools.islice is the workhorse here — it's the lazy equivalent of slicing, designed precisely to take a bounded window from a potentially unbounded stream. Keep it (and the rest of itertools) in mind whenever you work with generators; much of the toolkit exists to bound, combine, and shape lazy streams safely.

Treating a generator like a sequence is the mistake that produces immediate, loud errors rather than silent ones — which is almost a relief by comparison. len(g) raises TypeError because a generator has no notion of length; it doesn't know how many values remain without consuming them. g[0] raises TypeError because there's no indexing protocol.

Even the 'in' operator is dangerous: it works, but it consumes the generator up to the match, leaving it partially or fully drained. The lesson is that any time you find yourself wanting length or random access, you've outgrown a generator and should use a list — or restructure with next() and itertools to stay within the streaming model.

The decision diagram gives a quick triage for choosing between a generator and a list. First question: do you need the data more than once? If yes, use a list — generators are single-use and you'll hit exhaustion. If no, ask whether the source is infinite. If it is, you can still use a generator but you *must* bound it with islice or a break.

If the data is finite and traversed once, a generator is the ideal fit. This little flowchart encodes most of the practical judgment from the whole day into two questions, and running through it before reaching for yield will keep you out of the majority of generator traps.

The genexp-versus-list-comp comparison addresses a constant source of confusion: the only syntactic difference is parentheses versus square brackets, yet the semantic difference is total. Parentheses give a lazy, single-use, memory-light generator; brackets give an eager, reusable, indexable list that holds everything.

The practical guidance: use the genexp form when you'll consume the result once and want to save memory — especially as an argument to sum(), any(), max(), or another consumer. Use the list-comp form when you need to keep, reuse, index, or measure the result. The visual similarity is exactly why people pick the wrong one, so the bracket style deserves a deliberate moment of thought.

Deferred exceptions are a subtle gotcha that surprises people debugging generators. risky() contains a raise, but calling risky() raises nothing — it returns a paused generator. The first next() yields 1 cleanly. Only the *second* next(), which runs the body past the yield to the raise, surfaces the ValueError.

The implication is that errors in generator logic appear at *consumption* time, not creation time, and often far from where the generator was defined. A try/except around the generator's *creation* catches nothing; you must wrap the *iteration*. This deferred-error behavior is a direct consequence of laziness and is worth keeping in mind whenever a generator's exception seems to come from a surprising place.

These bullets compress the entire gotcha catalog into five habits. Assume single-use and materialize with list() if you need more. Bind loop variables explicitly inside closures to dodge late binding. Always bound infinite generators with islice or a break. Take the absence of len() and indexing as a signal you actually want a list. And remember that errors and side effects surface when you pull values, not when you call the generator function.

Internalize these five and you've covered the overwhelming majority of generator bugs you'll ever hit. They're not arbitrary rules — each follows directly from the laziness and single-use semantics that make generators powerful in the first place.

The cover and CTA close out Day 21. Having moved through concept, motivation, mechanism, a working example, and the failure modes, you now have a complete and practical command of generators and iterators — what they are, why they matter, how they work, how to build a real pipeline, and how to avoid the traps. The series rolls on to the next topic tomorrow; keep stacking these foundations one day at a time.