Generators & Iterators

This is the engine-room post. Having covered what generators are and why they matter, we now look at the actual machinery: the iterator protocol, the desugaring of the for loop, the role of StopIteration, and the suspend/resume behavior of yield. Once you see the mechanism, none of the earlier behavior is mysterious.

The payoff of understanding the protocol is that you can both consume and *build* iterators confidently, and you can debug the weird cases — exhaustion, nested iteration, manual pumping — by reasoning about iter() and next() directly.

The iterator protocol is just two methods. __iter__ returns an iterator object; for an iterator, that's typically self. __next__ returns the next value or raises StopIteration when exhausted. Any object implementing both participates fully in for loops, comprehensions, unpacking, and the next() builtin.

The separation matters: __iter__ on an *iterable* (like a list) returns a fresh, independent iterator each time, which is why lists are reusable. An iterator's __iter__ returns itself, which is why a generator — being its own iterator — can't restart. This one design detail explains the single-use behavior cleanly.

This desugaring is the single most clarifying piece of the whole day. A for loop is not a primitive — it's shorthand for: call iter() once to get an iterator, then loop calling next() inside a try, breaking when StopIteration is raised. The variable binding and the body run between the next() call and the next iteration.

Memorize this rewrite and a dozen confusions dissolve. Why can you loop a generator? Because it satisfies the next() side. Why does it empty out? Because iter() returns the same exhausted iterator. Why does breaking early leave a generator mid-stream? Because the loop simply stops calling next(). It's all here in six lines.

The cycle diagram captures the rhythm of generator execution: the consumer calls next(), the body runs forward until it hits a yield, that value is handed back, the function pauses with all its state frozen, and on the next next() it resumes from exactly that point. Round and round until the body ends.

This is the visual to keep when reasoning about any generator. The function isn't running continuously in the background — it's frozen between your requests, springing forward one yield at a time. That stop-start dance is the literal implementation of laziness.

StopIteration as the exhaustion signal surprises people who expect a sentinel return value like None or -1. Python chose an exception so that any legitimate value — including None — can be yielded without ambiguity. The end of iteration is structurally distinct from any data value.

The convenient part for generator authors: you never raise StopIteration yourself. Running off the end of the function body, or hitting a bare return, raises it automatically. In fact, since Python 3.7, raising StopIteration manually inside a generator is converted to a RuntimeError to prevent subtle bugs — another reason to let the function end naturally.

yield freezing the frame is the deepest mechanical idea here. When execution reaches yield, the interpreter suspends the function and keeps its entire stack frame alive: the instruction pointer, every local variable, the loop counters, all of it. The yielded value goes to the caller, and the frame sits frozen.

On the next next(), execution resumes on the line *after* the yield with that frame restored exactly as it was. This is why locals persist between yields without any explicit state-saving on your part — the frame itself is the state. It's also why generators are cheap: there's one frozen frame, not a stored list of results.

Building an iterator by hand shows what the generator is doing for you. Countdown is a full iterator class: __init__ sets up state, __iter__ returns self, and __next__ either produces the next value or raises StopIteration. The state — self.n — lives in an instance attribute that you mutate on each call.

This works and is sometimes the right choice when the iterator needs to be reusable or carry rich state. But notice the boilerplate: three methods, manual StopIteration, explicit attribute management. The next slide shows the generator collapsing all of it into a few lines.

Here's the same Countdown logic as a four-line generator. The yield handles __iter__, __next__, and StopIteration automatically; the loop variable n is the state, living naturally in the function's frame. There's no class, no self, no manual exception.

Comparing the two side by side is the strongest argument for generators as the default. Unless you specifically need a reusable, restartable, or attribute-rich iterator object, the generator function expresses the same behavior with a fraction of the code and zero protocol boilerplate. This is why idiomatic Python reaches for yield first.

The comparison table distills the choice. Writing the class gives you explicit control and a reusable object, at the cost of verbosity and the risk of getting the protocol wrong. The generator gives you conciseness and automatic protocol handling, at the cost of being single-use.

In practice the generator wins the vast majority of the time. You write a class iterator only when you need behavior generators can't easily give — multiple independent iterations over the same object, or an iterator that exposes extra methods and attributes. Otherwise, yield is the answer.

This slide names the two syntaxes for creating generators, because beginners often know only one. A generator *function* uses yield and can contain arbitrary logic — loops, conditionals, multiple yields, try/finally. A generator *expression* is the lazy cousin of a list comprehension, written with parentheses, ideal for a single transformation or filter.

Both produce identical generator objects and obey the same single-use, lazy semantics. The choice is about complexity: reach for the expression when the logic fits on one line, and the function when you need real control flow. The parentheses-vs-brackets distinction also separates a genexp from a list comprehension, a frequent source of confusion.

The state-machine diagram names the four states a generator moves through: created (function called, body not started), running (executing toward a yield), suspended (paused at a yield), and exhausted (StopIteration raised, permanently done). Understanding these states explains every generator behavior.

Notably, exhausted is terminal — there's no transition back to created or running. That's the mechanical reason a generator can't be reused: once it reaches exhausted, every next() just re-raises StopIteration. If you internalize this little state machine, the single-use rule stops being a quirk and becomes an obvious consequence of the design.

The mistake slide targets the classic hand-rolled-iterator bug: defining __next__ without __iter__, or vice versa. Since the for loop calls iter() before it ever calls next(), an object missing __iter__ raises TypeError immediately, even though __next__ is perfectly correct.

The deeper lesson is that this entire class of error simply doesn't exist with generator functions, because yield wires up both methods plus StopIteration handling for free. The bug is a tax you pay only when you choose to write the protocol manually — which is itself a strong argument for preferring generators unless you have a specific reason not to.

The cover and CTA mark this as the mechanism post. With the protocol, the desugaring, and the suspend/resume model in hand, post four gets practical: a complete, runnable lazy pipeline that reads a file, filters, parses, and aggregates — putting every concept from posts one through three into working code.