Generators & Iterators

This post sets the vocabulary for everything that follows. People throw around "iterator" and "generator" interchangeably, but they're distinct ideas, and the confusion is the root of most generator bugs. The goal here is to fix the three words — iterable, iterator, generator — in your head before any code.

Think of it as a map of the territory. Once you can say precisely what each term means, the why and the how stop feeling like separate magic tricks and become one coherent system you can reason about.

An iterable and an iterator are easy to conflate because a list is both, in a sense — but technically a list is an iterable that knows how to *produce* an iterator on demand. The iterator is the stateful object that actually walks through the values, holding a cursor that says "I've handed out elements 0 through k; the next one is k+1."

The practical takeaway: iter() converts an iterable into a fresh iterator, and next() advances that iterator. A list can give you many independent iterators, which is why you can loop over it again and again. A generator object, as you'll see, *is* its own iterator — which is exactly why it can only be looped once.

A generator is the ergonomic shortcut for building an iterator. Instead of writing a class with __iter__ and __next__, you write what looks like an ordinary function but with yield in it. The moment Python sees yield anywhere in a function body, it flags the whole function as a generator function.

The surprising part for newcomers is that calling the function runs none of the body. It immediately returns a generator object, paused at the very top. Only when you pull the first value — via next() or a loop — does execution begin and run until the first yield. This deferred execution is the heart of laziness.

This slide makes the yield-versus-return distinction concrete. with_return builds the entire list in memory the instant it's called and hands it back whole. with_yield, despite looking similar, hands back a generator object and has executed zero of its body.

The mental shift is from "a function computes and returns a result" to "a generator function defines a process that produces results over time." return says "I'm done, here's the answer." yield says "here's the next piece — hold my place, I'll continue when you ask." That pause-and-resume behavior is what separates a stream from a value.

The diagram traces the chain that powers every loop you've ever written. You start with an iterable. Python calls iter() on it to obtain an iterator. Then it repeatedly calls next() on that iterator, each call surfacing one value, until the iterator signals it's done.

Understanding this chain demystifies a lot. The for loop isn't a primitive — it's syntactic sugar over iter() and next(). Once you internalize that, you understand why some objects can be looped and others can't, and why a generator slots into a for loop seamlessly: it satisfies the iterator end of this exact chain.

Laziness is the property that makes generators worth learning. "Lazy" means a value isn't computed until the moment it's actually needed. A list is eager: ask for [x*x for x in range(1000000)] and Python builds a million squares right now, whether you use them or not.

A generator defers each computation to the instant you pull it, and discards it afterward. This is what lets a generator stand in for an infinite sequence — the values genuinely don't exist until you ask, so there's no contradiction in describing "all the natural numbers." You only ever conjure the finite prefix you actually consume.

This slide reveals that a for loop is just repeated next() calls with the StopIteration handling done for you. The generator expression (n*n for n in range(3)) produces a generator; each next() call yields the next square, and the fourth call raises StopIteration because the underlying range is exhausted.

Seeing next() raise StopIteration manually is valuable because it's normally invisible — the for loop catches it silently. When you understand that the loop's exit is an exception being caught, edge cases like nested iteration and manual pumping of generators stop being mysterious.

Recognizing the iterators you already use makes the concept feel less academic. range gives you a lazy sequence of integers. enumerate pairs each item with an index. zip walks several iterables in lockstep. map and filter wrap another iterable lazily. A file object yields its lines one at a time.

The point is that Python's standard library is saturated with this protocol, and you've been a fluent consumer of it without naming it. Learning to *produce* iterators with generators just moves you from the consuming side to the building side of a system you already rely on constantly.

These five lines are the compression of the whole post. Keep them and you can reason about almost any generator question. Iterable means loopable; iterator means it tracks where it is; a generator is the cheap way to make one; yield pauses while return ends; and values arrive one at a time, only when asked.

The last point — on demand — is the through-line into the next post about why this matters. Everything generators are good for flows from the fact that nothing is computed or stored until the consumer pulls it.

The comparison crystallizes the trade-off. A list front-loads all the work and all the memory in exchange for being reusable, indexable, and length-aware. A generator front-loads nothing, holding a single value at a time, in exchange for being single-use and opaque about its size.

Neither is universally better — they're different tools. The skill is matching them: lists for small, reusable, randomly accessed data; generators for large, streamed, or one-pass data. The rest of this day is largely about developing the judgment to choose correctly.

This is the single most common beginner mistake, which is why it closes the post. A generator object prints as <generator object> precisely because it isn't a container of values — it's a suspended computation. Trying to index it, call len() on it, or loop it twice all fail or silently misbehave.

The fix is conceptual, not syntactic: stop picturing a box of values and start picturing a recipe that produces them one at a time. Once that image is in place, the "weird" behaviors — exhaustion, no length, no indexing — become obvious consequences rather than surprises.

The cover and CTA bookend the series. This first post deliberately stays at the level of definitions and mental models, because the next four build directly on this vocabulary. Day 22 picks up where this leaves off, moving from "what these things are" to "why you'd ever reach for them" — the memory, streaming, and pipeline wins that justify the whole concept.