TensorFlow in 8 Slides

This is the mechanics post, and its job is to demystify the single line that intimidates beginners: the gradient computation. The reassuring message up front is that nothing magical happens. TF records what you do on the way forward, then replays it backward applying the chain rule.

The whole day builds toward depth, and this is where the depth lives. Once the tape stops being a black box, training loops and optimizers become obvious rather than memorized incantations.

tf.GradientTape is the heart of TF 2.x autodiff. Entering its context tells TF to start logging operations onto a 'tape'. Variables are watched automatically, which is why model weights get gradients without any extra work; other tensors must be watched explicitly.

The mental image to hold is a recorder. Every op you perform inside the context is written down along with how to compute its own local derivative. By the time you reach the loss, the tape holds a complete, replayable recipe for the backward pass.

This snippet is deliberately scalar so the math is checkable by hand. y equals x squared plus two x; its derivative is two x plus two; at x equals three that is eight, exactly what tape.gradient returns. Seeing the number match the calculus builds trust that the machinery is just calculus, automated.

The structural point is the shape of the code: do the computation inside the with block, then call tape.gradient(output, input) afterward. That pattern — record, then ask — repeats in every training loop you will write.

Reverse-mode autodiff is the specific algorithm, and understanding why TF uses it is worth a paragraph. It walks from the final loss backward to the inputs, multiplying local derivatives via the chain rule. Its defining property is efficiency for the many-to-one case: one scalar output, many input parameters.

That is precisely the shape of deep learning, where a single loss depends on millions of weights. Reverse mode computes all those gradients in roughly one backward pass, whereas forward mode would need one pass per input. This is why every major framework uses reverse mode for training.

The flow diagram compresses the whole forward-records-backward-replays story into four nodes. The forward pass logs ops on the tape; the computation funnels into a single scalar loss; tape.gradient walks that recording backward; the result is one gradient per trainable variable.

Keeping this picture in mind explains several practical rules at once: why the loss must be a scalar, why you compute gradients after the forward pass rather than during, and why each variable comes back with its own gradient tensor of matching shape.

The watched-versus-unwatched distinction is the source of many 'why is my gradient None' questions. tf.Variable objects are watched by default, so weights just work. A tf.constant is not watched; if you want its gradient you must call tape.watch(t) inside the context.

The corollary is that anything computed outside the tape's context is invisible to it, so its gradient is None. When a gradient unexpectedly comes back None, the first two things to check are whether the tensor is a Variable and whether the computation actually happened inside the with block.

@tf.function is where the speed comes from, and the explanation matters for avoiding the next post's mistakes. Decorating a function makes TF trace it once into a static graph, then reuse that graph on subsequent calls. The graph fuses ops and removes per-line Python overhead.

The catch, flagged here and detailed later, is that tracing keys on input shape and dtype. New shapes trigger new traces, and Python scalar arguments can trigger a trace per value. Used well it is a large speedup; used carelessly it silently does nothing.

This is the canonical training step, and it ties every concept in the post together. Inside the tape: run the model forward with training=True so layers like Dropout behave correctly, then compute the loss. Outside the tape: get gradients for all trainable variables, then hand them to the optimizer to apply.

Wrapping it in @tf.function compiles this hot path into a graph for speed. This exact pattern — tape, loss, gradient, apply, decorated for performance — is the skeleton of essentially every custom TensorFlow training loop, so it is worth internalizing line by line.

The ordered steps restate the training loop as a checklist, stripped of code, so the sequence is unambiguous. Forward inside the tape, compute the scalar loss, take gradients of the loss with respect to the weights, apply them through the optimizer, and repeat over batches.

The value of seeing it as five discrete steps is that any custom loop you read later maps onto this template. When a training loop looks unfamiliar, find these five moves inside it and the rest is just bookkeeping.

The gotchas slide turns hard-won experience into prevention. A default tape can only be used for one gradient call; if you need several, create it with persistent=True and delete it when done. Operations outside the context are not recorded. A tf.constant needs explicit watching. And re-tracing on every new shape destroys the @tf.function speedup.

These four issues account for a large share of real debugging time with TF autodiff. Knowing them in advance turns a confusing afternoon into a thirty-second fix.

The cover and CTA frame this as the 'how it works' chapter. With the tape and graph machinery understood, the next post assembles everything into a complete, runnable program — load data, build, compile, train, evaluate, and save — so the abstractions become a working artifact.