Activation Functions

This cover makes the persuasive case for the post: the same architecture can train smoothly or refuse to learn entirely, and frequently the only difference is the activation function. Leading with that surprising claim answers the 'why should I care about a function detail' question the angle demands.

The post pivots from post 1's neutral 'what it is' to a sharper 'here is why this one choice decides success or failure'. Each slide is a distinct reason — expressive power, gradient flow, vanishing gradients, ReLU's breakthrough, output matching, and the cost of error — chosen so the post argues a structured case rather than repeating 'it's important' five times.

Expressive power leads because it is the most fundamental reason activations matter. The slide ties directly back to post 1's collapse argument: without nonlinearity, any depth yields only linear boundaries. With it, stacked layers compose into arbitrarily complex shapes.

The nuance worth dwelling on is that the universal approximation theorem rests on the activation, not on depth alone. It is the nonlinearity that grants the capacity to approximate almost any function; depth merely makes that approximation efficient. So the activation is not a detail bolted onto a powerful model — it is the source of the power, which is why getting it right is foundational rather than cosmetic.

This slide introduces the second and more practical reason activations matter: they gate the gradient. Backpropagation works by multiplying derivatives layer by layer, and the activation's derivative is one of the factors at every step. A healthy derivative lets the learning signal pass; a tiny one chokes it.

The reframing here is that an activation has two distinct jobs, and beginners usually see only the first. Forward, it shapes the representation. Backward, it controls whether the early layers receive any learning signal at all. This dual role is why a function can look perfectly reasonable in the forward pass yet quietly prevent the network from training — the problem lives in the backward pass.

The compare diagram contrasts the gradient behavior of saturating functions against ReLU, making the abstract 'gating' idea concrete. On the left, sigmoid and tanh have derivatives near zero at their extremes, so gradients shrink each layer and early layers barely learn. On the right, ReLU's derivative is exactly 1 for positive inputs, so the gradient passes through unchanged.

Seeing the two side by side explains in one image why deep networks were hard to train before ReLU and easy after. It is the visual anchor for the whole post: the difference between an activation that lets the signal survive depth and one that strangles it before it reaches the bottom.

This slide explains the vanishing-gradient problem mechanically rather than as a vague warning. Sigmoid's derivative tops out at 0.25 and tanh's at 1.0, but both fall toward zero away from the origin. Backprop multiplies these factors across layers, so in a deep saturating network the gradient can shrink by orders of magnitude before reaching the first layers.

The consequence — early layers receiving almost no signal and effectively freezing — is the precise failure that stalled deep learning for years. Naming the numbers (0.25, 1.0) and the multiplication makes it a concrete, calculable phenomenon, not folklore, which is exactly what the next code slide demonstrates.

This code slide turns the vanishing-gradient explanation into a runnable demonstration. It defines sigmoid's derivative, then multiplies it across ten saturated layers and prints the result — roughly 1e-9, a gradient that has all but disappeared. Watching the number collapse is far more persuasive than the verbal claim.

The choice to evaluate the derivative at z = 3.0 (a moderately saturated input) is deliberate: it shows the problem does not require extreme values, just a moderately deep stack of saturating units. A reader can lower the depth or move toward z = 0 and watch the gradient survive longer, which builds genuine intuition for when saturation bites.

This slide delivers the post's pivotal historical point: ReLU's derivative of exactly 1 for positive inputs is what made very deep networks trainable. Because the gradient passes through active ReLU units unchanged, the multiplication that destroyed signal in saturating networks no longer shrinks it.

The practical takeaway is that this single property, not a complex new theory, ended the long stall where deeper networks performed worse. It explains why ReLU and its variants are the near-universal default in modern hidden layers. Framing it as the unlock rather than just 'a good activation' gives the reader the right weight for the choice.

The timeline grounds the ReLU breakthrough in history, showing the arc from the sigmoid/tanh era where deep nets stalled, through ReLU going mainstream around 2011, to the 100+ layer networks of 2015 that combined ReLU with residual connections and normalization. It frames activations as a pivotal piece of a larger story.

Including residuals and normalization in the final entry is honest: ReLU was necessary but not sufficient for extreme depth. This prevents the oversimplification that ReLU alone solved everything, while still crediting it as the change that made the gradient survive — the foundation the later tricks built on.

This slide draws the crucial distinction between hidden-layer and output activations. Hidden activations shape how the network learns; the output activation shapes what the answer means. Sigmoid yields a single probability, softmax a distribution over exclusive classes, and linear (no activation) an unbounded regression value.

The warning that mismatch produces nonsense even when training runs is the practical heart of the slide. A probability above 1 or class scores that do not sum to one are signs of a wrong output activation, and crucially the code does not error — it just produces meaningless numbers. This is the conceptual setup for the decision diagram and the wrong-output mistake in post 5.

This slide tallies the concrete costs of choosing wrong, reinforcing why the topic earns a whole day. A saturating activation deep in the hidden stack makes training crawl or freeze; the wrong output activation makes the numbers meaningless; pairing softmax with the wrong loss double-applies it.

The unifying theme — none of these throw an error — is what makes activation mistakes uniquely dangerous and is the bridge to post 5's failure manual. The model either underperforms quietly or produces confident nonsense, so a practitioner who does not understand activations cannot even tell that something is wrong, which is precisely why the choice gets so much attention.

This recap distills the post's six reasons into one screenshot-able list: nonlinearity grants expressive power, the derivative gates the gradient, saturation causes vanishing gradients, ReLU's flat-1 slope unlocked depth, output activation must match the task, and wrong choices fail silently.

Ordered to mirror the post's argument, the list lets a reader reconstruct the full case from memory. It also serves as the bridge into post 3, where each of these claims is backed by the actual formulas and derivatives, turning the persuasive 'why' into mechanical 'how'.

The CTA hands off to the mechanics post. Having been convinced that the activation choice decides whether a network trains, the reader is primed to see the exact formulas and derivatives that produce these behaviors.

Naming the deliverable — formulas, shapes, and derivatives of each function — sets the expectation that post 3 is concrete math, the payoff for accepting post 2's argument that the derivative is what matters most.