Activation Functions

This cover frames post 3 as the mechanics, shifting from why activations matter to the exact math behind each one. The promise is reassuring: four short formulas, each with a characteristic shape and a derivative that explains its training behavior, cover the overwhelming majority of activations in practice.

The key organizing idea is that for each function the derivative is the most important part, because post 2 established that the derivative is what gates the gradient. So this post is structured to always connect a function's formula to the behavior its derivative produces, rather than presenting equations in isolation.

ReLU leads because it is the simplest and the default. f(z) = max(0, z) needs no exotic math — pass positives, zero negatives. The slide's real content is the derivative: 1 for positive inputs, 0 for negative, a clean on/off gate.

The practical consequence stated here previews post 5's dead-neuron problem. Because the gradient is exactly zero in the negative region, a neuron pushed permanently negative receives no gradient and cannot recover. This is the price of ReLU's clean positive-side gradient, and naming it now connects ReLU's greatest strength and its characteristic failure in a single slide.

The sigmoid slide gives the formula 1/(1+e^(-z)) and its key properties: it maps any real number into (0,1), so its output reads naturally as a probability. The slide's emphasis, again, is the derivative — f(z)·(1−f(z)) — which peaks at just 0.25 at z = 0 and collapses toward zero for large magnitudes.

That 0.25 peak is the load-bearing number. It means even in the best case sigmoid shrinks the gradient to a quarter of its size, and away from the center it shrinks it far more. Tying the formula directly to the vanishing-gradient problem from post 2 turns an abstract warning into a specific, quantified property of this exact function.

This code slide makes sigmoid and its derivative concrete and verifiable. It implements both functions and prints their values at z = 0, confirming the textbook facts: the output is 0.5 and the derivative is 0.25 at the center. Seeing the peak derivative as an actual printed number cements why sigmoid saturates.

Implementing the derivative as s*(1-s) rather than re-deriving it shows a useful practical trick — once you have the sigmoid value, its derivative is cheap. A reader can evaluate the derivative at larger inputs and watch it shrink, directly experiencing the saturation that post 2 demonstrated across layers.

The tanh slide presents the formula and its (−1, 1) range, then highlights what distinguishes it from sigmoid: it is zero-centered. Because its outputs are balanced around zero rather than all positive, the gradients flowing to the next layer are better behaved, which tends to make optimization smoother in practice.

The derivative, 1 − f(z)², peaks at a full 1.0 — four times sigmoid's peak — so tanh saturates less aggressively. But the slide is honest that it still saturates at the extremes, so tanh mitigates the vanishing-gradient problem without solving it. This positions tanh as a better squashing function than sigmoid but still inferior to ReLU for deep hidden layers.

This compare diagram organizes the four functions into two camps by their most consequential property. On the left, sigmoid and tanh are bounded and saturating: they flatten at the extremes and their derivatives shrink to zero there. On the right, ReLU and its variants are unbounded on the positive side with derivatives that stay healthy.

The grouping is more useful than memorizing four separate curves because it captures the one distinction that predicts training behavior. A reader who internalizes 'saturating versus non-saturating' can reason about a new activation they have never seen by asking which camp it falls into and what that implies for gradient flow.

The softmax slide handles the function that behaves differently from the rest. It acts on an entire vector: exponentiate each score, divide by the sum, yielding positive numbers that add to 1 — a probability distribution. This is why it belongs at multi-class output layers and nowhere else.

The numerical-stability note is practical and important. Exponentiating large scores overflows, so the standard implementation subtracts the maximum score first, which leaves the result unchanged mathematically but keeps the exponentials in a safe range. Flagging this here prevents the overflow bug and previews the stable implementation in the next code slide and post 4.

This code slide implements softmax with the stability trick built in. Subtracting np.max(z) before exponentiating is the load-bearing line — it prevents overflow on large inputs while leaving the output identical. The print confirms the outputs sum to 1, the defining property of a distribution.

Showing the stable version as the default, rather than the naive exp/sum, models good practice and explains why production implementations always include the subtraction. A reader who copies this gets a softmax that will not silently produce NaNs on large logits, a real bug that catches beginners who implement the formula literally.

This slide states the post's unifying thesis explicitly: the derivative is what decides trainability, because backprop multiplies it into the gradient at every layer. ReLU contributes a clean 1, so gradients survive depth; sigmoid and tanh contribute fractions that compound toward zero.

This is the precise, mechanical answer to why ReLU dominates hidden layers and saturating functions are confined to outputs. It ties every formula in the post back to post 2's gating argument, making clear that the choice between activations is really a choice about how their derivatives behave under repeated multiplication through depth.

The bars diagram visualizes the single most decision-relevant number for each function: its peak derivative. ReLU and tanh both reach 1.0, while sigmoid maxes out at just 0.25. The visual contrast makes the gradient-survival argument immediate and quantitative.

Showing ReLU and tanh tied at the peak is deliberately nuanced — tanh's peak is as high as ReLU's, but tanh only achieves it at z = 0 and saturates away from there, whereas ReLU holds its derivative at 1 across the entire positive range. The bar captures the peak; the earlier saturating-versus-non-saturating framing supplies the rest of the story.

This slide introduces the modern upgrades to ReLU. Leaky ReLU fixes dead neurons by allowing a small slope (such as 0.01·z) for negative inputs, so those neurons keep a nonzero gradient and can recover. GELU, the activation behind most transformers, is a smooth probabilistic variant that often trains marginally better.

The unifying point is that both variants preserve ReLU's healthy positive-side gradient while softening its hard zero. This frames the field's evolution as incremental refinement of ReLU's core strength rather than a search for something fundamentally different, and it previews the one-line swaps demonstrated in post 4.

This recap condenses the four formulas and their peak derivatives into a single reference card: ReLU's max(0,z) with derivative 1 or 0, sigmoid's 1/(1+e^-z) peaking at 0.25, tanh's zero-centered curve peaking at 1.0, and softmax's vector-wide normalization. The closing line restates the thesis: a healthy derivative is what makes depth trainable.

Ordered to mirror the post, the list lets a reader reconstruct each function and its key property from memory. It is also the direct setup for post 4, where these exact formulas are implemented in code and wired into a working network.

The CTA transitions from math to hands-on implementation. Having seen the formulas and derivatives, the reader is ready to write each activation in code, verify the values they just learned, and wire the right ones into a real PyTorch model.

Naming the deliverable — implement every activation and use them in a network — sets the expectation that post 4 is a buildable, runnable walkthrough, the practical reward after a formula-heavy post.