Activation Functions

This cover names the defining hazard of activation mistakes: they fail silently. A flat loss, neurons stuck outputting zero forever, or outputs that have quietly stopped being valid probabilities — none of these raise an error. Framing them as expected dangers to actively guard against sets the right defensive mindset for the post.

The common-mistakes angle works best as a failure manual, so each slide is a specific, nameable error with its fix. The cover lists the lineup — saturating hidden activations, dead ReLUs, double softmax, wrong output activation, and input saturation — so the reader knows they are getting concrete traps, not vague cautions.

Using sigmoid or tanh throughout deep hidden layers leads because it is the most fundamental activation mistake and ties straight back to the vanishing-gradient problem from posts 2 and 3. The mechanism is exact: their small derivatives compound toward zero across layers, so early layers receive almost no gradient and training stalls with no error message.

The fix is the modern default: ReLU or a variant in hidden layers, with sigmoid and tanh reserved for outputs or genuinely shallow networks where saturation across few layers is not yet fatal. Stating the fix as a default rather than a case-by-case decision gives beginners a reliable rule that avoids this trap in the vast majority of architectures.

Dead ReLUs are the flip side of ReLU's clean gradient, the cost of its hard zero. The mechanism is precise: a neuron whose input is always negative outputs zero and, because ReLU's derivative is zero there, receives zero gradient — so it can never adjust its way back to life. The neuron is permanently silent.

The common causes are worth naming: a too-high learning rate that overshoots weights into a bad region, or a large negative bias. Because a significant fraction of a layer can die this way, silently reducing the network's effective capacity, this is a real and underappreciated failure. The diagnosis sets up the LeakyReLU fix in the next slide.

This code slide pairs the dead-ReLU diagnosis with its concrete fix. The commented contrast — plain ReLU gives zero gradient for negative inputs, LeakyReLU's small 0.01 slope keeps a gradient alive — makes the remedy immediately copyable. The negative-side slope means a struggling neuron still receives signal and can recover.

The trailing comment about lowering the learning rate and checking weight initialization is important because LeakyReLU treats the symptom while these address the cause. A too-high learning rate is the most common reason neurons die in the first place, so the complete fix combines a more forgiving activation with sane training settings, not just a one-line swap.

Applying softmax twice is the activation mistake most specific to framework conventions, and it is invisible. The mechanism is that PyTorch's CrossEntropyLoss already applies log-softmax internally, so adding a softmax layer to the model means the operation runs twice — over-smoothing the distribution, flattening the gradients, and slowing or halting learning.

The fix restates the post 4 convention: output raw logits and let the loss handle softmax. Emphasizing that no exception is raised connects this to the day's theme of silent failure — the model trains, the loss decreases oddly slowly, and nothing signals the cause. This is the exact bug the build post was structured to prevent.

This compare diagram makes the double-softmax bug recognizable at a glance by contrasting the wrong and right model endings. On the left, a model ending in Softmax() fed to CrossEntropyLoss applies softmax twice and flattens gradients. On the right, a model ending in Linear() lets the loss apply log-softmax once, preserving healthy gradients.

Presenting it as a structural code pattern — what the last layer is — rather than abstract advice makes it actionable. A reader can scan their own model definition, check whether the final layer is a Softmax or a Linear, and immediately know whether they have the bug. That recognizability is the point of the visual.

The wrong-output-activation mistake covers the full space of mismatches: softmax on a regression head, no activation where a probability is needed, or sigmoid where softmax belonged. Each produces output that still runs but means nothing, which is why it is so easy to ship.

The fix is a clear mapping of task to output activation: linear for unbounded regression, sigmoid for a single binary probability, softmax for mutually exclusive classes, and independent sigmoids for multi-label problems where classes are not exclusive. Including the multi-label case is deliberate because it is the most commonly confused — people reach for softmax when labels can co-occur, forcing a false competition between classes.

This decision diagram operationalizes the output-activation choice into a quick flowchart a reader can run mentally before building any output layer. First ask whether the output is an unbounded number — if so, use no activation (linear). Otherwise, ask whether the classes are mutually exclusive — softmax if yes, sigmoid if no.

Reducing the choice to two questions removes the guesswork that causes the wrong-output mistake. The branch structure also encodes the subtle but critical distinction between exclusive classification (softmax) and per-label probabilities (sigmoid), which is the decision people most often get wrong. A reader who memorizes this tree will rarely mismatch an output activation again.

Unscaled inputs causing saturation is a setup mistake that disables learning before the first update. The mechanism is sharp: large raw values pushed into sigmoid or tanh saturate every neuron at 0 or 1 instantly, where the derivative is essentially zero, so no gradient flows. Even ReLU misbehaves with wildly scaled inputs.

The fix is to standardize inputs to roughly zero mean and unit variance before the first layer, which keeps the weighted sums in the range where activations are responsive and their derivatives are healthy. This connects activations back to data preprocessing — a reminder that the activation can only do its job if the inputs reaching it are in a sane range, which the next code slide implements.

This code slide implements the input-scaling fix with scikit-learn's StandardScaler and reinforces the train-only discipline from earlier in the series. The scaler learns mean and variance from the training set, then applies those same statistics to both train and test data.

The load-bearing comment — that standardizing brings the weighted sums into the activation's responsive range — ties the preprocessing directly to the saturation problem on the previous slide. Fitting on training data only also quietly prevents data leakage, the quieter cousin of saturation: scale correctly and the activations can actually learn from the very first batch.

The ReLU-versus-softmax confusion closes the post because it captures a conceptual mix-up that produces several of the earlier bugs. The two functions solve different problems: ReLU is a per-neuron hidden-layer nonlinearity, while softmax is a layer-wide output normalizer. They are not interchangeable.

The consequences of confusing them are concrete. Using ReLU as a classifier output gives raw unnormalized numbers instead of probabilities; using softmax in hidden layers forces neurons to compete and sum to one, crippling their ability to represent features independently. Keeping their jobs separate — ReLU for hidden, softmax for output — is the clean mental model that prevents both errors.

This recap consolidates all the fixes into a single pre-flight checklist a reader can run before trusting any network: ReLU-family in hidden layers, LeakyReLU plus a lower learning rate for dead neurons, raw logits with CrossEntropyLoss to avoid double softmax, output activation matched to the task, standardized inputs to avoid saturation, and the clean ReLU-hidden / softmax-output split.

The list is intentionally portable. Some items, like input standardization and matching the output to the task, apply to almost any model; others, like avoiding double softmax, are framework-specific habits that prevent the most silent damage in PyTorch. Together they form the practical wisdom that turns formula knowledge into working models.

The CTA closes both this post and the day, pointing forward to Loss Functions as the natural sequel. The framing is deliberate: this day repeatedly showed that the output activation must be paired with the right loss — softmax with cross-entropy, sigmoid with binary cross-entropy — so the loss function is the obvious next topic.

This creates a clean narrative arc. The reader finishes understanding what activations are, why the choice decides training, the math behind each, how to wire them in code, and how they fail — and is now primed to learn how a model measures its own wrongness, the other half of the training objective.