Activation Functions

This cover signals the gear change to hands-on code. The previous post gave the formulas; this one implements every activation by hand, verifies them, and then wires the correct ones into a real PyTorch network. That progression — derive, verify, apply — is exactly what the code-example angle demands.

The deliberate structure is to first build the activations in plain numpy so the math feels real and inspectable, then drop into PyTorch to use them idiomatically: ReLU in the hidden layers and the right output activation folded into the loss. Seeing both the from-scratch version and the framework version side by side is what makes the concepts portable.

This slide implements all four activations from scratch in a few compact lines: ReLU as np.maximum, sigmoid as the logistic formula, tanh via numpy's built-in, and the stable softmax with the max-subtraction trick from post 3. Having them together shows how little code each one really is.

Including the stability subtraction in softmax even in this minimal version is intentional — it reinforces the post 3 lesson and models the habit of writing the safe version by default. A reader who copies this block has correct, runnable implementations of every activation discussed in the day, which they can poke at and modify freely.

The verification slide closes the loop between the formulas of post 3 and the code of the previous slide. It runs all four activations on the same input vector and prints the results, letting the reader confirm the expected behavior: ReLU zeroes the negative, sigmoid lands in (0,1), tanh in (−1,1), and softmax sums to 1.

Verifying outputs is a practice worth modeling in itself. Rather than trusting an implementation, you feed known inputs and check the results match the math. The printed comments give expected values so a reader running the code can immediately spot a typo, building the habit of validating numerical code against hand-computable cases.

This slide shows the idiomatic PyTorch pattern: ReLU between every pair of Linear hidden layers, and crucially no activation on the final layer, which outputs raw logits. The architecture mirrors the day's recurring 20→64→32→3 classifier so the focus stays on the activations, not the data.

The comment that the loss applies the softmax is the load-bearing instruction and the setup for the next slide. Beginners instinctively want to add a softmax at the end, so explicitly leaving it out and explaining why is the single most important habit this post teaches. The output is logits because the loss function will handle the normalization more stably.

This slide explains the most counterintuitive convention in PyTorch: the output layer emits raw logits and CrossEntropyLoss applies log-softmax internally. The code shows the loss consuming raw model output during training, and separately shows applying torch.softmax explicitly only at inference when you actually want probabilities.

The separation of training and inference behavior is the key takeaway. During training you never apply softmax yourself; at inference you apply it once to read probabilities. This convention exists for numerical stability — combining log-softmax and the loss in one operation avoids precision problems — and getting it wrong is the double-softmax bug that post 5 dissects.

This explanatory slide states the rule that the previous code slides set up: do not put a softmax layer in your model when using CrossEntropyLoss. The mechanism is precise — the loss already applies log-softmax internally for stability, so an explicit softmax means applying it twice, which flattens the gradients and slows or stalls learning.

The practical guidance is clean: output raw logits during training, apply softmax only at inference for readable probabilities. Emphasizing that no exception is raised — the bug is silent — connects this directly to the day's overarching theme that activation mistakes degrade models without any error message, and it foreshadows the dedicated mistake slide in post 5.

This slide demonstrates how trivially activations can be swapped, replacing ReLU with LeakyReLU in one hidden layer and GELU in another. The point is that these are one-line changes; the rest of the pipeline — the loss, the training loop, the evaluation — is completely unchanged.

Showing the swap as effortless encourages experimentation, which is how a practitioner develops intuition for which activation helps on a given problem. It also reinforces post 3's framing of LeakyReLU and GELU as refinements of ReLU rather than fundamentally different tools: you can drop them into ReLU's slot without rethinking anything else.

The pipeline diagram abstracts the day's central practical lesson — activation depends on position — into a clean three-stage map. Hidden layers use ReLU or GELU; the output uses softmax, sigmoid, or nothing depending on the task. It is the portable rule a reader can carry to any network they build.

Separating hidden from output stages visually reinforces the post 2 distinction between activations that shape learning and activations that shape the answer. A reader who internalizes this diagram has the single most useful heuristic in the entire topic: ReLU-family in the middle, task-matched activation at the end.

This slide covers the binary-classification case, which uses a different output activation and loss than the multi-class examples. The model ends in a single logit, and BCEWithLogitsLoss combines a sigmoid with binary cross-entropy internally — the same logits-stay-raw convention as CrossEntropyLoss, just for the two-class case.

Applying torch.sigmoid only at inference mirrors the softmax pattern from earlier, reinforcing the consistent rule across the post: the loss handles the output activation during training, you apply it yourself only when you want a readable probability. Including the binary case makes the post's coverage complete, since binary and multi-class are the two most common output configurations a reader will hit.

This recap orders the build's lessons into an actionable checklist: ReLU or GELU in hidden layers, raw logits plus CrossEntropyLoss for multi-class, one logit plus BCEWithLogitsLoss for binary, never stack softmax before CrossEntropyLoss, apply softmax or sigmoid only at inference, and swap activations in one line.

Each bullet maps to a slide, so the recap doubles as an index and a portable template. The emphasis on never double-applying softmax and on the loss handling the output activation deliberately bakes in the habits whose omission causes the most common silent activation bugs — exactly the failures the final post is built around.

The CTA pivots from the happy path to the cautionary one. Having implemented and correctly wired every activation, the reader is ready to learn the specific ways the same code goes wrong — dead ReLUs, double softmax, wrong output activations, and input saturation.

Naming these traps in the teaser signals that real competence is knowing the failure modes, not just the happy path, and creates anticipation for the failure manual that closes the day.