Linear Algebra for ML

This is the hands-on post: we build a complete neural-network forward pass using nothing but NumPy. The point is demystification. Frameworks like PyTorch wrap these operations in convenient layers, but underneath, a forward pass is just matrix multiplies, bias adds, and activations — exactly the operations from the previous post.

By writing it by hand and tracking shapes at every step, you'll see that there's no hidden magic in a dense layer. If you can read and run these snippets, you understand what a framework's Linear/Dense layer actually computes, which makes everything you build on top of frameworks far less mysterious.

We're building a two-layer feed-forward network: input, a hidden layer with a ReLU activation, then an output layer. The only operations involved are matrix multiplication, bias addition, and a max() for the activation. That's genuinely the entire core of a Dense (a.k.a. Linear or fully-connected) layer in any deep-learning framework.

Keeping the architecture tiny is intentional. A small net with concrete shapes lets you verify every step, and the structure scales without conceptual change — real networks just stack more layers of the same operations and learn the weights via gradient descent.

Step one sets up the data and the first layer's parameters. X is a (4,3) matrix: 4 examples, each with 3 features. We seed the random generator so results are reproducible. W1 is (3,5), mapping 3 input features to 5 hidden units, and b1 is a length-5 bias vector — one bias per hidden unit.

Notice how the weight shape encodes the layer's job: (input_dim, output_dim). Reading a weight matrix's shape tells you how many features go in and how many activations come out. This convention is the same one frameworks use internally for their dense layers.

Step two is the linear layer itself: z1 = X @ W1 + b1. The matmul (4,3)·(3,5) produces (4,5) — the inner 3's match and cancel, leaving 4 examples by 5 hidden units. Then the bias b1 of shape (5,) is broadcast across all 4 rows, adding the same per-unit bias to every example.

This one line is the heart of a neural network. Every dense layer in every framework computes exactly this. The shapes confirm the inner-dimension rule from the previous post and show broadcasting doing its job — the two ideas from before now working together in real code.

Step three adds the activation. ReLU (Rectified Linear Unit) is just max(0, x): it leaves positive values unchanged and clamps negatives to zero. Applied to z1, it produces a1 of the same shape (4,5) — activations never change the shape, only the values.

The reason we need an activation is crucial: without a non-linearity between layers, stacking linear layers collapses into a single linear transformation, and the network could only learn straight-line relationships. ReLU's simple bend is enough to let stacked layers approximate complex, curved functions, which is what gives deep networks their power.

Step four adds a second layer and produces the final output. W2 is (5,1), mapping the 5 hidden units down to a single output, with bias b2 of length 1. The multiply a1 @ W2 is (4,5)·(5,1) = (4,1): one prediction per example. .ravel() just flattens that (4,1) column into a flat array for easy reading.

That's a complete forward pass. The data entered as (4,3), flowed through a hidden layer to (4,5), passed through ReLU unchanged in shape, and emerged as (4,1) predictions. Every transition obeyed the inner-dimension rule, and the only operations used were matmul, bias add, and a max.

The network diagram shows the architecture as the classic node-and-edge picture: 3 input units, 5 hidden units, 1 output. Each layer is fully connected to the next, and those connections are exactly the entries of the weight matrices W1 (3×5) and W2 (5×1) we built.

Seeing the code and the diagram together closes the loop between the visual intuition most people have of neural networks and the matrix math that actually implements it. The 'lines between circles' you see in textbooks are literally the numbers in your weight matrices.

The pipeline diagram tracks the shape at each stage of the forward pass: X is (4,3), the first linear layer makes it (4,5), ReLU keeps it (4,5), and the second linear layer brings it to (4,1). This is the single most useful debugging artifact you can carry in your head when building networks.

The discipline of writing down the shape after every operation catches errors immediately. If a stage produces an unexpected shape, you know exactly which operation went wrong, instead of discovering a problem many lines later when something finally throws an error.

This slide makes the payoff explicit: what you just wrote is the whole trick. A real, deep network stacks more of these layers, may swap ReLU for other activations, and learns the weight matrices via gradient descent rather than random initialization — but the forward arithmetic is identical to what you coded.

What frameworks add is automatic differentiation (so gradients are computed for you) and GPU acceleration (so the matmuls run in parallel on dedicated hardware). They do not add new fundamental operations. This is why understanding the NumPy version transfers directly to understanding PyTorch or TensorFlow.

These checklist items are the practical habits that prevent the most common forward-pass bugs. Print .shape after every line so you always know what you're holding. Confirm inner dimensions match before each multiply. Make the bias length equal the number of output units. Remember activations preserve shape. And check that output rows equal the number of input examples.

Following this checklist turns building networks from a guessing game into a deterministic process. Most 'it doesn't work and I don't know why' situations dissolve when you verify shapes step by step.

The teaser points to the final angle in this day: common mistakes. Now that you've built a forward pass and seen how cleanly the shapes flow when everything is correct, the next post catalogs the specific errors that derail beginners — subtle shape confusions, broadcasting traps, and habits to avoid — so you can sidestep the hours they typically cost.