Linear Algebra for ML

This post shifts from motivation to mechanics. The claim is that four operations — dot product, matrix multiplication, transpose, and broadcasting — carry the vast majority of the arithmetic in ML. A forward pass through a neural network is mostly these, repeated layer after layer.

The teaching strategy is deliberate: work each operation on tiny, hand-checkable examples. When you can compute a 3-element dot product or a (2,3)·(3,2) multiply by hand, the million-element versions stop being intimidating, because you know they're the identical operation scaled up. This grounding is what makes the later code post feel obvious instead of magical.

The dot product is the atom of ML arithmetic. You multiply corresponding elements of two equal-length vectors and add the products into a single number: [1,2,3]·[4,5,6] = 4 + 10 + 18 = 32. That's the whole operation.

Its importance is that a single neuron computes exactly this: a weighted sum of its inputs (wᵀx) plus a bias. So when you understand the dot product, you understand what one neuron does. Everything else in a dense network is many dot products organized into a grid — which is precisely what matrix multiplication is.

The trace diagram walks the dot product line by line so there's no hand-waving. Inputs a and b are listed, then we form the elementwise products, sum them, and arrive at 32. Seeing each intermediate value makes the operation concrete and gives you a template to check any dot product by hand.

This kind of step-by-step trace is also a great debugging habit. When a computed value looks wrong, reproducing it on paper for a tiny example quickly reveals whether the bug is in your math, your shapes, or your data.

Matrix multiplication is the dot product applied in a grid. To compute A·B, you take each row of A and dot it with each column of B; the entry at position [i,j] of the result is row i of A dotted with column j of B. That single rule generates the entire output matrix.

This is why the dense (fully-connected) layer is the workhorse of neural networks: multiplying an input matrix by a weight matrix computes, in one operation, every neuron's weighted sum for every example in the batch. The efficiency comes from doing all those dot products together rather than one at a time.

The inner-dimension rule is the single most important shape rule in ML: (m,n)·(n,p) = (m,p). The two inner numbers must be equal or the multiplication is undefined; the two outer numbers become the shape of the result. So (2,3)·(3,4) is legal and yields (2,4), while (2,3)·(2,4) is illegal because the inner 3 and 2 disagree.

Internalizing this rule eliminates the majority of shape-mismatch errors. Before any multiply, glance at the inner dimensions: if they match, you're fine and you already know the output shape; if they don't, you probably need to transpose one operand.

The flow diagram visualizes the rule in action: A is (2,3), B is (3,2)... the inner 3's match and cancel, and the surviving outer dimensions (2 and 2) form the result. Reading it left to right, you can see how the inner dimensions 'meet' and disappear while the outer ones pass through to the output.

This mental animation — inner dims annihilate, outer dims survive — is worth memorizing. It lets you predict the shape of any chain of multiplications instantly, which is exactly what you need when tracing data through a multi-layer model.

This code slide confirms the rule with real output. A is (2,3) and B is (3,2); their product is (2,2), matching the inner-dimension prediction. Printing both the shape and the values lets you verify the arithmetic by hand: the top-left entry is row [1,2,3] dotted with column [1,0,1] = 1·1 + 2·0 + 3·1 = 4, exactly as shown.

The @ operator is the standard way to matrix-multiply in NumPy (and it works in PyTorch too). Practicing on small matrices like this builds the confidence to trust — and to debug — the large ones in real models.

The transpose swaps an array's rows and columns, turning a (2,3) into a (3,2). In math notation it's written Xᵀ. You'll reach for it constantly, usually to make two shapes compatible for a multiply when their inner dimensions don't currently line up.

Beyond plumbing, the transpose appears in core formulas everywhere: the normal equations for linear regression (XᵀX), covariance matrices, and the attention mechanism where keys are transposed to score against queries. Recognizing Xᵀ in a paper and knowing it just flips the axes removes a lot of friction when reading.

Broadcasting is NumPy's rule for combining arrays of different shapes without writing loops. When a dimension is size 1 or absent, NumPy virtually stretches it to match the other array. Adding a bias vector of shape (3,) to a (2,3) matrix copies that vector across both rows — which is exactly how you add a per-feature bias to a whole batch in a single line.

Broadcasting is what makes vectorized code concise, but it's a double-edged sword: because it succeeds silently in many cases, it can also hide bugs when shapes align in a way you didn't intend. Understanding its rules lets you use its power without getting bitten.

This snippet demonstrates broadcasting concretely: a bias vector b of shape (3,) is added to a (2,3) matrix X, and b is applied to every row. The output shows each column shifted by the corresponding bias value. No loop, no manual tiling — NumPy handles the replication.

This is precisely the operation in z = X @ W + b inside a neural layer: after the matrix multiply, a bias vector is broadcast across all examples in the batch. Recognizing the pattern here means you'll immediately understand the bias-add step when it appears in the next post's forward pass.

These five rules are the portable summary of the post. Dot product: multiply matching elements, then sum. Matmul: the inner dimensions must match. Result shape: the two outer dimensions. Transpose: flip axes to fix alignment. Broadcasting: size-1 (or missing) dimensions get stretched to fit.

With these five rules you can hand-trace almost any forward pass and predict every intermediate shape. They are the practical toolkit that the upcoming code post puts to work when we build a network from scratch.

The teaser leads into the code-heavy post. Now that you know the operations and their shape rules, the next post assembles them into a full forward pass written in pure NumPy — input to linear layer to activation to output — so you can watch the shapes flow through a real, runnable example.