Linear Algebra for ML

This closing post is deliberately practical: it lists the six linear-algebra mistakes that cost beginners the most time. The framing is important — none of these are hard math. They're shape confusions and bad habits that produce either loud errors or, worse, silent wrong answers.

The value of cataloging them is preventive. Once you've seen the (n,) vs (n,1) trap or the silent-broadcasting bug named explicitly, you start recognizing them in your own code before they bite. Fixing these six dramatically improves the reliability of any matrix code you write.

Mistake one is the most common and most subtle: a 1-D array of shape (5,) is neither a row vector nor a column vector. It's a flat array that broadcasts and multiplies in ways that surprise people coming from a math background where vectors are explicitly rows or columns.

When you genuinely need a column vector — for instance to build a (5,1) output or to control how an operation broadcasts — reshape explicitly with .reshape(5,1) or .reshape(-1,1). A huge fraction of 'why is my result the wrong shape?' confusion traces directly back to treating a (n,) array as if it were a 2-D column.

This snippet exposes the (n,) trap concretely. v * v does elementwise multiplication and returns another (3,) array, not a matrix — multiplication of a 1-D array by itself is not a matrix product. Meanwhile v @ v computes the dot product and returns a single scalar (14), again not a matrix.

The fix shown is v.reshape(3,1) to get a true column vector. The lesson: when working with 1-D arrays, be deliberate about whether you want elementwise behavior, a dot product, or an actual 2-D vector, and reshape when the distinction matters. Printing shapes before and after operations makes the difference visible.

Mistake two is forgetting the inner-dimension rule. Attempting (2,3) @ (2,3) raises a ValueError because the inner dimensions — 3 and 2 — don't match. The fix is almost always to transpose one operand so the inner dimensions align, for example (2,3) @ (3,2).

The good news is that this mistake is loud: NumPy throws an error immediately rather than producing garbage. The habit to build is to mentally check (m,n)·(n,p) before every multiply. When the error appears, your first instinct should be 'do I need a transpose here?', which resolves it the vast majority of the time.

The compare diagram contrasts the failing and succeeding cases directly. On the left, (2,3) @ (2,3) has mismatched inner dimensions (3 vs 2) and raises a ValueError. On the right, (2,3) @ (3,2) has matching inner dimensions (3 == 3), is valid, and produces a (2,2) result.

Seeing them side by side reinforces the diagnostic: when a multiply fails, look at the inner two numbers. If they disagree, transpose one operand so they meet. This visual pairing of wrong-vs-right is the fastest way to lock the rule into reflex.

Mistake three is insidious because it doesn't error — it just ruins your results. If you feed the model a (features, examples) matrix where it expects (examples, features), the shapes may still be multiply-compatible, so everything runs. But the model now treats features as examples and vice versa, and learns nonsense.

The defense is to make 'rows = examples, columns = features' a hard convention you verify whenever data enters your pipeline. Because this bug shows up as poor accuracy rather than a crash, it can waste enormous amounts of time if you're not explicitly checking orientation. A quick X.shape sanity check against your expected (n_examples, n_features) usually catches it.

Mistake four is the dark side of broadcasting. Adding a (3,) bias to a (4,3) matrix works and is usually exactly what you want — the bias is applied per column across all rows. But adding a (4,) to a (4,3) fails because it can't align, and a (1,) added to anything succeeds quietly, which can mask a bug where you expected a per-element operation.

Broadcasting is powerful precisely because it's automatic, but that automation can hide mistakes. When combining arrays of different shapes, it's worth pausing to confirm the broadcast is doing what you intend rather than just what's legal.

This snippet demonstrates the broadcasting boundaries. Adding np.zeros(3) to a (4,3) works because (3,) aligns with the last axis and is applied to each row. Adding np.zeros(4) fails because (4,) cannot align with a trailing dimension of 3. Adding a (4,1) column works because the size-1 second dimension broadcasts across the 3 columns.

The practical guidance is in the comment: print shapes before adding when you're unsure. A ten-second shape check prevents both the loud failures and, more importantly, the quiet successes that aren't doing what you think.

Mistake five is reaching for Python loops where vectorized operations belong. Looping over array elements to sum or scale them is slow (interpreter overhead per element), longer to write, and more error-prone than the one-line vectorized equivalent. a @ b, X.sum(axis=0), and X * 2 do the same work using compiled, parallel code.

Beyond performance, vectorized code is usually clearer once you're fluent — it states what you want (the whole-array operation) rather than how to iterate. Whenever you catch yourself writing a for-loop over array indices in numerical code, treat it as a prompt to look for the vectorized form, which almost always exists.

Mistake six is ignoring the numerical scale of your data. Features on wildly different scales — age from 0–100 alongside income from 0 to a million — distort distance calculations, dominate gradients, and skew techniques like PCA that are sensitive to variance. Standardizing (subtracting the mean, dividing by the standard deviation) puts features on comparable footing.

Scale also causes numerical instability: large values fed into exp() during softmax can overflow. The standard fix is to subtract the maximum before exponentiating, which is mathematically equivalent but numerically safe. Respecting scale is a small habit that prevents a whole class of subtle training failures.

These habits collectively neutralize all six mistakes. Print shapes obsessively to stay oriented. Reshape (n,) to (n,1) when you need a real column. Confirm rows are examples to avoid silent orientation bugs. Prefer vectorized operations over loops for speed and clarity. And standardize features before training to keep distances, gradients, and numerics well-behaved.

None of these require advanced math — they're disciplines. Adopting them early is what separates code that mysteriously misbehaves from code you can trust and debug quickly. They compound: each habit makes the others easier to maintain.

The teaser closes the day and bridges to the next. With the language (Post 1), the motivation (Post 2), the mechanics (Post 3), a hands-on build (Post 4), and the pitfalls (Post 5) all covered, you have a working foundation in linear algebra for ML. The next day builds on this math to go deeper into the ML foundations that rest on top of it.