Linear Algebra for ML

This post sets the foundation for the entire Math-for-ML thread: linear algebra is not optional background, it is the actual representation that every model uses. The goal here is to replace the vague feeling that 'ML is complicated math' with a concrete, four-object mental model you can hold in your head.

The promise of the post is modest but powerful — you don't need proofs or a semester of theory to be effective. You need to know what a scalar, vector, matrix, and tensor are, how your data maps onto them, and how shapes flow through operations. Get that, and the rest of ML stops feeling like magic.

Linear algebra is the study of vectors, matrices, and the linear operations (addition, scaling, multiplication) between them. The key reframe for an ML learner is that these objects aren't abstractions floating in a textbook — they are exactly how data and parameters are stored in memory. A row of features is a vector; a batch of rows is a matrix; the weights of a layer are a matrix too.

Because everything is numbers in arrays, 'training a model' reduces to doing arithmetic on those arrays repeatedly and efficiently. That's why frameworks like NumPy, PyTorch, and TensorFlow are, at their core, fast linear-algebra engines. Understanding the math is understanding what the framework is actually computing.

The four objects form a ladder of dimensionality. A scalar is a single number with no axes. A vector is a 1-D ordered list — think of one data point's features, like [bedrooms, bathrooms, sqft]. A matrix is a 2-D grid with rows and columns, like an entire dataset. A tensor generalizes this to any number of dimensions: a color image is a 3-D tensor (height × width × channels), and a batch of images is 4-D.

The word 'tensor' sounds intimidating but it just means 'n-dimensional array.' Everything you do at higher dimensions is the same arithmetic, applied along more axes. Once the ladder clicks, terms like 'a batch of sequences of embeddings' decode into a concrete shape rather than jargon.

The stack diagram makes the dimensionality ladder visual: each level adds one axis. Reading it top to bottom, you go from a single number, to a row of numbers, to a grid, to stacked grids. This is the single most useful picture to keep in mind when you see an unfamiliar shape in code.

When you later meet a shape like (32, 128, 768) in a Transformer, you can decode it by counting axes: 32 sequences in a batch, 128 tokens each, 768 numbers per token. The ladder tells you it's a 3-D tensor — three stacked levels above a scalar — and that's exactly how to reason about it.

This slide delivers the post's central insight: your dataset is literally a matrix. People picture ML as operating on 'houses' or 'customers,' but the model only ever sees a numeric grid. Each row is one example and each column is one feature. A table of 1000 houses described by 5 features is a 1000×5 matrix of floats.

This reframing matters because it explains why preprocessing (encoding categories as numbers, scaling features) is mandatory — the model can't consume anything that isn't already a number in the grid. It also explains why the model's job is to learn a set of weights that maps the feature columns to a target column.

The vectors diagram shows the geometric payoff: once a house is a vector like [3, 2], it becomes a point you can plot. Two houses are two points, and 'similar houses' literally means 'points that are close together.' This geometric view underlies clustering, nearest-neighbor methods, and the whole idea of similarity in ML.

The leap to make is that this works in any number of dimensions, not just two. We can only draw 2-D, but the math of distance and direction extends unchanged to 5, 50, or 768 dimensions. The picture is a crutch for intuition; the algebra is what actually runs.

Here we nail down what 'dimensions' means, because the word causes endless confusion. In ML, the number of dimensions of a data point equals the number of features — the number of columns. Two features means each example is a point in a 2-D plane; five features means a point in 5-D space.

You cannot visualize 5-D, and that's fine — the computer never visualizes anything. It manipulates coordinates with the same operations regardless of how many there are. 'High-dimensional data' is not mystical; it just means many feature columns. This also previews the curse of dimensionality: as columns grow, points spread out and distances become less informative.

This code slide grounds the abstract objects in something you can run. Creating a scalar, vector, and matrix in NumPy and printing their .shape attribute is the fastest way to internalize the dimensionality ladder. Notice the shapes: () for the scalar (no axes), (3,) for the vector (one axis of length 3), and (2,3) for the matrix (two rows, three columns).

Make a habit of running this kind of snippet whenever you're unsure. The .shape attribute is the ground truth of what an array actually is, independent of how it prints. Beginners who check shapes early avoid the majority of confusing errors later.

Shape literacy is arguably the highest-leverage skill in practical ML. A shape like (2,3) tells you everything about how an array can combine with others: which multiplications are legal, how broadcasting will behave, and what the output will look like. The overwhelming majority of day-to-day errors in ML code are shape mismatches, not algorithmic mistakes.

The skill to build is predictive: before you run a line, ask 'what shape comes out?' If you can answer that reliably, you can read model code top to bottom and trace data through it, which is exactly what debugging and paper-reading require.

This comparison bridges math notation and code, which trips up self-taught learners constantly. Papers use lowercase x for a vector and capital X for a matrix; the i-th example is written xᵢ and corresponds to a row of X. The expression wᵀx is a dot product — a weighted sum — which in code is simply w @ x.

Knowing this dictionary lets you move between a research paper and a Jupyter notebook without losing the thread. When you see wᵀx + b in a paper, you can immediately picture np.dot(w, x) + b in code, and vice versa. That fluency removes a huge amount of friction from learning.

These five takeaways are the durable core to memorize before moving on. Row equals one example and column equals one feature is the orientation convention almost all libraries assume. Vectors as points in space gives you geometric intuition. Matrices as transformations of vectors previews the next post. Shapes lining up is the rule that governs multiplication. And tensors as stacked matrices closes the dimensionality ladder.

If you remember nothing else from this post, remember these five lines. They are enough to keep you oriented through everything that follows in the series.

The teaser points forward to the 'why it matters' angle. Now that you have the vocabulary — the four objects, shapes, and the row/column convention — the next post argues why this math is non-negotiable rather than nice-to-have. We'll connect speed, geometry, and intuition to concrete ML wins so the motivation is unmistakable before we go deeper into mechanics.