What is NLP?

This third post is the mechanical core of the series, and the cover frames the central truth that motivates the entire pipeline: a model cannot operate on the word 'cat', only on numbers. Every step exists to bridge from symbols humans read to numbers machines compute, and back again.

The phrase 'hidden assembly line' is chosen on purpose. The pipeline runs invisibly behind every NLP product, and making it visible is what turns the black box into understandable plumbing.

The pipeline diagram gives the four-stage skeleton that every NLP system shares: tokenize, embed, model, decode. Text comes in on the left and a prediction or new text comes out on the right; the four stages are the transformations in between.

It's worth stressing that modern Transformers fuse some of these stages internally — embedding and contextual modeling happen inside one network — but the conceptual shape is unchanged. Holding this four-stage map in mind makes every NLP architecture, old or new, easier to place.

Tokenization is stage one and the most underestimated. Raw text must be split into units the model can index, and the modern choice is sub-word tokenization. Instead of one entry per word (which explodes the vocabulary and chokes on rare words), the tokenizer breaks words into reusable fragments: 'unhappiness' becomes 'un' + 'happi' + 'ness'.

This design elegantly handles words the model has never seen — typos, slang, technical terms — by composing them from known pieces. Each resulting token maps to an integer ID, which is the first numeric representation of the text and the input the rest of the pipeline consumes.

The code slide shows tokenization happening on a real model so the concept isn't abstract. Running BERT's tokenizer on 'NLP is unbelievable' returns sub-word pieces, including the '##' prefixes BERT uses to mark continuation fragments.

Seeing 'unbelievable' shatter into 'un', '##bel', '##iev', '##able' is the clearest possible demonstration of sub-word tokenization. It also quietly previews a mistake from the final post — token counts are not word counts, which is exactly why input-length limits surprise people.

Embeddings are stage two, where token IDs become dense vectors that place each word in a continuous 'meaning space'. The crucial property is that words appearing in similar contexts end up near each other geometrically, so distance in the space approximates similarity in meaning.

This is the move that makes language amenable to math. Once words are vectors, you can measure similarity, cluster topics, and feed the result into a neural network. The famous 'king − man + woman ≈ queen' arithmetic is a consequence of this geometry, and it's why embeddings were a watershed for the field.

The vector diagram visualizes meaning space in two dimensions for intuition. Real embeddings have hundreds of dimensions, but the principle is the same: 'king' and 'queen' cluster together because they share contexts, while 'dog' sits elsewhere.

The simplification to 2-D is a teaching device, not a claim about the real geometry. The takeaway is qualitative: similar meanings occupy nearby positions, and that spatial relationship is what later stages exploit to reason about language.

Attention is stage three and the breakthrough that powers modern NLP. A word's meaning is not fixed — 'bank' means something different next to 'river' than next to 'money'. Attention lets every token look at every other token in the sentence and reweight its own representation based on what's relevant.

This is why the Transformer was such a leap. Earlier models processed text left to right and struggled to connect distant words; attention gives every token direct access to the whole context at once. The contextual vectors that come out the other side encode meaning-in-context, not just dictionary definitions.

The flow diagram shows attention doing its job: the ambiguous phrase 'the river bank' goes in, attention lets 'bank' attend to 'river', and the output is a contextualized vector that resolves 'bank' to a shoreline rather than a financial institution.

This disambiguation is the single most important thing attention buys you. It's the difference between treating each word as an isolated symbol and treating it as a word that knows its neighbors — which is what fluent language understanding requires.

Decoding is the final stage, and its form depends on the task. For classification, the model's final vector goes through a softmax that produces a probability over labels — sentiment, intent, topic. For generation, the model runs a loop: predict the next token, append it to the input, predict again, and repeat until done.

That autoregressive loop is exactly how ChatGPT writes. There is no plan for the whole sentence in advance; the model produces one token at a time, each conditioned on everything written so far. Understanding this demystifies generation and explains both its fluency and its tendency to wander.

The full-pipeline code slide runs named-entity recognition end to end, exercising all four stages in one call. The grouped-entities NER pipeline tokenizes the sentence, embeds it, contextualizes it, and decodes per-token labels into clean entity spans: a person, an organization, and a location.

The output mapping each word to its entity type makes the abstract pipeline concrete. Everything discussed in the previous slides happens inside this single function call — the library hides the assembly line, but now the reader knows exactly what's running underneath.

The recap compresses the pipeline into five beats: split, embed, add context, decode, and the note that Transformers fuse the middle two. Reciting this sequence gives the reader a durable mental model they can apply to any NLP system they encounter.

The final bullet — Transformers fuse the middle — is the bridge to modern architectures. It reassures the reader that the classic four-stage view still applies even when a single neural network blurs the embed-and-model boundary internally.

The CTA hands off to the code-heavy fourth post, promising a full runnable example. Having seen the pipeline conceptually, the reader is now ready to execute it — to type the code, run it, and watch text turn into machine-readable output.

This is the natural pedagogical progression: definition, motivation, mechanism, then hands-on practice. The next post is where the reader stops reading and starts running.