GPT vs BERT

This is the mechanics post, and the cover reframes the comparison around training objectives rather than architecture. The architectures differ, yes, but the deeper driver is the game each model is trained to play: GPT plays 'guess the next word' and BERT plays 'fill in the blank'. Those two objectives shape everything downstream.

Framing objectives as games is accurate and memorable. Neither model is ever told what words mean. Each just gets very good at a self-supervised prediction game on raw text, and the useful behavior — generation or understanding — emerges as a byproduct of mastering that game.

This comparison lays out the two objectives precisely before the deep dive. GPT's causal language modeling predicts the next token, sees only the past, and optimizes P(x_t given x_1 through x_{t-1}) — all self-supervised on raw text. BERT's masked language modeling predicts masked tokens using both sides, optimizing P(masked given surrounding context), and originally added a next-sentence-prediction task.

Stating the probability factorizations makes the difference rigorous. GPT factorizes the joint probability of a sequence left-to-right, which is exactly what enables generation. BERT does not model a left-to-right joint; it models conditionals of masked positions, which is what enables deep bidirectional representations but rules out clean generation.

This slide explains GPT's objective concretely. You feed in a sequence, shift the targets by one position, and train the model to predict each token from everything that came before it. The loss is standard cross-entropy over the vocabulary at every position, averaged across the sequence.

The profound part is what emerges. Nobody trains GPT to 'write well' — they train it to minimize next-token prediction error over trillions of tokens. Fluent generation, and eventually in-context learning, appear as side effects of becoming an excellent next-token predictor. The objective is humble; the emergent capability is not.

The trace diagram makes GPT's causal training visible step by step. Given 'The cat sat on', at t=1 the model sees 'The' and predicts 'cat'; at t=2 it sees 'The cat' and predicts 'sat'; at t=3 it sees 'The cat sat' and predicts 'on'. At every step, future tokens are masked out so the model can't cheat.

This step-by-step view is the clearest way to grasp autoregression. Each prediction depends only on the prefix, and the same mechanism used here for training is reused at inference time: feed the model its own output and let it extend the sequence one token at a time. Training and generation share the identical left-to-right structure.

This slide explains BERT's masked language modeling. Instead of predicting the next word, BERT corrupts about 15% of the input tokens and learns to recover the originals using context from both directions. Because the blanks sit in the middle of complete sentences, the model must fuse left and right context to fill them.

This bidirectional fusion is the entire source of BERT's understanding power. To predict a masked word in the middle of a sentence, the model genuinely has to integrate everything around it, producing representations far richer than a left-to-right model can. The cost of this power — as later slides show — is that BERT never learns to generate.

The trace diagram visualizes BERT's fill-in-the-blank training. The original sentence 'the cat sat on the mat' becomes 'the cat [MASK] on the mat', and the model must predict 'sat' using both the left context ('the cat') and the right context ('on the mat'). The comment makes explicit that both sides feed the prediction.

Contrasting this trace with the GPT trace on the previous slides is the payoff. Same kind of diagram, completely different information flow: GPT only ever looks backward, BERT looks both ways. Seeing them side by side cements why one generates and the other understands.

This slide covers an important implementation detail that's often glossed over: the 80/10/10 masking strategy. Of the 15% of tokens selected for prediction, BERT replaces 80% with the [MASK] symbol, swaps 10% for a random word, and leaves 10% unchanged.

The reason is subtle but important. The [MASK] token never appears during fine-tuning or inference, so if BERT only ever saw [MASK] during pretraining, it would learn representations specialized to a symbol it never encounters in real use. By sometimes showing a random or correct token, the strategy forces BERT to build robust representations of every token, not just to special-case [MASK]. It's a small trick with a real effect on downstream quality.

This code slide grounds the objective difference in actual Hugging Face classes. AutoModelForCausalLM loads GPT-2 with a causal LM head that predicts the next token. AutoModelForMaskedLM loads BERT with a masked LM head that predicts masked tokens. Printing the class names shows they're genuinely different model types.

The lesson is that the objective isn't an abstract concept — it's baked into which head sits on top of the transformer. The 'AutoModelFor...' naming convention in the library directly encodes the task, which is a practical detail the code post later builds on. Different objective, different head, different class.

This slide explains the direct consequence of BERT's objective: it can't generate well. Masked LM only ever asks the model to fill a handful of blanks inside otherwise-complete sentences. The model never practices building text from nothing, token by token, so it has no learned procedure for fluent left-to-right generation.

This is the crucial tradeoff to internalize. The very objective that grants BERT its bidirectional understanding — predicting masked middles using both sides — is exactly what removes its ability to generate. You can't have both from a single objective, which is precisely why GPT and BERT exist as separate models and why encoder-decoder models like T5 later tried to bridge the gap.

The pipeline diagram shows the shared training paradigm that both models follow despite their different objectives: pretrain, then fine-tune. Both start by pretraining on enormous amounts of raw text with their self-supervised objective. Then both fine-tune on a small labeled dataset for a specific task, and finally deploy — for generation or classification respectively.

This is the transfer-learning recipe that defines the modern NLP era. The expensive, general language learning happens once during pretraining; adapting to a specific task is cheap. Showing that GPT and BERT share this skeleton, differing only in the pretraining objective, ties the mechanics back to why both were such influential models.

The recap nails the mechanics into five carousel lines. GPT trains by predicting the next token; BERT trains by filling masked blanks; a causal mask hides the future for GPT; BERT sees both sides and therefore can't generate cleanly; and both pretrain on raw text then fine-tune.

These are the load-bearing ideas of the post. A reader who internalizes them understands not just that GPT generates and BERT understands, but the specific training objectives and attention masks that cause those behaviors.

The CTA closes the mechanics post and hands off to the hands-on code post. Having seen the objectives in principle, the natural next move is to run both models and watch the difference directly.

The teaser promises the concrete payoff: generating text with GPT-2 and classifying with BERT in real, runnable code, turning the abstract causal-versus-masked distinction into something the reader can execute.