Chain-of-Thought Prompting

This cover leads with the single most striking fact about CoT: a one-sentence prompt change roughly doubled accuracy on grade-school math for large models, with zero retraining. That's the kind of result that made the technique famous and reframed how people thought about prompting.

Post two is the 'so what.' Having defined CoT in post one, here we make the case for why it was a genuine inflection point — both practically, in the accuracy it delivers, and conceptually, in what it revealed about large models.

The first reason CoT matters is the raw accuracy gain. On benchmarks like GSM8K — a dataset of grade-school math word problems that require several arithmetic steps — large models given a direct prompt score surprisingly poorly, often in the teens to twenties percent. Prompted to reason step by step, the same models jump dramatically, into the fifties and beyond.

The crucial qualifier is 'the same model.' Nothing about the weights changed; the only difference is that the model was given room to work the problem out. That makes CoT a rare free lunch: a large capability gain available purely at inference time, through prompt design.

This is why CoT became a default technique for any task involving multi-step reasoning.

The bar chart puts the gain in visual terms with illustrative numbers in the spirit of the original results: a direct prompt around the high teens, zero-shot CoT lifting that to the forties, and well-chosen few-shot CoT pushing past fifty percent.

The exact figures vary by model and dataset, which is why these are framed as illustrative rather than exact. The shape, however, is robust and repeatedly reproduced: adding reasoning steps moves accuracy on multi-step problems by a large margin, and good exemplars beat a bare trigger phrase.

The takeaway is the magnitude — this is not a one or two point tweak, it's the difference between mostly wrong and mostly right.

The second reason CoT matters is conceptual: it is an emergent ability. Small models gain little or nothing from CoT prompting, and in some cases their attempts at reasoning actively degrade the answer. The benefit appears sharply only once the model crosses a scale threshold — in the original work, roughly the tens-of-billions-of-parameters range.

This made CoT a textbook example of emergence: a capability that is essentially absent at small scale and switches on once the model is large enough. It fed directly into the broader conversation about what new abilities appear as models grow.

For practitioners, the lesson is that CoT is not a universal trick — it's a behavior that big models have and small ones largely don't.

The second bar chart visualizes that emergence: at small scale CoT barely moves accuracy, at medium scale there's a slight gain, and at large scale it unlocks the capability and accuracy climbs steeply.

The non-linear, threshold-like shape is the point. If the relationship were smooth and linear you'd expect proportional gains at every size, but instead the technique is nearly useless below the threshold and transformative above it. This is exactly the signature of an emergent ability, and it's why claims about CoT have to be tied to model scale to be meaningful.

The third reason CoT matters is that it reframed prompting itself. Before CoT, the standard path to better reasoning was to collect reasoning data and fine-tune. CoT demonstrated that the capability was already latent in the pretrained weights — you just had to ask for the steps to surface it.

This shifted the mental model of prompting from 'phrasing a request nicely' to 'allocating computation.' More reasoning tokens mean more forward passes spent on the answer, so a prompt isn't just instructions — it's a way to control how much the model thinks. That reframing underpins a lot of modern prompt and inference-time-compute techniques.

The fourth reason is auditability. A bare answer is a black box; if it's wrong, you have no idea why. A chain of reasoning steps lets you locate the failure precisely — a misread quantity, a flipped operation, a wrong assumption carried forward.

This matters enormously in any setting where wrong answers are costly: finance, medicine, legal analysis, anything where a human needs to review the work. CoT turns 'the model said X' into 'the model said X because of steps 1 through 4,' and that traceability is often as valuable as the accuracy gain itself — with the caveat, raised later, that the chain is not a perfectly faithful trace.

This code slide makes the 'same model, different result' point tangible. The identical train-speed question is sent two ways: a direct prompt that asks for just a number, and a CoT prompt that asks the model to reason first. The direct version frequently fumbles the proportional arithmetic; the CoT version works it out and lands correctly far more often.

The code deliberately stops at constructing the two prompts rather than running them, to keep the focus on the difference in framing. It's the prompt design, not any new machinery, that drives the accuracy gap.

The mindmap shows that CoT wasn't an endpoint — it seeded a whole family of methods. Self-consistency samples many independent chains and takes a majority vote, smoothing out individual reasoning errors. Tree of Thoughts treats reasoning as a search over branching possibilities rather than a single linear chain. ReAct interleaves reasoning with actions, letting the model call tools or look things up mid-chain.

All three inherit CoT's core idea — make the model's intermediate reasoning explicit — and extend it. Understanding CoT is the prerequisite for understanding the more advanced reasoning techniques that followed.

The tips slide distills why CoT is a big deal: large accuracy gains with no retraining, an emergence above a scale threshold, the reframing of reasoning as a latent capability you elicit, the auditability of outputs, and the fact that it spawned a family of follow-on methods.

These five points are the 'so what' a reader should carry away — enough to justify reaching for CoT and to appreciate why it occupies such a central place in prompt engineering.

The closing mistake guards against over-application. CoT costs tokens, latency, and money, and on simple tasks the extra reasoning can introduce noise that makes answers worse, not better. It also can't manufacture knowledge the model lacks — it only helps when the model can do the steps but would otherwise skip them.

The honest framing is that CoT is powerful for a specific class of problems and a waste or a liability outside it. Treating it as a universal upgrade is a mistake, which is exactly the theme the final post of the series develops in depth.

The CTA hands off to post three, which opens the engine room: autoregression, the token-by-token feedback loop, why generating steps amounts to spending more compute, and how the trigger phrase steers generation. The teaser promises the mechanism behind the results just shown.