Chain-of-Thought Prompting

This cover sets the agenda for the mechanics post and corrects the most common misframing in one line: CoT isn't the model 'showing its work,' it's the model giving itself more tokens to compute over before committing to an answer.

Post three is the engine room. We've established what CoT is and why it matters; now we explain the actual mechanism — autoregression, conditioning, and the compute argument — so that the technique stops feeling like folklore and starts feeling like something you can reason about.

The foundation is autoregression. A language model predicts a probability distribution over the next token given everything before it, samples or picks a token, appends it to the sequence, and repeats. Critically, the text the model has generated so far is fed back in as part of its own input on the next step.

That feedback loop is the entire basis of Chain-of-Thought. The reasoning tokens the model writes don't vanish — they become context that shapes every subsequent prediction. Once you internalize that the model is reading its own output, CoT stops being mysterious: it's just the model conditioning on the steps it has already produced.

The cycle diagram makes the loop explicit: read the context (prompt plus everything generated so far), predict the next-token distribution, append the chosen token, and repeat until a stop condition.

The loop runs once per token. So a ten-token answer is ten passes through this cycle, and a 200-token chain of reasoning is 200 passes. This is the structural fact that the next slide turns into the central compute argument — every reasoning token is another trip around this loop, another increment of computation toward the final answer.

Here is the compute argument stated directly. A single forward pass performs a fixed amount of computation. A hard problem may simply require more computation than one pass can deliver. By generating intermediate steps, the model runs many forward passes before it has to emit the final token — effectively spending more compute on the problem.

This reframes 'thinking step by step' as buying a thinking budget. The model isn't accessing hidden knowledge it was withholding; it's giving itself more sequential computation to combine partial results. That's why CoT helps most on problems whose difficulty comes from needing several composed operations.

The conditioning flow diagram traces a concrete chain: step one computes 20 x 17 = 340, which conditions step two computing 3 x 17 = 51, which conditions the final sum 340 + 51 = 391.

The word 'conditions' between steps is the crux. Once a true intermediate result sits in the context, it directly shapes the probability distribution over what comes next. The model isn't re-deriving everything from scratch at each step — it's building on the partial results it already wrote down, the same way a person works a problem on paper.

This slide explains why conditioning improves accuracy. Once '20 x 17 = 340' is in the context, the model's distribution for the next tokens collapses sharply toward correctly continuing the arithmetic — the right next step becomes far more probable than it would be if the model tried to leap straight to the final answer.

In other words, each written step removes uncertainty from the next one. A direct answer asks the model to resolve all the uncertainty at once, in a single distribution over final answers, which is exactly where it tends to slip. CoT breaks that into a series of low-uncertainty moves.

This zero-shot code slide shows the mechanism in action with no exemplars: a multi-step word problem plus 'Let's think step by step.' The comment shows the model generating the intermediate computations (1.2 x 3 = 3.6, then 5 - 3.6 = 1.4) before stating the final answer.

The point is that with zero-shot CoT, the model invents the reasoning structure itself, guided only by the trigger phrase. Each line it writes is fed back in and conditions the next, exactly as the diagrams describe. This is autoregression and conditioning made concrete in a real call.

The few-shot code slide shows the alternative mechanism: instead of a trigger phrase, you seed the prompt with worked examples that demonstrate the reasoning format. The model then imitates that pattern on the new question (3 x 6 = 18, Answer: 18).

Mechanically, the exemplars condition the model just like any other context — they shift its distribution toward producing the same kind of step-by-step structure. Few-shot CoT is often more reliable precisely because you're demonstrating the exact format you want rather than hoping the trigger phrase lands the right behavior.

This slide explains why the trigger phrase works at all. 'Let's think step by step' is effective because, in the model's training data, that exact phrasing is overwhelmingly followed by step-by-step reasoning. The phrase shifts the model into a region of its output distribution where structured reasoning continues, instead of a terse, immediate answer.

The key nuance is that the phrase adds no knowledge — it selects a behavior the model already learned during pretraining. That's why slightly different phrasings can have noticeably different effects, and why finding good trigger phrasing is itself a small part of prompt engineering.

The tips slide covers the decoding settings that make CoT work in practice. Give enough max-tokens for the full chain or it gets truncated. Use a moderate temperature when you plan to do self-consistency sampling, so chains differ. Ask for the answer on a clearly marked line so it's parseable. Parse that final line rather than the whole text. And add a stop sequence after the answer to avoid paying for trailing tokens.

These are the practical knobs that turn the mechanism into reliable output, and they set up the code-heavy post that follows.

The closing mistake names the most common operational failure: cutting the chain off too early. Because the final answer comes after the reasoning, a max-tokens cap set too low truncates the chain and you get a half-finished thought with no usable conclusion.

The fix is to budget tokens for the entire chain plus the answer line. This single mistake accounts for a large share of 'my CoT prompt stopped working' reports, and it's entirely avoidable once you understand that the answer is always downstream of the reasoning in the token stream.

The CTA hands off to post four, the code-heavy one: real CoT prompts, robust answer parsing, and self-consistency voting. The teaser promises concrete, runnable patterns to take the mechanics just explained and turn them into working code.