Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Ask a 5-year-old "what is 17 x 24?" and they will guess. Maybe they will say 100, or 200, or some random number that feels right. Now ask them to work it out step by step: "First, what is 17 x 4? Now what is 17 x 20? Now add those together." Suddenly, with each sub-problem small enough to handle, even a child can reach the correct answer.

This is exactly what was happening with large language models before 2022. A 540-billion parameter model -- trained on essentially the entire internet -- could answer trivia questions, write essays, and translate languages. But ask it a grade-school math problem and it would confidently produce a wrong answer. The knowledge was there. The arithmetic capability was there. What was missing was a way to organize the computation into manageable steps.

Wei et al. (2022) discovered something remarkable: simply providing examples that include intermediate reasoning steps in the prompt caused the model to generate its own reasoning chains. No fine-tuning, no architecture changes, no new training data -- just a different kind of prompt. It is like showing your work in math class: the teacher does not make you smarter by requiring you to write out steps, but the act of writing forces you to think through each part sequentially, catching errors and building on intermediate results.

Wei et al. did not just show one example and declare victory. They ran a systematic, large-scale evaluation across three language model families (GPT-3, LaMDA, PaLM), five model sizes (ranging from 350M to 540B parameters), and eight reasoning benchmarks spanning arithmetic, commonsense, and symbolic reasoning. The results were remarkably consistent.

The experimental setup was elegant in its simplicity. For each benchmark, the authors wrote 8 exemplars -- question-answer pairs where the answer included a step-by-step reasoning chain. These exemplars were hand-crafted by the authors themselves, not generated by the model. The same 8 exemplars were used for all model sizes, allowing a clean comparison of how scale affects CoT's effectiveness.

The fundamental mechanism of CoT is the reasoning chain -- a sequence of intermediate natural language steps that bridge a question and its answer. Unlike formal proofs or structured programs, these chains use everyday language, making them both human-readable and model-friendly.

Why does writing out steps help? Because each intermediate step reduces the problem complexity. Consider the difference between solving "17 x 24" in your head versus on paper. On paper, you break it into 17 x 4 = 68 and 17 x 20 = 340, then add. Each sub-problem is trivial. The paper -- or in the model's case, the generated tokens -- acts as external working memory. The chain of thought is the model's scratch paper.

Chain-of-thought works across diverse reasoning domains: arithmetic reasoning (word problems), commonsense reasoning (physical world knowledge), symbolic reasoning (logical manipulation), and multi-hop reasoning (combining multiple facts). The key insight is that the chain acts as an external working memory, letting the model handle problems that exceed its single-pass computational capacity.

The original CoT paper used few-shot prompting: providing 8 exemplars, each with a question, a detailed reasoning chain, and the final answer. The model then generates a similar chain for new questions. But subsequent work discovered multiple variations, from zero-shot to self-consistency, each with different tradeoffs.

The choice of prompting strategy matters enormously. Self-consistency can boost GSM8K accuracy from 58.1% to 74.4% -- a massive 16-point improvement over basic CoT. Zero-shot CoT requires no examples at all and still captures a large fraction of the gains. Understanding these strategies is essential for getting the most out of modern language models.

Chain-of-thought is not just a clever trick -- it has deep computational foundations. A standard Transformer can only perform a fixed amount of computation per token (determined by its depth -- the number of layers). Complex reasoning problems may require more computation than a single forward pass provides. CoT solves this by letting the model trade inference time for computational depth.

Here is the key insight: each reasoning token the model generates gets fed back as input for the next prediction. This creates a feedback loop where the model can effectively perform sequential computation across multiple forward passes. A 5-step chain of thought means 5 forward passes, each building on the results of the previous -- dramatically more compute than a direct answer. It is analogous to how a 5-layer neural network can compute more complex functions than a 1-layer network.

One of the paper's most important findings is that CoT depends heavily on model scale. For small models (around 1B parameters), adding CoT prompting actually hurts performance -- the model generates plausible-sounding but incorrect reasoning steps. The benefits only emerge around 100B parameters and become dramatic at 540B.

This scale dependence is one of the strongest pieces of evidence for emergent abilities in large language models. It suggests that at sufficient scale, models develop qualitatively new capabilities -- not just more of the same. Chain-of-thought reasoning does not gradually improve with scale; it suddenly appears above a threshold. Below the threshold, CoT is actively harmful. Above it, it is transformative.

The CoT paper tested on 8 benchmarks across three categories: arithmetic reasoning, commonsense reasoning, and symbolic reasoning. The results were striking not just for their magnitude, but for their consistency across tasks.

The original CoT paper opened a floodgate of research into reasoning strategies. Within two years, dozens of variants emerged -- each optimizing for different tradeoffs between accuracy, cost, and applicability. From self-consistency (which boosts accuracy by sampling multiple chains) to tree-of-thought (which adds backtracking and search), the CoT family has grown remarkably rich.

The key dimension is the accuracy-cost tradeoff. Basic CoT is cheap but can be unreliable. Self-consistency samples 5-40 chains and votes, costing 5-40x more but significantly boosting accuracy. Tree-of-thought adds systematic search, yielding even better results at much higher cost. Understanding this frontier is essential for choosing the right strategy for your use case.

Q: Roger has 5 tennis balls. He buys 2 more cans of 3. How many now?
A: Roger started with 5 balls. 2 cans of 3 = 6 balls. 5 + 6 = 11. The answer is 11.

Q: [Your question here]
A:

Perhaps the most surprising follow-up to the original CoT paper was Kojima et al.'s (2022) discovery that you do not need any examples. Simply appending "Let's think step by step" to a prompt triggers chain-of-thought reasoning in large models. This zero-shot approach achieves a large fraction of few-shot CoT's gains with zero effort.

This finding has profound implications: it means the reasoning capability is already encoded in the model's weights from pretraining. The model has seen millions of examples of step-by-step reasoning in its training data (textbooks, tutorials, Stack Overflow). It already knows HOW to reason -- CoT prompting (whether few-shot or zero-shot) just tells it WHEN to reason.

Chain-of-thought is not just an academic curiosity -- it has become a fundamental technique used in virtually every modern AI application that involves reasoning. From math tutoring to code generation, from medical diagnosis to legal analysis, CoT is the backbone of how large language models handle complex tasks.

Chain-of-thought is powerful but not infallible. Understanding its failure modes is crucial for using it effectively. CoT can produce confident, detailed reasoning that leads to wrong answers. The chain can look perfect while hiding subtle errors that propagate through subsequent steps.

Knowing the theory is great, but how do you actually use CoT effectively in practice? Here are battle-tested guidelines from both the original paper and subsequent research, distilled into actionable advice.

Chain-of-thought prompting did not just improve benchmark scores -- it launched an entirely new paradigm in AI. The idea that models should "show their work" has become foundational to how we build, evaluate, and deploy language models. From prompt engineering to reasoning models, CoT's influence pervades every corner of modern AI.

The paper's citation count reflects its impact: it became one of the most-cited AI papers of its era. But more importantly, it changed how people think about AI capabilities. Before CoT, reasoning was considered something that needed to be explicitly trained. After CoT, we understood that reasoning capabilities could be latent in a model and elicited through the right prompts.