Chain-of-thought prompting breaks complex problems into intermediate reasoning steps before a final answer. Google researchers found that just eight such exemplars enabled a 540-billion-parameter model to beat a fine-tuned GPT-3 with a verifier on the GSM8K math benchmark, with gains also confirmed across three large language models on arithmetic, commonsense, and symbolic reasoning tasks.
What Is Chain-of-Thought Prompting? Definition and Nature
IBM defines chain-of-thought (CoT) as a prompt engineering technique that enhances the output of large language models, particularly for complex tasks involving multistep reasoningCITE:E5. Researchers at Google, in a paper posted to arXiv, describe chain of thought as a series of intermediate reasoning steps that significantly improves a large language model's ability to perform complex reasoningCITE:E1. Hugging Face adds that chain-of-thought works by providing a series of prompts that help a model "think" more thoroughly about a topic, generating more coherent and well-reasoned outputsCITE:E7.
How Does Chain-of-Thought Prompting Work? Breaking Reasoning Into Intermediate Steps
Chain-of-thought prompting simulates human-like reasoning by breaking an elaborate problem into manageable, intermediate steps that sequentially lead to a conclusive answer, IBM explainsCITE:E6. Hugging Face frames the same mechanism from a different angle: rather than one direct instruction, the model receives a series of prompts that guide it to reason through a topic before producing its final outputCITE:E7.
What Empirical Improvements Does Chain-of-Thought Prompting Deliver?
Google's researchers (Wei et al.) report that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks, based on experiments run across three large language modelsCITE:E4. The same paper states that generating a chain of thought significantly improves a large language model's ability to perform complex reasoningCITE:E1.
| Metric | Value | Source |
|---|
| Large language models tested | Three | CITE:E4 |
| Chain-of-thought exemplars used to reach SOTA on GSM8K | 8 | CITE:E3 |
| Model size achieving that result | 540 billion parameters | CITE:E3 |
How Do Few-Shot Exemplars Drive Reasoning Ability?
Reasoning abilities emerge naturally in sufficiently large language models once a prompt includes a few chain-of-thought demonstrations as exemplars, Google's researchers findCITE:E2. Prompting a 540-billion-parameter language model with just eight chain-of-thought exemplars achieves state-of-the-art accuracy on the GSM8K math word problem benchmark, surpassing even a fine-tuned GPT-3 model equipped with a verifierCITE:E3.
What Does This Mean?
IBM's framing of chain-of-thought as a technique for multistep reasoningCITE:E5 and Hugging Face's description of it as a series of guiding promptsCITE:E7 converge with Google's benchmark result: a 540-billion-parameter model needed only eight exemplars to beat a fine-tuned GPT-3 with a verifier on GSM8KCITE:E3, while the same underlying method improved arithmetic, commonsense, and symbolic reasoning across three tested modelsCITE:E4. Together, the vendor definitions and the benchmark result describe the same mechanism from two angles: what it is, and what it measurably changes.
Author's Take・林紀旭 James Lin
The notable detail here is the ratio, not just the outcome: eight exemplars were enough to push a 540-billion-parameter model past a fine-tuned GPT-3 that also relied on a verifier, on GSM8K. That suggests the exemplars are doing more work than raw parameter count alone would predict, since prompting substituted for an entire fine-tuning-plus-verifier pipeline. Because the underlying paper also reports gains across three separate large language models on arithmetic, commonsense, and symbolic reasoning tasks, the effect is not confined to one benchmark or architecture. Anyone evaluating chain-of-thought prompting for a reasoning-heavy task should track exemplar count against accuracy gain, not just model size against accuracy gain — the eight-exemplar result implies the former may matter more.