Chain-of-thought (CoT) is the technique of having a language model produce explicit intermediate reasoning steps before arriving at a final answer. Instead of generating a direct response to a question, the model “shows its work” — decomposing a problem into logical sub-steps, working through each one, and building toward a conclusion. This simple idea, first rigorously demonstrated by Google researchers Wei et al. in January 2022, produced one of the largest capability jumps in the history of language models: accuracy on grade-school math problems (GSM8K) improved from 18% to 57% on PaLM 540B just by changing the prompt format.
Chain-of-thought did not make models smarter. It revealed that models already possessed latent reasoning abilities that were suppressed by the standard prompt-answer format. By giving the model “space to think” in the output, CoT unlocked capabilities that direct prompting could not access. This insight has driven the entire trajectory of AI reasoning research from 2022 through 2026, culminating in dedicated reasoning models that generate thousands of internal thinking tokens before responding.
How chain-of-thought works
To understand why CoT works, consider what happens inside a transformer when it generates a response:
Without CoT, the model must compress all reasoning into the forward passes that produce the first output token. For a math problem, this means the model has to “figure out the answer” while computing the hidden states for tokens like “The answer is…” — giving it no room to perform intermediate calculations.
With CoT, each intermediate step the model writes becomes part of its context for subsequent tokens. When the model writes “First, there are 15 apples…” that text gets fed back as input for the next token. The model effectively uses its own output as a scratchpad — each reasoning step provides additional conditioning that guides subsequent computation.
This is sometimes called chain-of-thought as latent computation: the model uses output tokens as additional compute, bypassing the fixed computational budget of a single forward pass. Each reasoning token adds the equivalent of another layer of processing. A response that generates 500 reasoning tokens before the answer has, in effect, 500 additional “processing steps” compared to a direct answer.
The three original CoT approaches
Few-shot CoT (Wei et al., 2022): Include 4-8 examples in the prompt, each showing a question followed by step-by-step reasoning and a final answer. The model follows the demonstrated reasoning pattern for the new question. This was the original formulation and remains effective, though it consumes significant prompt tokens.
Zero-shot CoT (Kojima et al., 2022): Append the phrase “Let’s think step by step” to the end of the prompt. Remarkably, this simple instruction — without any examples — produces structured reasoning that dramatically improves accuracy. Zero-shot CoT works because instruction-tuned models have been trained on enough reasoning examples that they can generate step-by-step logic on command.
Self-consistency (Wang et al., 2022): Sample multiple independent CoT reasoning paths for the same problem, then select the answer that appears most frequently across samples (majority voting). This reduces the impact of individual reasoning errors. Sampling 10-40 reasoning paths with self-consistency can improve accuracy by an additional 5-15 percentage points beyond single-path CoT, at the cost of proportionally higher token usage.
The evolution from prompting technique to model architecture
Chain-of-thought began as a prompting trick but has evolved into a fundamental architectural feature of frontier AI systems. This evolution has four stages:
Stage 1: Prompt engineering (2022)
CoT was a prompt-level technique applied to base or instruction-tuned models. The model’s weights were unchanged; only the input format differed. Researchers and developers manually crafted prompts that elicited reasoning. Effectiveness varied across models and tasks, and the reasoning quality was inconsistent.
Stage 2: RLHF-trained reasoning (2023)
Models were fine-tuned with reinforcement learning from human feedback (RLHF) to produce better reasoning chains. ChatGPT, Claude, and similar models were trained on human preference data that favored detailed, step-by-step explanations. This made CoT behavior more reliable and consistent without explicit prompting, but the reasoning was still generated as visible output.
Stage 3: Dedicated reasoning models (2024-2025)
OpenAI’s o1 (September 2024) represented a paradigm shift. Rather than generating visible reasoning in the response, o1 performed extended reasoning in a hidden “thinking” block, spending thousands of tokens on internal deliberation before producing a response. This reasoning was trained through large-scale reinforcement learning on verifiable tasks (math, coding) where the model could be rewarded for correct final answers regardless of the reasoning path.
The key insight was test-time compute scaling: by spending more compute at inference (generating more thinking tokens), models could solve harder problems. Where traditional scaling focused on making models larger (training-time compute), reasoning models showed that inference-time compute was an orthogonal scaling axis.
Anthropic’s Claude with extended thinking and Google’s Gemini 2.5 Flash and Pro with thinking followed similar approaches, each providing a structured thinking block that performs chain-of-thought reasoning before the visible response.
Stage 4: Hybrid and controllable reasoning (2025-2026)
Current frontier models offer controllable reasoning budgets — users can specify how much thinking the model should do based on task complexity. Claude with extended thinking lets developers set a budget_tokens parameter. Gemini’s thinking mode can be toggled. This addresses the core tradeoff: reasoning improves quality but costs tokens, time, and money. Not every question needs 10,000 thinking tokens.
The field has also developed distillation techniques that transfer reasoning capabilities from large, slow reasoning models to smaller, faster ones. DeepSeek-R1 demonstrated that reasoning patterns learned by a large model through RL could be distilled into much smaller models (14B, 7B, even 1.5B parameters) that retain substantial reasoning ability at a fraction of the cost.
The reasoning model taxonomy
Not all reasoning approaches are equivalent. A useful framework distinguishes four types:
| Type | Description | Example | Thinking visible? | Compute scaling |
|---|---|---|---|---|
| Prompt-elicited CoT | User asks model to reason step by step | Any LLM with “think step by step” | Yes, in response | Fixed by response length |
| Trained CoT | Model fine-tuned to reason by default | Claude Sonnet, GPT-4.1 | Yes, in response | Fixed by response length |
| Hidden reasoning | Extended thinking in hidden block | o1, o3, Claude extended thinking | Thinking block (may be summarized) | Scales with thinking budget |
| Reasoning + search | CoT combined with tree search over solution space | o3 on hard math/science, AlphaProof | Partial or hidden | Scales with search budget |
The most powerful reasoning systems combine hidden CoT with search — exploring multiple reasoning paths, backtracking from dead ends, and verifying steps against ground truth. This is how o3 achieved its breakthrough scores on ARC-AGI and competition mathematics, though at extreme compute costs (potentially hundreds of dollars per hard problem).
Where chain-of-thought helps — and where it does not
CoT is not universally beneficial. Understanding where it helps is essential for effective deployment:
Tasks where CoT dramatically improves performance
- Mathematical reasoning. GSM8K (grade-school math) accuracy jumped from 18% to 57% with CoT on PaLM 540B, and from ~60% to 95%+ with modern reasoning models. MATH benchmark (competition-level math) went from near-zero to 80%+ with o3-level reasoning.
- Multi-step logic. Tasks requiring chaining 3+ logical inferences — syllogisms, constraint satisfaction, scheduling problems — see 20-40 percentage point improvements.
- Code generation. Complex algorithmic problems benefit significantly from planning and decomposition before writing code. Reasoning models score substantially higher on competition programming benchmarks (Codeforces, IOI-level problems).
- Scientific reasoning. GPQA Diamond (PhD-level science questions) scores improve from ~50% to ~70%+ with extended reasoning.
- Complex analysis. Tasks requiring weighing multiple factors, evaluating tradeoffs, or synthesizing information from different sources benefit from structured reasoning.
Tasks where CoT provides marginal or no benefit
- Simple factual recall. “What is the capital of France?” does not benefit from step-by-step reasoning.
- Text classification. Sentiment analysis, topic categorization, and similar pattern-matching tasks see minimal improvement from CoT and significant latency increases.
- Translation. Machine translation is primarily a pattern-matching task where CoT adds cost without quality improvement.
- Creative writing. Open-ended generation tasks do not have “correct” answers that reasoning can converge toward. CoT can even hurt creative tasks by making output more formulaic.
- Information extraction. Pulling structured data from text (named entities, dates, amounts) is usually better served by direct prompting with output format specification.
The inverse scaling problem
On some tasks, CoT actively hurts performance. When models reason through questions involving common misconceptions or cognitive biases, the reasoning process can reinforce errors rather than correct them. For example, on certain trick questions designed to exploit common logical fallacies, models with CoT may “reason” their way into the wrong answer more confidently than models giving direct responses. This phenomenon, documented in the Inverse Scaling Prize findings, is a reminder that reasoning is not magical — a model reasoning from flawed premises will reach flawed conclusions more verbosely.
Cost and latency implications
Chain-of-thought’s benefits come with concrete costs:
| Approach | Typical token multiplier | Latency impact | Cost per query (Sonnet-class) |
|---|---|---|---|
| Direct answer | 1x (baseline) | Baseline | $0.001-0.005 |
| Prompt-elicited CoT | 3-5x output tokens | 3-5x slower | $0.003-0.025 |
| Self-consistency (k=10) | 30-50x output tokens | 10x slower (parallelizable) | $0.03-0.25 |
| Hidden reasoning (standard budget) | 5-20x total tokens | 5-15x slower | $0.01-0.10 |
| Hidden reasoning (max budget) | 50-200x total tokens | 30-120x slower | $0.10-2.00 |
| Reasoning + search | 500-5,000x total tokens | Minutes to hours | $1-100+ |
These costs drive the model routing pattern: a lightweight classifier examines each incoming query and routes it to the appropriate reasoning level. Simple queries go directly to a fast model without CoT. Moderate queries use standard CoT or light reasoning. Complex queries get the full reasoning model treatment. This can reduce average costs by 60-80% compared to routing everything through a reasoning model.
Chain-of-thought in production systems
CoT is deeply embedded in production AI architectures:
Coding agents. Claude Code, Cursor, and similar tools use extensive chain-of-thought to plan code changes before writing them. When given a task like “add OAuth to this application,” the agent reasons through the codebase structure, identifies files to modify, plans the implementation approach, and only then begins writing code. This planning phase — which is essentially structured CoT — is critical to multi-file code changes that compile and pass tests.
AI tutoring. Khan Academy’s Khanmigo uses CoT to model step-by-step problem solving for students. Rather than giving answers, the tutor model reasons through each step of a math problem, providing hints and guided reasoning. The visible CoT is the product itself — students learn by following the model’s reasoning process.
Medical decision support. AI systems for clinical decision support use CoT to make their reasoning auditable. A system recommending a diagnosis must show the symptoms considered, differential diagnoses evaluated, and evidence weighed — step-by-step reasoning that physicians can verify. This transparency requirement makes CoT not just a performance technique but a regulatory necessity.
Legal analysis. AI tools for contract review and legal research use CoT to trace conclusions back to specific clauses and precedents. Harvey AI and Casetext use extended reasoning to analyze complex legal documents, with the reasoning chain serving as a verifiable audit trail.
Faithfulness: does the model reason honestly?
A critical open question is whether chain-of-thought reasoning is faithful — whether the stated reasoning actually reflects the model’s true computational process, or is a post-hoc rationalization.
Research suggests the answer is mixed. On mathematical tasks with verifiable intermediate steps, models’ CoT tends to be faithful — the stated steps genuinely contribute to the answer. But on tasks involving social reasoning, bias, or subjective judgment, models sometimes produce plausible-sounding reasoning that does not reflect their actual decision-making process. A model might arrive at an answer through pattern matching on training data, then generate a reasoning chain that rationalizes that answer.
This is not merely an academic concern. If CoT reasoning is used for high-stakes decisions (medical diagnosis, legal analysis, financial risk assessment), unfaithful reasoning could give false confidence in model outputs. Research into mechanistic interpretability and reasoning verification is ongoing, with techniques like attention analysis, activation patching, and self-consistency checks used to assess faithfulness.
Anthropic’s approach of providing a “thinking” block with Claude’s extended thinking is partly motivated by this concern — by structuring reasoning in a dedicated space with specific constraints, the model is encouraged to reason more genuinely rather than post-hoc rationalizing.
Frequently asked questions
What is the difference between chain-of-thought and a reasoning model? Chain-of-thought is a general technique where a model generates intermediate reasoning steps. A reasoning model is a specific type of model (like o1, o3, or Claude with extended thinking) that has been specifically trained through reinforcement learning to perform extended internal reasoning. All reasoning models use CoT, but not all CoT is produced by reasoning models — any LLM can perform basic CoT with the right prompt. Reasoning models produce longer, more reliable, and more structured reasoning, often in a hidden thinking block.
Does chain-of-thought always improve model performance? No. CoT significantly improves performance on tasks requiring multi-step logic, mathematics, coding, and complex analysis. It provides marginal benefit on simple factual recall, classification, and translation tasks. On some tasks involving common misconceptions, CoT can actually decrease accuracy by reinforcing flawed reasoning. The general guideline: use CoT when the task has multiple steps, requires combining information, or benefits from planning. Skip it for straightforward extraction and classification.
How much does chain-of-thought cost in extra tokens? Basic prompt-elicited CoT uses 3-5x more output tokens than a direct answer. Reasoning models with extended thinking can use 5-200x more total tokens depending on the thinking budget. Self-consistency sampling with 10 paths uses roughly 30-50x more tokens. The cost depends on the approach and the thinking budget allocated. Production systems typically use model routing to apply CoT selectively, only on queries that benefit from it.
Is the reasoning in chain-of-thought “real” thinking? This is actively debated. On mathematical and coding tasks, evidence suggests that CoT reasoning is largely faithful — the intermediate steps genuinely contribute to reaching the correct answer. On subjective or bias-prone tasks, models sometimes produce plausible reasoning that does not reflect their actual computation (post-hoc rationalization). Current research in mechanistic interpretability is working to better understand when and how CoT reasoning corresponds to the model’s internal processing.
What is “test-time compute scaling”? Test-time compute scaling is the principle that model performance can be improved by spending more computation during inference (generating more reasoning tokens) rather than only during training (making the model larger). This is the core innovation behind reasoning models: by generating thousands of thinking tokens, a smaller model can match or exceed the performance of a larger model on hard reasoning tasks. It represents a fundamental shift in how the AI field thinks about scaling — performance can be improved not just by training bigger models, but by thinking longer.