The MATH benchmark is the standard evaluation for mathematical reasoning in language models, testing whether AI systems can solve problems that challenge mathematically talented high school and undergraduate students. Created by Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt — published in 2021 — it draws 5,000 competition-level mathematics problems from the American Mathematics Competition (AMC), the American Invitational Mathematics Examination (AIME), and other competition sources. Problems span seven mathematical subjects and five difficulty levels, requiring genuine multi-step reasoning, symbolic manipulation, and mathematical insight that cannot be solved through pattern matching alone.
What MATH Measures
MATH evaluates a model’s ability to perform rigorous mathematical reasoning — the kind required in competition mathematics. Each problem demands that the model understand the mathematical question, identify the relevant concepts and techniques, execute a multi-step solution, and produce an exact final answer (numerical, algebraic, or symbolic). Unlike multiple-choice benchmarks where models can use process of elimination, MATH requires open-ended answers that must exactly match the ground truth.
The seven subject areas are: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. Each subject contains problems at five difficulty levels. Level 1 problems are straightforward applications of standard techniques, while Level 5 problems require creative insight, multi-concept integration, and extended chains of logical reasoning that challenge strong competitive math students.
The benchmark was created by the same UC Berkeley research group behind MMLU, motivated by the observation that existing math evaluations (like GSM8K) tested only basic arithmetic and word problem translation. MATH was designed to push into genuine mathematical reasoning — the kind that requires understanding abstract structures, applying theorems, and constructing proofs or derivations rather than merely computing.
Current Leaderboard Analysis
The MATH leaderboard reveals meaningful stratification across frontier models, with Gemini 2.5 Pro leading at 83.2%, followed by Claude Opus 4 at 81.4% — a 1.8-point gap. GPT-4.1 sits at 78.3%, establishing a clear third position with a wider gap below the top two. The spread from first to last (Llama 4 Maverick at 69.2%) is 14.0 percentage points — far more discriminative than saturated benchmarks.
Google DeepMind’s lead aligns with their publicly stated training emphasis on mathematical reasoning. The gap between Gemini 2.5 Pro and Claude Opus 4 is modest but consistent, suggesting a real advantage rather than statistical noise. More notable is the 4.9-point gap between Claude Opus 4 and GPT-4.1, indicating that mathematical reasoning capability does not simply track with overall model quality — models have meaningfully different strengths in this domain.
An interesting anomaly: GPT-4o from May 2024 scored 76.6%, which is higher than the current Grok-3 (73.5%) and Llama 4 Maverick (69.2%) despite being from an older generation. This suggests that some newer models have not prioritized mathematical reasoning to the same degree in their training, and that MATH scores reflect deliberate training choices as much as raw capability.
The 14-point spread confirms MATH remains a useful frontier differentiator in 2025, unlike MMLU or HumanEval where the top models are nearly indistinguishable.
Methodology Deep Dive
The benchmark contains exactly 5,000 problems in the test set, with a separate 7,500-problem training set that models may use for fine-tuning or few-shot examples. Each problem includes a step-by-step solution and a final answer in standardized format.
Evaluation uses exact match on the final answer. Answers can be integers, fractions, radicals, or other mathematical expressions, normalized to a canonical form before comparison. The model’s intermediate reasoning steps are not evaluated — only the final answer matters. This means a model that arrives at the correct answer through flawed reasoning receives full credit, while a model with correct reasoning but a computational error at the final step receives zero credit.
The five difficulty levels are calibrated against human performance in mathematics competitions. Level 1 problems are solvable by students with basic mathematical training. Level 5 problems represent the difficulty found in AIME and similar competitions, where even mathematically talented students may solve only a fraction correctly. Aggregate scores can mask large performance differences by level — a model scoring 80% overall might achieve 95% on Level 1-2 problems but only 45% on Level 5.
Chain-of-thought prompting is critical for MATH evaluation. Models that are prompted to show their reasoning step-by-step before giving a final answer consistently score 10-20 percentage points higher than models that are asked to directly output an answer. Most published scores use chain-of-thought prompting, and scores without it should not be compared to scores with it.
The seven subjects are not equally represented. Number Theory and Geometry tend to be the hardest categories for language models, while Prealgebra and basic Algebra are the easiest. Subject-level breakdowns provide significantly more insight than the aggregate score but are less commonly reported.
Why This Benchmark Matters
MATH matters because mathematical reasoning is one of the purest tests of logical capability in language models. Unlike knowledge benchmarks (where memorization can inflate scores) or language understanding tests (where pattern matching often suffices), mathematics requires building logical chains where each step must follow rigorously from the previous one. An error in any step typically produces a wrong answer, making the benchmark resistant to approximation strategies.
The benchmark also tracks closely with practical utility for quantitative applications. Models that score well on MATH tend to perform better at financial analysis, scientific computation, engineering calculations, and data analysis — tasks where precise numerical reasoning is essential. The correlation is not perfect (MATH problems are more abstract than most real-world calculations), but it captures the underlying reasoning capability that these applications demand.
For AI researchers studying reasoning, MATH provides granular insight through its difficulty levels. The gap between a model’s Level 1 and Level 5 performance reveals how well its reasoning holds up as complexity increases — a model that scores 95% on Level 1 but 30% on Level 5 has fundamentally different reasoning characteristics than one scoring 75% on both.
Known Limitations and Criticisms
Data contamination is a significant concern. MATH problems are sourced from published competition mathematics, meaning many of the exact problems (and their solutions) appear on websites like Art of Problem Solving, mathematics forums, and educational resources that are included in web-scraped training data. Research has found that some models produce solutions matching published solutions word-for-word, strongly suggesting memorization of specific problems.
The exact-match evaluation can be both too strict and too lenient. It is too strict when the model produces a mathematically equivalent answer in a different form (e.g., 0.5 instead of 1/2, or an unsimplified expression). It is too lenient in that it gives full credit for correct answers regardless of solution quality — a model that guesses correctly or arrives at the right answer through flawed reasoning scores the same as one with a rigorous derivation.
The competition mathematics format may not capture the mathematical reasoning most relevant to practical applications. Competition problems reward clever tricks and pattern recognition within specific problem types, while real-world mathematical reasoning often requires formulating problems, choosing appropriate methods, and interpreting results — skills MATH does not test.
The difficulty distribution also skews the aggregate score. With roughly equal numbers of problems at each level, a model can achieve a respectable overall score by mastering the easier levels while performing poorly on genuinely hard problems. Level-stratified scores are more informative but less commonly reported.
The benchmark covers only pre-university and early undergraduate mathematics. It does not test advanced topics like abstract algebra, real analysis, topology, or differential equations, which limits its predictive value for applications requiring graduate-level mathematical sophistication.
How Scores Have Changed Over Time
| Year | Top Model | Score | Key Insight |
|---|---|---|---|
| 2021 | GPT-3 (few-shot) | 10.2% | Math was nearly impossible for early LLMs |
| 2022 | Minerva (540B) | 33.6% | Math-specialized training showed promise |
| 2023 | GPT-4 | 42.5% | General capability approached math-specialized models |
| 2023 | GPT-4 (with CoT) | 52.9% | Chain-of-thought reasoning was transformative |
| 2024 | GPT-4o | 76.6% | Major leap through improved reasoning architecture |
| 2025 | Gemini 2.5 Pro | 83.2% | Continued gains; Level 5 problems remain challenging |
The progression from 10.2% to 83.2% in four years is remarkable, but the rate of improvement has decelerated. The jump from 10% to 50% happened in roughly two years, while the climb from 50% to 83% took another two. Level 5 problems remain a major challenge — even the best models solve them at rates well below their aggregate scores, suggesting that the hardest mathematical reasoning is fundamentally more difficult to scale.
MATH vs Other Benchmarks
| Benchmark | Focus | Problem Count | Format | Current Top Score |
|---|---|---|---|---|
| MATH | Competition mathematics | 5,000 | Open-ended, exact match | 83.2% |
| GSM8K | Grade-school arithmetic | 1,319 | Word problems | ~95% (saturated) |
| AIME 2024 | Competition math (fresh) | 30 | Open-ended | ~85% |
| GPQA Diamond | PhD-level science | 198 | Multiple choice | 74.9% |
| Minerva MATH | Same benchmark | 5,000 | Same format | Same scores |
MATH occupies the middle ground between basic arithmetic (GSM8K, which is fully saturated) and PhD-level scientific reasoning (GPQA Diamond, which includes math-adjacent reasoning). For evaluating mathematical reasoning specifically, MATH remains the primary benchmark, though fresh competition problems (like each year’s AIME) provide contamination-free data points that serve as valuable cross-checks.
GSM8K, which tests grade-school math word problems, is now trivially solved by all frontier models (above 95%). Users sometimes confuse MATH with GSM8K — the former is orders of magnitude harder and remains a meaningful differentiator.
Practical Implications
For users who need a model for quantitative work — data analysis, financial modeling, engineering calculations, scientific computing — MATH scores provide the most relevant signal. The difference between a model scoring 69% and 83% is practically significant: the higher-scoring model will handle multi-step calculations more reliably, catch its own errors during reasoning chains, and produce correct results on problems requiring several mathematical concepts.
For education applications, MATH scores predict how well a model can serve as a math tutor or homework assistant. Models scoring above 80% can reliably solve the vast majority of undergraduate-level mathematics problems, making them useful study aids. However, they may still struggle with problems requiring creative mathematical insight (Level 5), and their solutions should be verified for correctness rather than trusted implicitly.
For developers building math-heavy applications, the gap between models is large enough to affect user experience. A model scoring 83% on MATH will make noticeably fewer mathematical errors than one scoring 73%, and these errors compound in applications that chain multiple calculations together.
The competition-math format means MATH scores may overstate capability for applied mathematics (statistics, numerical methods, differential equations) and understate it for abstract mathematics (proof construction, set theory). Users working in specific mathematical domains should supplement MATH scores with domain-relevant evaluations.
Frequently Asked Questions
Why is MATH harder than GSM8K?
GSM8K tests basic arithmetic word problems at a grade-school level — the kind that require translating a short story into a simple equation and computing the answer. MATH tests competition-level problems that require understanding abstract mathematical concepts, applying theorems, performing symbolic manipulation, and constructing multi-step derivations. A model that scores 95% on GSM8K may score below 70% on MATH because the latter demands fundamentally different reasoning capabilities.
How much does chain-of-thought prompting matter for MATH?
Enormously. Scores improve by 10-20 percentage points when models are allowed to reason step-by-step before giving a final answer, compared to direct-answer prompting. This is because MATH problems genuinely require multi-step reasoning — there is rarely a shortcut to the answer. Models that can articulate and follow reasoning chains solve problems that they would otherwise get wrong. Most published MATH scores use chain-of-thought prompting.
Are MATH scores affected by data contamination?
Yes, and this is a well-documented concern. MATH problems come from published mathematics competitions, and their solutions are available on numerous websites. Research has shown evidence of memorization in several models. This means published MATH scores may overstate genuine mathematical reasoning ability. Fresh competition problems from recent AIME or AMC exams provide cleaner measurement, though sample sizes are small.
What do the difficulty levels mean practically?
Level 1-2 problems correspond roughly to high school algebra and precalculus — well within the capability of all frontier models. Level 3-4 problems are at the AMC 10/12 difficulty, requiring solid mathematical knowledge and some problem-solving skill. Level 5 problems are AIME-level and above, demanding creative mathematical insight. Even the best models score below 50% on Level 5, making it the tier where genuine mathematical reasoning capability is most visible.
Why does Gemini 2.5 Pro lead on MATH despite trailing on other benchmarks?
Gemini 2.5 Pro’s leadership on MATH reflects Google DeepMind’s significant investment in mathematical reasoning during model training and their implementation of extended thinking capabilities that allow the model to allocate more compute to complex problems. Mathematical reasoning is a distinct capability that can be developed somewhat independently of other skills, and Google’s emphasis on scientific and mathematical capabilities is a deliberate strategic choice that manifests in MATH performance even when other benchmarks favor competing models.