MMLU (Massive Multitask Language Understanding) is the most widely cited benchmark for measuring general knowledge and reasoning in large language models. Created by Dan Hendrycks and colleagues at UC Berkeley and published in 2021, MMLU tests models across 57 academic subjects — from abstract algebra and anatomy to world religions and virology — using multiple-choice questions that range from elementary to professional difficulty. It has become the default shorthand that researchers, journalists, and engineers use when comparing how much a model “knows.”
What MMLU Measures
MMLU evaluates a model’s breadth and depth of factual knowledge across four broad academic domains: STEM (mathematics, physics, computer science, engineering), humanities (history, philosophy, law), social sciences (psychology, economics, political science), and other professional fields (medicine, accounting, business). Each of the 57 subjects contains between 100 and several hundred multiple-choice questions, with four answer options per question.
The benchmark was designed by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt at UC Berkeley. Their stated goal was to create a single evaluation that could capture the kind of broad, multi-domain knowledge that distinguishes a well-educated human from a narrow specialist. The questions are sourced from practice exams, textbooks, and standardized tests at levels ranging from high school to graduate professional examinations.
MMLU matters because it provides a standardized, reproducible measure of general intelligence that can be compared across models, labs, and time. Before MMLU, comparing model knowledge required assembling ad hoc collections of subject-specific tests. MMLU unified these into a single number that, while imperfect, gives a quick read on how well a model handles knowledge-intensive tasks.
Current Leaderboard Analysis
The current MMLU leaderboard tells a story of convergence at the top. GPT-4.1 leads with 90.2%, followed closely by Gemini 2.5 Pro at 89.8% and Claude Opus 4 at 89.1%. The gap between first and third place is just 1.1 percentage points — a margin that is difficult to interpret as a meaningful capability difference given the benchmark’s inherent noise at these performance levels.
The more informative divide is between the top three and the rest. Claude Sonnet 4 at 86.5%, Llama 4 Maverick at 85.2%, and Grok-3 at 84.7% form a clear second tier, roughly 4-5 points behind the leaders. DeepSeek V3 anchors the tracked models at 83.1%. This tiering suggests that the top three models have each crossed a capability threshold in knowledge encoding that the second tier has not quite reached, though even the lowest-scoring model here would have been state-of-the-art as recently as early 2024.
GPT-4o’s 87.2% from May 2024 is worth noting — it slots between the current frontier and second tiers, illustrating how quickly the field moves. A score that was near the top of the leaderboard a year ago is now solidly mid-pack.
Methodology Deep Dive
MMLU uses a 5-shot evaluation format: each test question is preceded by five example question-answer pairs from the same subject, giving the model a brief “study session” before answering. This few-shot approach reduces variance from formatting misunderstandings and focuses evaluation on actual knowledge rather than prompt sensitivity.
The benchmark contains approximately 15,900 questions total across the 57 subjects. Questions are presented in standard multiple-choice format with four options (A, B, C, D). Scoring is straightforward accuracy — the percentage of questions answered correctly. There is no partial credit, no penalty for guessing, and no weighting by difficulty.
Difficulty varies enormously by subject. Elementary mathematics questions may test basic arithmetic, while clinical knowledge questions draw from medical board exams. This heterogeneity is both a strength (it captures breadth) and a weakness (aggregate scores can mask wildly different performance across domains). A model scoring 85% overall might achieve 95% in history and 60% in advanced mathematics, but the headline number obscures this.
The test set is fixed and publicly available, which is important for reproducibility but creates data contamination risks. Most labs evaluate using the standard 5-shot protocol, though some report 0-shot results that are typically 2-5 points lower.
Why This Benchmark Matters
MMLU matters because it was the first benchmark to convincingly demonstrate that language models could acquire broad, cross-domain knowledge at expert levels. When GPT-3 scored 43.9% in 2020, the result was noteworthy but unimpressive — barely above the 25% random baseline for four-option multiple choice. By 2023, GPT-4’s 86.4% showed that language models had crossed from “somewhat knowledgeable” to “comparable to educated humans” across most academic fields.
This trajectory reshaped how AI researchers, policymakers, and the public understood the capabilities of large language models. MMLU scores became the standard evidence cited in discussions about AI replacing knowledge workers, the need for AI safety research, and the economic impact of foundation models. Whether or not MMLU perfectly captures “understanding,” its scores track closely enough with real-world utility that they remain a meaningful signal.
Known Limitations and Criticisms
MMLU’s most serious limitation in 2025 is saturation. With top models scoring above 89%, the benchmark no longer provides meaningful separation between frontier systems. The difference between 89.1% and 90.2% is within the range of variance introduced by prompt formatting, sampling temperature, and minor differences in evaluation setup.
Data contamination is a persistent concern. Because MMLU questions are publicly available, they may appear in training data — either directly or through paraphrased versions in web text that references the benchmark. Labs generally claim to decontaminate their training sets, but verification is difficult, and even indirect exposure to similar questions could inflate scores.
The multiple-choice format introduces its own artifacts. Models can sometimes identify correct answers through process-of-elimination heuristics or by detecting statistical patterns in answer distributions rather than through genuine knowledge. Research has shown that some models perform better when answer choices are reordered, suggesting they are not always reasoning from first principles.
Subject-level granularity is often lost in reporting. The aggregate MMLU score treats a question about Renaissance art as equivalent to a question about organic chemistry. Models with different training data distributions may score similarly overall while having dramatically different capability profiles across subjects.
MMLU-Pro, introduced in 2024, attempts to address saturation by adding harder questions with ten answer options instead of four, reducing the benefit of guessing and requiring more precise knowledge. Several labs have begun reporting MMLU-Pro scores alongside standard MMLU.
How Scores Have Changed Over Time
| Year | Top Model | Score | Key Insight |
|---|---|---|---|
| 2020 | GPT-3 (175B) | 43.9% | Barely above 25% random chance on many subjects |
| 2022 | Chinchilla (70B) | 67.6% | Scaling laws research pushed efficiency gains |
| 2023 | GPT-4 | 86.4% | First model to approach expert-level breadth |
| 2023 | Gemini Ultra | 90.0% | First model to break 90%, sparking saturation debate |
| 2024 | GPT-4o | 87.2% | Strong but incremental improvement over GPT-4 |
| 2025 | GPT-4.1 | 90.2% | Top score, but only 0.2 points above 2023’s Gemini Ultra |
The most striking feature of this progression is the deceleration. The jump from 43.9% to 67.6% (23.7 points) took two years. The jump from 67.6% to 86.4% (18.8 points) took roughly one year. Since then, gains have slowed to single-digit increments — reflecting genuine saturation rather than a lack of progress.
MMLU vs Other Benchmarks
| Benchmark | Focus | Question Count | Format | Saturation Status |
|---|---|---|---|---|
| MMLU | Broad knowledge, 57 subjects | ~15,900 | 4-option MC, 5-shot | Saturated (top >90%) |
| MMLU-Pro | Harder knowledge, 10 options | ~12,000 | 10-option MC | Active differentiation |
| GPQA Diamond | PhD-level science | 198 | 4-option MC | Still differentiating |
| ARC-Challenge | Grade-school science | 2,590 | 4-option MC | Saturated |
| TriviaQA | Factual recall | 95,000 | Open-ended | Mostly saturated |
MMLU occupies a middle ground — harder than grade-school benchmarks like ARC but easier than PhD-level tests like GPQA Diamond. For evaluating frontier models in 2025, MMLU is most useful as a sanity check (a top model should score above 85%) rather than a differentiator. MMLU-Pro and GPQA Diamond are where the meaningful separation happens.
MMLU complements reasoning-focused benchmarks like MATH and coding benchmarks like HumanEval by measuring a different axis entirely. A model can score highly on MATH through strong reasoning with limited factual knowledge, but MMLU requires breadth across dozens of unrelated fields.
Practical Implications
For users choosing between frontier models, MMLU scores above 88% are effectively equivalent. The difference between GPT-4.1 at 90.2% and Claude Opus 4 at 89.1% will not be perceptible in any real-world knowledge task. Users should instead compare models on the specific domains that matter to them — a medical professional should look at clinical knowledge sub-scores, while a software engineer should prioritize coding benchmarks.
The more actionable signal in MMLU comes from the second tier. If a model scores below 85%, users may encounter noticeable gaps in specialized knowledge — incorrect legal citations, confused medical terminology, or unreliable historical facts. The jump from 83% to 87% represents a meaningful improvement in the reliability of knowledge-intensive responses.
For enterprises evaluating models, MMLU’s subject-level breakdown is more valuable than the aggregate score. A model that scores 92% in computer science but 78% in law has a very different value proposition than one with a flat 85% across both. Most labs publish or can provide subject-level results on request.
Frequently Asked Questions
What does a 90% MMLU score actually mean?
A 90% score means the model correctly answers 9 out of 10 questions across all 57 subjects. However, performance varies by subject — the model might achieve 98% on US history and 75% on abstract algebra. The aggregate score masks significant subject-level variation, so evaluating models on specific domains of interest is more informative than comparing headline numbers.
Is MMLU still relevant in 2025?
MMLU remains relevant as a baseline check and for evaluating mid-tier and open-weight models where meaningful score differences still exist. For comparing frontier models from OpenAI, Anthropic, and Google DeepMind, MMLU is largely saturated and should be supplemented with harder benchmarks like MMLU-Pro, GPQA Diamond, or domain-specific evaluations.
How does MMLU-Pro differ from MMLU?
MMLU-Pro increases difficulty by offering ten answer choices instead of four (reducing the benefit of random guessing from 25% to 10%), including more questions that require multi-step reasoning, and sourcing harder questions that better separate frontier models. Top models that score 89-90% on standard MMLU typically score 10-15 points lower on MMLU-Pro.
Can models cheat on MMLU through data contamination?
Because MMLU questions are publicly available, there is a risk that models have seen them — or close paraphrases — during training. Labs implement decontamination procedures, but these are imperfect. Independent research has found evidence of partial contamination in several models. This is one reason why harder, newer benchmarks with private test sets provide more reliable comparisons.
Why do different sources report different MMLU scores for the same model?
Variations arise from differences in evaluation protocol: the number of few-shot examples (0-shot vs 5-shot), the exact prompt template, whether chain-of-thought reasoning is allowed, and how answer parsing handles ambiguous outputs. A model might score 87% with one evaluation harness and 89% with another. When comparing scores, verify that the same methodology was used.