MMMU (Massive Multi-discipline Multimodal Understanding) is the definitive benchmark for evaluating whether vision-language models can understand and reason about visual information at the level a college student would need — not just describe images, but interpret circuit diagrams, analyze medical scans, read architectural blueprints, and solve problems that require integrating visual perception with domain-specific knowledge. Created by Xiang Yue, Yuansheng Ni, Kai Zhang, and colleagues from multiple universities in 2023, MMMU draws its 11,500 questions from real college-level exams across 30 subjects spanning art, business, science, health, humanities, and engineering. Each question includes at least one image that is essential to answering it, making MMMU the most comprehensive test of multimodal academic reasoning available.
What MMMU Measures
MMMU evaluates the intersection of visual understanding and domain knowledge — a capability that text-only benchmarks cannot assess. Each question presents one or more images alongside a text prompt, and the model must jointly reason over both modalities to arrive at the correct answer. The images are not decorative; they carry essential information. A physics question might show a free-body diagram that the model must interpret to set up equations. A biology question might display a gel electrophoresis result that requires understanding band patterns. An art history question might show a painting that must be attributed to a specific movement or artist.
The 30 subjects are organized into six macro-categories: Art and Design (art history, design, music), Business (accounting, economics, finance, marketing, management), Science (biology, chemistry, computer science, math, physics), Health and Medicine (basic medical science, clinical medicine, diagnostics, pharmacy, public health), Humanities and Social Science (geography, history, literature, psychology, sociology), and Engineering (civil, electrical, materials, mechanical engineering). This breadth ensures the benchmark tests multimodal reasoning across diverse domains rather than rewarding narrow visual specialization.
The questions are sourced from real college-level textbooks, course materials, and examinations, ensuring they represent the kind of multimodal reasoning that students and professionals actually perform. Difficulty corresponds roughly to undergraduate coursework, placing MMMU between basic visual question answering (which tests object recognition and simple visual reasoning) and PhD-level scientific benchmarks like GPQA Diamond.
Current Leaderboard Analysis
The MMMU leaderboard shows Gemini 2.5 Pro leading at 72.7%, followed by Claude Opus 4 at 70.4% and GPT-4.1 at 69.1%. The top three are separated by just 3.6 percentage points, but this gap is meaningful given MMMU’s demonstrated ability to differentiate models. Below the leaders, a significant drop occurs: Claude Sonnet 4 (64.8%), GPT-4o (63.1%), Grok-3 (61.3%), and Llama 4 Maverick (58.7%) form a clearly separated second tier.
Google DeepMind’s consistent leadership on MMMU reflects their sustained investment in natively multimodal architectures. The Gemini family has been designed from the ground up to process images and text together, rather than using separate vision encoders bolted onto a language model. This architectural advantage is most visible in subjects requiring fine-grained visual interpretation — engineering diagrams, scientific figures, and medical imaging — where the quality of visual feature extraction directly determines reasoning quality.
The 14-point spread from top (72.7%) to bottom (58.7%) confirms MMMU’s discriminative power. With expert human performance averaging approximately 88%, significant headroom remains. The benchmark will continue to differentiate frontier models for the foreseeable future, unlike saturated evaluations such as HellaSwag or standard HumanEval.
The gap between Claude Opus 4 (70.4%) and Claude Sonnet 4 (64.8%) — a 5.6-point difference within the same model family — illustrates that multimodal reasoning scales with model capability. This contrasts with benchmarks like HellaSwag where the gap between the same models is under 2 points.
Methodology Deep Dive
MMMU contains 11,500 questions, with a validation set of 900 questions and a test set of 10,500 questions. The test set labels are withheld — submissions must be evaluated through the official benchmark server to prevent data contamination. Each question includes one or more images (diagrams, charts, photographs, maps, chemical structures, medical images, artwork, or technical figures) and is presented in multiple-choice format.
Questions are categorized by subject, sub-subject, and image type. The diversity of image types is a distinguishing feature: models must handle natural photographs, hand-drawn diagrams, computer-generated charts, microscopy images, architectural floor plans, circuit schematics, musical scores, and more. This variety prevents models from overfitting to any single type of visual input.
Scoring is accuracy on the multiple-choice questions. Most questions have four options, though some have more. The benchmark reports both overall accuracy and per-subject accuracy, enabling analysis of a model’s strengths and weaknesses across domains. Subject-level breakdowns are often more informative than the aggregate score — a model might excel at chart reading (business questions) while struggling with spatial reasoning (engineering questions).
Evaluation is typically performed zero-shot: the model receives only the question and its associated images, without few-shot examples. This tests the model’s inherent ability to interpret visual information and apply domain knowledge, without the benefit of in-context learning from examples.
Why This Benchmark Matters
MMMU matters because it tests a capability that is essential for AI to be useful in professional and educational contexts: the ability to understand documents, diagrams, and visual data as a human expert would. The vast majority of human knowledge is communicated through combinations of text and images — textbooks, research papers, technical manuals, medical records, financial reports, and engineering drawings all require multimodal understanding.
Before MMMU, multimodal benchmarks primarily tested basic capabilities like image captioning, visual question answering about photographs, and optical character recognition. These evaluations failed to capture whether models could reason about visual information in the way that college-level and professional work demands. A model that can describe a photograph accurately may still fail to interpret a phase diagram, read a musical score, or extract quantitative data from a complex chart.
MMMU fills this gap by testing visual reasoning in the context of genuine academic tasks. Its scores predict how well a model can assist with document analysis, data interpretation, technical problem-solving, and educational tutoring — applications where multimodal understanding is not optional but essential.
Known Limitations and Criticisms
MMMU’s multiple-choice format means it tests recognition rather than generation. A model that can identify the correct interpretation of a diagram from four options may not be able to describe or analyze the same diagram in an open-ended setting. Real-world multimodal tasks often require generating explanations, annotations, or analyses rather than selecting from predefined choices.
The benchmark’s image quality and resolution vary significantly across questions, reflecting the diversity of source materials. Some models may underperform on questions with low-resolution or complex images not because they lack reasoning ability but because their vision encoders fail to extract sufficient detail. This confounds the measurement of visual reasoning with the measurement of visual acuity.
Cultural and linguistic bias is present in the question set. Questions are drawn primarily from English-language educational materials, and some subjects (particularly humanities and social sciences) contain culturally specific content that may disadvantage models trained primarily on non-Western data.
The static nature of the benchmark means that as models improve, the same set of questions provides less discriminative power. Unlike SWE-bench (which draws from an ever-growing pool of GitHub issues) or Chatbot Arena (which continuously collects new human preferences), MMMU’s question set is fixed. An MMMU-v2 or continuously updated variant would provide more durable evaluation signal.
Data contamination is partially mitigated by the withheld test set, but the validation set (900 questions) is public and may appear in training data. Models that report validation-set scores may benefit from contamination, while test-set scores submitted through the official server are more reliable.
How Scores Have Changed Over Time
| Year | Top Model | Score | Key Insight |
|---|---|---|---|
| 2023 | GPT-4V (launch) | 56.8% | First multimodal model to show meaningful visual reasoning |
| 2024 | Gemini 1.5 Pro | 62.2% | Natively multimodal architecture showed advantage |
| 2024 | GPT-4o | 63.1% | Incremental improvement, vision still a weakness |
| 2024 | Claude 3.5 Sonnet | 65.4% | Vision capabilities closing the gap |
| 2025 | Gemini 2.5 Pro | 72.7% | Significant jump, architecture advantage widening |
The trajectory shows steady improvement, with the most significant gains coming from models that process images and text through unified architectures rather than separate pipelines. The jump from 56.8% to 72.7% over two years represents meaningful progress, but the 15+ point gap to human expert performance (88%) indicates that visual reasoning remains a harder problem for AI than text-only knowledge tasks.
MMMU vs Other Benchmarks
| Benchmark | Modality | Scope | Difficulty | Status |
|---|---|---|---|---|
| MMMU | Text + Images | 30 academic subjects | College-level | Active differentiation |
| VQAv2 | Text + Images | Natural image QA | Easy | Saturated |
| MathVista | Text + Images | Mathematical diagrams | Medium-Hard | Active |
| DocVQA | Text + Images | Document understanding | Medium | Near-saturated |
| MMLU | Text only | 57 academic subjects | Medium | Saturated |
MMMU is the multimodal counterpart to MMLU — both test broad academic knowledge, but MMMU adds the critical dimension of visual understanding. A model that scores well on MMLU but poorly on MMMU has strong text-based knowledge but weak visual reasoning. The reverse pattern (high MMMU, low MMLU) is uncommon because MMMU questions require both visual and textual knowledge.
MathVista is a complementary benchmark focusing specifically on mathematical reasoning with visual inputs (charts, plots, geometry diagrams). Models that excel on MMMU’s science and engineering subjects tend to also score well on MathVista, as both require quantitative visual reasoning.
Practical Implications
For users who work with visual documents — researchers reading papers with figures, engineers reviewing technical drawings, medical professionals interpreting diagnostic images, students studying from textbooks — MMMU scores are the most relevant benchmark. A model scoring above 70% can be expected to provide useful analysis of college-level visual content across a range of disciplines, though expert verification remains important for high-stakes applications.
The 72.7% top score indicates that current models handle many common multimodal tasks adequately — reading charts, interpreting basic diagrams, understanding annotated photographs — but struggle with tasks requiring precise spatial reasoning, fine-grained visual discrimination, or integration of complex visual information with specialized domain knowledge.
For enterprise applications involving document processing, data extraction from visual reports, or automated analysis of technical diagrams, MMMU performance predicts real-world utility better than text-only benchmarks. Organizations processing large volumes of visual documents (insurance claims with photographs, medical records with imaging, engineering inspection reports) should weight MMMU scores heavily in model selection.
The gap between the top models (70-73%) and expert humans (88%) also signals that fully autonomous visual document analysis is not yet reliable for critical applications. AI-assisted workflows, where models perform initial analysis that humans review and correct, remain the appropriate deployment pattern for multimodal tasks.
Frequently Asked Questions
What types of images does MMMU include?
MMMU includes a wide variety of image types: scientific diagrams (molecular structures, free-body diagrams, circuit schematics), charts and graphs (bar charts, scatter plots, phase diagrams), medical images (X-rays, histology slides, ECG tracings), artwork and photographs (paintings, architectural photographs), maps and floor plans, musical scores, engineering drawings, and more. This diversity is intentional — it tests whether models can handle the full range of visual information encountered in academic and professional contexts.
Why does Gemini consistently lead on MMMU?
Gemini’s advantage on MMMU reflects Google DeepMind’s architectural choice to build natively multimodal models that process images and text through a unified system, rather than connecting separate vision and language components. This unified approach provides better integration between visual features and language reasoning, which is particularly advantageous for tasks requiring fine-grained visual interpretation combined with domain knowledge.
How does MMMU differ from simpler vision benchmarks?
Simpler benchmarks like VQAv2 test basic visual question answering (identifying objects, counting, describing scenes). MMMU tests whether models can reason about visual information — interpreting a circuit diagram to solve for voltage, reading a gel electrophoresis result to determine protein sizes, or analyzing an architectural drawing to identify structural issues. The questions require domain knowledge that cannot be inferred from the image alone.
Is MMMU useful for evaluating text-only models?
No. MMMU requires processing images, so text-only models cannot be evaluated on it. For text-only academic knowledge evaluation, MMLU is the appropriate benchmark. MMMU’s value is specifically in assessing the visual reasoning capabilities that text-only benchmarks cannot measure.
What score would a model need to be practically useful for document analysis?
Models scoring above 65% on MMMU can handle straightforward multimodal tasks — reading clearly presented charts, interpreting standard diagrams, extracting information from well-formatted visual documents. For complex tasks requiring fine-grained visual analysis or specialized domain knowledge (medical imaging, engineering schematics), scores above 75% are likely needed for reliable autonomous use, though human verification remains recommended regardless of score.