MT-Bench is an automated evaluation of multi-turn conversational quality, introduced by Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, and the LMSYS team at UC Berkeley in 2023 alongside Chatbot Arena. It uses 80 carefully designed two-turn questions across eight categories — writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities — with GPT-4 serving as an automated judge that rates each response on a 1-10 scale. MT-Bench was created to provide a reproducible, fast, and inexpensive complement to the crowdsourced Arena, giving researchers and developers a way to evaluate conversational model quality in minutes rather than months.
What MT-Bench Measures
MT-Bench measures a model’s ability to carry on a coherent, helpful multi-turn conversation across diverse task types. Each of the 80 questions consists of two turns. The first turn presents a task — write an essay, solve a math problem, explain a scientific concept, roleplay a scenario, extract information from text, or write code. The second turn deliberately builds on the first, asking the model to refine its answer, correct a mistake, extend the analysis, change the format, or address a follow-up question that tests whether it truly understood the original task.
This two-turn structure is the benchmark’s distinguishing feature. Many evaluations test single-turn performance, where the model responds to an isolated prompt. MT-Bench tests whether models can maintain context, adapt to user feedback, and demonstrate coherent reasoning across multiple exchanges — which is how people actually interact with AI assistants. A model that produces excellent first-turn responses but fails to follow up coherently will score lower than one with consistent quality across both turns.
The eight categories provide balanced coverage of both creative capabilities (writing, roleplay) and analytical ones (math, coding, reasoning, STEM, extraction, humanities). Each category contains 10 questions, producing 80 total scored interactions. An LLM judge (GPT-4) rates each turn on a 1-10 scale evaluating helpfulness, relevance, accuracy, depth, creativity, and detail. The final MT-Bench score is the average across all 160 individual turn ratings.
Current Leaderboard Analysis
MT-Bench is approaching saturation at the frontier, with the top three models separated by just 0.07 points on a 10-point scale. Claude Opus 4 leads at 9.52, Gemini 2.5 Pro follows at 9.48, and GPT-4.1 sits at 9.45. These differences are within the margin of noise for a benchmark with only 80 questions and a subjective LLM judge — practically, these three models are equivalent on MT-Bench.
The more meaningful separation occurs below the top tier. Grok-3 (9.31), GPT-4o (9.32), and Claude Sonnet 4 (9.28) cluster around 9.3, roughly 0.2 points behind the leaders. Llama 4 Maverick (9.05) and DeepSeek V3 (8.91) form a third tier. The total spread from top to bottom is 0.61 points — small enough to confirm the benchmark is nearing its useful ceiling for frontier comparison.
GPT-4o’s 9.32 from May 2024 is worth noting — it matches or exceeds several newer models (Grok-3, Claude Sonnet 4), suggesting that conversational quality improvements have slowed even as specialized capabilities like coding and scientific reasoning have advanced significantly.
Category-level analysis reveals where differences persist. In math and coding categories, score variance between models is higher than in writing, roleplay, or humanities. The top models score 9.7-9.8 on writing and roleplay but drop to 8.8-9.3 on math and coding, where precise multi-step reasoning under conversational constraints separates models more effectively.
Methodology Deep Dive
MT-Bench uses 80 two-turn prompts, 10 per category across writing, roleplay, extraction, reasoning, math, coding, STEM, and humanities. The questions were designed by the LMSYS research team to cover the range of tasks users bring to conversational AI assistants.
The automated judge is GPT-4, which receives each model’s response along with the original prompt and a scoring rubric. The judge rates each turn independently on a 1-10 scale. Some implementations also report per-category scores, which are often more informative than the aggregate. The total benchmark score is the mean of all 160 turn ratings.
Research by the LMSYS team has shown that GPT-4 judge ratings correlate at approximately 80% with human expert judgments — strong but not perfect. Known biases include: preference for longer, more detailed responses; slight favoritism toward responses matching OpenAI’s stylistic patterns; and positional bias when used in pairwise comparisons.
The 80-question size is deliberately small for fast iteration, but it means score variance is relatively high. Running the same model twice with different random seeds can produce scores differing by 0.1-0.2 points. Score differences smaller than approximately 0.15 points should not be treated as statistically meaningful. A single MT-Bench evaluation costs less than $10 in API calls and completes in minutes, making it practical for continuous quality testing during model development.
Why This Benchmark Matters
MT-Bench matters because it provides a fast, reproducible measure of conversational quality that approximates human judgment at a fraction of the cost. While Chatbot Arena requires millions of user votes over weeks or months to produce stable rankings for a new model, MT-Bench can evaluate any model in minutes. This makes it the standard benchmark for rapid model iteration, ablation studies, and quality regression testing during development.
The multi-turn format captures something most benchmarks miss: conversational coherence. A model can score well on single-turn evaluations (MMLU, HumanEval, MATH) while struggling to maintain coherent dialogue across turns — losing context, contradicting earlier statements, or failing to adapt to user feedback. MT-Bench’s two-turn structure explicitly tests these capabilities, which are essential for chatbot and assistant applications.
The strong correlation between MT-Bench and Chatbot Arena Elo (approximately r=0.9) validates MT-Bench as a reliable proxy for human preference. Labs typically use MT-Bench for rapid screening during development, with Arena serving as the final validation of conversational quality.
Known Limitations and Criticisms
The GPT-4 judge introduces systematic bias. Because the judge is itself a language model from OpenAI, it may favor response patterns similar to its own training. This creates a potential advantage for OpenAI models and a disadvantage for models with different conversational styles, even if those styles are equally valued by human users. Multiple research groups have documented this bias.
The 80-question sample is too small for high-confidence ranking at the frontier. With top models separated by tenths of a point, the benchmark lacks the statistical power to produce reliable orderings. Category-level scores (based on just 10 questions each) are even noisier. MT-Bench is better treated as a tier classifier than a precise ranking tool.
The two-turn format, while better than single-turn evaluation, does not capture the full complexity of real conversations. Actual assistant interactions often involve 5-20 turns with topic changes, ambiguous instructions, and evolving requirements. MT-Bench tests only the most basic form of multi-turn coherence.
Verbosity bias is well-documented. The GPT-4 judge tends to award higher scores to longer, more detailed responses, even when shorter responses would be more appropriate. This incentivizes unnecessary verbosity that may not align with user preferences.
The fixed question set creates overfitting risk. The 80 questions have been publicly available since 2023, and models may have encountered them during training. Labs could theoretically optimize specifically for MT-Bench without improving general conversational quality.
How Scores Have Changed Over Time
| Year | Top Model | Score | Key Insight |
|---|---|---|---|
| 2023 | GPT-4 (launch) | 8.99 | Set the initial bar for conversational quality |
| 2023 | Claude 2 | 8.06 | Competitive but clear gap to GPT-4 |
| 2024 | Claude 3 Opus | 9.00 | Multiple models crossed the 9.0 threshold |
| 2024 | GPT-4o | 9.32 | Steady improvement, convergence beginning |
| 2025 | Claude Opus 4 | 9.52 | Top models clustered within 0.1 points |
The trajectory shows rapid convergence. In early 2023, GPT-4 (8.99) had a clear lead over competitors. By 2025, five models score above 9.2 and three above 9.4. The benchmark has progressed from a differentiator to a qualification test — scoring above 9.0 is now table stakes for any serious frontier model.
MT-Bench vs Other Benchmarks
| Benchmark | Evaluation Method | Questions | Multi-turn | Cost to Run |
|---|---|---|---|---|
| MT-Bench | LLM judge (GPT-4) | 80 | Yes (2 turns) | Low (~$5) |
| Chatbot Arena | Human crowdsourcing | Millions of votes | Varies | Very high |
| AlpacaEval 2.0 | LLM judge | 805 | No | Moderate (~$15) |
| WildBench | LLM judge | 1,024 | No | Moderate |
| LMSYS Hard Prompts | Human + LLM | Curated | Yes | Moderate |
MT-Bench and Chatbot Arena are complementary evaluations from the same LMSYS team. Arena provides the most accurate ranking through massive human preference data but requires months to accumulate enough votes for a new model. MT-Bench provides instant, reproducible evaluation at the cost of some accuracy. For model developers, MT-Bench is used during development, with Arena serving as final validation.
AlpacaEval 2.0 offers a larger question set (805 vs 80) with length-controlled judging that mitigates verbosity bias, making it a useful complement. However, it tests only single-turn responses, missing the multi-turn coherence MT-Bench captures.
Practical Implications
For users choosing a general-purpose AI assistant, MT-Bench scores above 9.0 indicate the model will provide high-quality conversational responses across a wide range of tasks. The difference between models scoring 9.45 and 9.52 will not be perceptible in daily use. Users should base their choice on other factors — pricing, speed, specific domain performance, and preferred conversational style.
The more actionable signal comes from per-category breakdowns. A user who primarily needs a coding assistant should compare models’ coding category scores rather than the overall average. Users focused on creative writing, mathematical reasoning, or information extraction should examine the relevant category scores, where meaningful differences may still exist among frontier models.
For model developers, MT-Bench remains valuable as a rapid quality check during training and fine-tuning. A new model variant that drops more than 0.2 points likely has a conversational quality regression worth investigating. The benchmark’s low cost and fast execution make it suitable for continuous integration testing.
For enterprises evaluating chatbot deployments, MT-Bench scores serve as a minimum quality threshold. Models scoring below 8.5 may produce noticeably lower-quality conversations that affect user satisfaction and task completion rates.
Frequently Asked Questions
How reliable is GPT-4 as a judge?
GPT-4 judge ratings correlate at approximately 80% with human expert judgments. This is strong enough for rough tier classification (distinguishing a 8.0 model from a 9.0 model) but not precise enough for fine-grained ranking (distinguishing 9.45 from 9.50). Known biases include preferences for longer responses, positional effects in pairwise comparisons, and potential stylistic affinity for OpenAI models.
Why does MT-Bench use only two turns?
Two turns was chosen as the minimum necessary to test multi-turn coherence while keeping the benchmark small and fast. The first turn establishes a baseline response, and the second tests adaptation — can the model refine, correct, or extend its answer? More turns would provide richer evaluation but increase cost, complexity, and the cascading variance from earlier turns affecting later scoring.
Is MT-Bench still useful alongside Chatbot Arena?
Yes, for different purposes. Arena provides the most accurate overall ranking but takes months and cannot be run privately. MT-Bench evaluates any model in minutes at minimal cost, making it essential for development iteration, ablation studies, and regression testing. MT-Bench is a fast approximation; Arena is the ground truth.
What does a 0.1-point difference on MT-Bench mean?
Very little at the frontier. With only 80 questions and an LLM judge, a 0.1-point difference is within normal variance. Score differences of 0.3 points or more are generally meaningful, while differences below 0.15 points should be treated as noise. The benchmark is better for tier classification (below 8.5, 8.5-9.0, above 9.0) than precise ranking.
Does scoring well on MT-Bench guarantee a good assistant experience?
Not entirely. MT-Bench does not test response latency, instruction-following precision, safety behavior, multilingual ability, long-context handling, or hallucination rates. A model scoring 9.5 on MT-Bench is likely a strong conversational model, but users should evaluate additional dimensions relevant to their specific needs.