HumanEval is the foundational benchmark for AI code generation, introduced by OpenAI’s Mark Chen, Jerry Tworek, and colleagues in July 2021 alongside the Codex model. It consists of 164 hand-written Python programming problems, each providing a function signature and docstring — the model generates the function body, which is then verified against a hidden test suite. The pass@1 metric (percentage of problems solved correctly on the first attempt) has become the universal shorthand for comparing basic code generation ability across language models.
What HumanEval Measures
HumanEval tests a model’s ability to translate a natural-language specification into working Python code. Each problem provides a function signature (name and parameters), a docstring describing the expected behavior, and sometimes example inputs and outputs. The model must generate the function body — the implementation that satisfies the specification.
Problems span a range of programming concepts: string manipulation, list processing, mathematical computation, recursion, sorting algorithms, data structure operations, and basic algorithmic reasoning. The difficulty ranges from trivial (reverse a string) to moderately challenging (implement a topological sort or parse nested parentheses). None of the problems are at the level of competitive programming contests, and most could be solved by a competent second-year computer science student.
OpenAI created HumanEval specifically to evaluate Codex, their first code-specialized language model. The benchmark was designed to be simple enough that a capable model should eventually solve most problems, while providing enough variety to reveal weaknesses in specific programming areas. The 164 problems were hand-crafted by researchers to avoid overlap with common online coding exercise databases, though this distinctiveness has eroded over time as the benchmark has been widely published and discussed.
Current Leaderboard Analysis
The HumanEval leaderboard in 2025 tells a clear story: the benchmark is saturated at the frontier. Claude Opus 4 leads at 94.5%, followed by GPT-4.1 at 93.4%, Claude Sonnet 4 at 92.0%, and Gemini 2.5 Pro at 91.8%. The top four models are separated by just 2.7 percentage points, all scoring above 91%. At this level, the differences are not practically meaningful — all of these models can reliably generate correct Python functions for the vast majority of standard programming tasks.
The second tier — GPT-4o at 90.2%, DeepSeek V3 at 89.6%, Grok-3 at 88.7%, and Llama 4 Maverick at 87.1% — scores within a few points of the leaders, further confirming saturation. Even the lowest-scored model in this group solves 87 out of every 100 problems correctly on its first attempt.
The spread from top to bottom is only 7.4 percentage points. Compare this to SWE-bench Verified, where the gap between first and last place exceeds 34 points. HumanEval simply does not have the difficulty headroom to separate frontier models in any meaningful way.
Methodology Deep Dive
Each of the 164 problems follows a consistent format. The model receives a Python function skeleton: the function name, parameter list, type annotations (in most cases), and a docstring explaining the function’s purpose including example inputs and outputs. The model generates the function body, which is then executed against a set of unit tests.
The primary metric is pass@1 — the probability that a single generated sample passes all test cases. This is the strictest practical metric, reflecting the experience of a developer who generates one completion and expects it to work. Researchers also report pass@10 and pass@100, which measure the probability that at least one of 10 or 100 generated samples is correct, but pass@1 is the standard comparison number.
The original test suite contains approximately 7-8 test cases per problem. EvalPlus, an extension created by researchers at the University of Illinois, adds thousands of additional test cases — typically 80-100 per problem — specifically designed to catch edge cases, boundary conditions, and subtle implementation errors that the original tests miss. Models that score 93-94% on standard HumanEval typically drop 3-6 points on the EvalPlus variant (HumanEval+), revealing correctness gaps that the sparse original tests fail to catch.
Evaluation is deterministic when temperature is set to 0. Most published pass@1 scores use greedy decoding (temperature 0) for reproducibility, though some report averaged scores across multiple samples at low temperature.
Why This Benchmark Matters
HumanEval’s importance is primarily historical and referential. It established the methodology for evaluating code generation — provide a specification, generate code, run tests — that every subsequent coding benchmark has built upon. When Codex scored 28.8% in 2021, it was the first convincing demonstration that language models could write functional code. The rapid climb from 28.8% to 94.5% in under four years is one of the most dramatic capability progressions in AI research.
The benchmark also serves as a universal baseline. Because virtually every model reports HumanEval scores, it provides the most comprehensive cross-model comparison available for any coding benchmark. It is the one test where you can find scores for frontier models, mid-tier models, small open-weight models, and code-specialized models alike — enabling comparisons that newer, harder benchmarks cannot yet support.
For smaller and open-weight models, HumanEval remains genuinely informative. Models in the 7B-13B parameter range still show meaningful variation on HumanEval, making it useful for evaluating code-capable models that are not yet at the frontier.
Known Limitations and Criticisms
HumanEval’s most fundamental limitation is its narrow scope: 164 Python-only problems testing isolated function generation. Real software engineering involves reading existing code, modifying files, handling dependencies, writing tests, and maintaining large codebases — none of which HumanEval addresses. A model that aces HumanEval may still struggle with the kind of code work that developers actually do daily.
Data contamination is a severe concern. The 164 problems and their solutions have been published, discussed, and reproduced across thousands of blog posts, tutorials, GitHub repositories, and research papers since 2021. It is nearly impossible for a model trained on web data to not have been exposed to at least some of these problems in its training set. This makes it increasingly difficult to know whether high scores reflect genuine coding ability or memorization.
The test suite’s sparsity is another issue. With only 7-8 tests per problem, a function that handles the common cases correctly but fails on edge cases can still pass. The EvalPlus extension partially addresses this, but many published scores still use the original test suite, inflating apparent capability.
The problems are exclusively Python, which has relatively forgiving syntax, dynamic typing, and extensive standard library support. Models that score 93% on Python HumanEval may perform significantly worse on equivalent problems in statically-typed languages like Rust or Go, where type system constraints and memory management add complexity that Python abstracts away.
How Scores Have Changed Over Time
| Year | Top Model | Score | Key Insight |
|---|---|---|---|
| 2021 | Codex (12B) | 28.8% | First convincing demonstration of AI code generation |
| 2022 | code-davinci-002 | 47.0% | Code-specialized training dramatically improved results |
| 2023 | GPT-4 | 67.0% | General-purpose model rivaled code-specialized ones |
| 2023 | GPT-4 (0-shot CoT) | 82.0% | Chain-of-thought prompting unlocked further gains |
| 2024 | GPT-4o | 90.2% | Broke the 90% barrier, signaling saturation |
| 2025 | Claude Opus 4 | 94.5% | Incremental gains confirm benchmark ceiling |
The trajectory reveals two distinct phases. From 2021-2023, scores climbed rapidly as models improved at understanding programming concepts and generating correct code. From 2024 onward, gains have been incremental, with top models asymptotically approaching 100%. The remaining unsolved problems tend to require complex algorithmic reasoning or precise handling of unusual edge cases that even strong models find challenging.
HumanEval vs Other Benchmarks
| Benchmark | Scope | Language | Problems | Difficulty Level |
|---|---|---|---|---|
| HumanEval | Function generation | Python | 164 | Easy-Medium |
| MBPP | Simple generation | Python | 974 | Easy |
| SWE-bench Verified | Real-world engineering | Python | 500 | Hard |
| Aider Polyglot | Code editing | 12+ languages | 133 | Medium |
| CodeContests | Competition problems | Multiple | 165 | Very Hard |
| LiveCodeBench | Fresh problems | Python | Rolling | Medium-Hard |
HumanEval sits at the easy end of the coding benchmark spectrum. For evaluating frontier models in 2025, it has been largely superseded by SWE-bench (real-world engineering), Aider Polyglot (multi-language editing), and LiveCodeBench (fresh problems to avoid contamination). Its continued value lies in its universality — it remains the one benchmark where every model has a score.
LiveCodeBench deserves special mention as a benchmark designed to address HumanEval’s contamination problem by using programming problems published after a specific cutoff date, ensuring models have never seen them during training.
Practical Implications
For developers choosing a model for code generation, HumanEval scores above 90% are functionally equivalent. Every frontier model can write Python functions from specifications with high reliability. The practical differences between these models show up in harder, more realistic tasks — editing existing code (measured by Aider Polyglot), fixing bugs in real codebases (SWE-bench), and handling languages beyond Python.
If you are evaluating a smaller or open-weight model for code generation — say a 7B or 13B parameter model — HumanEval remains a useful discriminator. The spread between a model scoring 65% and one scoring 80% is substantial and will be noticeable in practical use. For these models, HumanEval still measures something meaningful.
For enterprises building coding assistants or code generation pipelines, HumanEval should be treated as a minimum viable threshold, not a differentiator. A model that cannot score above 85% on HumanEval likely has fundamental code generation weaknesses that will manifest in production. But among models scoring above 90%, other benchmarks and domain-specific evaluations are far more informative.
The EvalPlus variant provides a simple way to get more signal: if two models score similarly on standard HumanEval, their HumanEval+ scores may reveal which one handles edge cases more robustly. This matters for production code generation where edge case handling is the difference between functional and reliable.
Frequently Asked Questions
What is pass@1 and why is it the standard metric?
Pass@1 measures the probability that a single generated code sample passes all unit tests. It reflects the most common use case: a developer requests one code completion and expects it to work without modification. Higher pass@k values (pass@10, pass@100) indicate the model can produce a correct solution among multiple attempts, which matters for systems that generate multiple candidates and select the best one. Pass@1 is the standard because it is the most practically relevant metric.
Is HumanEval still worth tracking in 2025?
As a frontier differentiator, no — every top model scores above 91%, making the differences statistically insignificant. As a baseline qualification test and for evaluating smaller models, yes. HumanEval also retains value as the most widely-reported coding benchmark, enabling comparisons across a broader range of models than any alternative. Think of it as a pass/fail test for code generation competence rather than a ranking tool for the best models.
What is EvalPlus and why does it matter?
EvalPlus is an extension that adds approximately 80-100 additional unit tests per HumanEval problem, specifically targeting edge cases, boundary conditions, and uncommon inputs. Models that appear to score similarly on standard HumanEval can differ by 3-6 points on EvalPlus, revealing which ones generate code that is truly robust versus merely handling the happy path. For evaluating code generation reliability, EvalPlus scores are more informative than standard HumanEval.
How does data contamination affect HumanEval scores?
Significantly. The 164 problems have been circulating online since 2021, appearing in blog posts, tutorials, training datasets, and code repositories. Research has found that many models produce solutions that closely match known solutions posted online, suggesting memorization rather than genuine problem-solving. Newer benchmarks like LiveCodeBench use post-training-cutoff problems to address this issue.
Why do models score lower on HumanEval+ than HumanEval?
Standard HumanEval has only 7-8 test cases per problem, which often test only common cases. HumanEval+ adds tests for empty inputs, large inputs, negative numbers, special characters, and other edge cases. Models that generate code handling the typical case but failing on boundary conditions pass standard HumanEval but fail HumanEval+. The drop is typically 3-6 percentage points for frontier models.