HumanEval

Overview

HumanEval is OpenAI’s benchmark of 164 hand-crafted programming problems designed to evaluate the functional correctness of code generation models. Unlike text-matching metrics, it tests whether generated code actually executes correctly against unit tests.

What’s In It

The dataset contains 164 Python programming problems, each with a function signature, docstring, example usage, and unit tests. Problems range from simple string manipulation to algorithmic challenges with approximately 7-8 test cases each.

How It’s Used

HumanEval’s pass@k metric became the standard for evaluating code generation models. Extensions include HumanEval+ and MultiPL-E. Every major code model reports HumanEval scores.

Controversies

With only 164 problems, HumanEval is susceptible to overfitting and data contamination. Many problems are simple enough that newer models saturate the benchmark. The Python-only focus limits multilingual assessment.