HellaSwag (Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations) is a commonsense reasoning benchmark that tests whether AI models understand how everyday situations unfold. Created by Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi at the University of Washington and the Allen Institute for AI in 2019, HellaSwag presents a scenario and asks the model to select the most plausible continuation from four options — one correct and three adversarially generated to be superficially plausible but subtly wrong. The benchmark was designed to be trivial for humans (95.6% accuracy) but challenging for AI, though frontier models have now matched and exceeded human performance.
What HellaSwag Measures
HellaSwag measures commonsense reasoning — the intuitive understanding of how physical and social situations develop over time. This includes knowledge about everyday activities (cooking, sports, home repair), physical causality (what happens when you drop a glass), social interactions (how conversations typically proceed), and procedural sequences (the steps involved in common tasks).
Each question presents an opening scenario derived from either ActivityNet video captions (describing physical activities) or WikiHow articles (describing procedural tasks). The model must select which of four possible continuations most plausibly follows. The correct answer is the actual continuation from the source material. The three wrong answers were generated by a language model and then filtered through an adversarial process to select distractors that are grammatically correct, topically relevant, but factually or logically implausible.
The adversarial generation process is what made HellaSwag novel when it was introduced. Rather than using hand-written wrong answers (which tend to be obviously incorrect), the researchers used a language model to generate candidate continuations, then selected the ones most likely to fool another language model while remaining clearly wrong to humans. This “adversarial filtering” methodology ensured the benchmark tested genuine understanding rather than surface-level statistical patterns.
Current Leaderboard Analysis
HellaSwag is effectively solved. The entire leaderboard is compressed into a 3.2-point range, with GPT-4.1 at 97.1% and DeepSeek V3 at 93.9%. Every frontier model exceeds the average human accuracy of 95.6%, and the top three — GPT-4.1 (97.1%), Claude Opus 4 (96.8%), and Gemini 2.5 Pro (96.5%) — are clustered within 0.6 points of each other. These differences are not statistically meaningful given the benchmark’s variance.
The practical interpretation is clear: commonsense reasoning, as measured by HellaSwag, is a solved problem for frontier language models. The benchmark can no longer differentiate between the best systems. Even the lowest-scoring model tracked here (DeepSeek V3 at 93.9%) approaches human-level performance.
The leaderboard does reveal one useful pattern: models that score below 94% on HellaSwag tend to have broader capability gaps. HellaSwag functions as a floor test — failing it signals that a model may have fundamental deficiencies in understanding real-world situations, which will manifest as errors in downstream tasks like instruction following, question answering, and dialogue.
Methodology Deep Dive
HellaSwag contains 10,042 multiple-choice questions in the validation set (the test set labels were originally withheld but have since become effectively public). Each question has four options. The questions are divided roughly evenly between ActivityNet-derived scenarios (physical activities captured in video) and WikiHow-derived scenarios (step-by-step procedural instructions).
The adversarial filtering pipeline works as follows: for each correct continuation, a language model generates multiple candidate wrong answers. These candidates are then filtered to select those that a discriminator model finds most difficult to distinguish from the correct answer, while human annotators can still easily identify them as wrong. This iterative process was repeated with progressively stronger generator and discriminator models until the resulting distractors were maximally challenging for AI while remaining obvious to humans.
Scoring typically uses likelihood-based evaluation: the model assigns a probability to each of the four continuations, and the answer with the highest probability is selected. This is different from generation-based evaluation (where the model generates a response and it is matched against options) and is important because it leverages the model’s language modeling objective directly.
Some evaluations use zero-shot prompting (just the scenario and options) while others use few-shot prompting (with example questions and answers). The evaluation protocol can affect scores by 1-2 percentage points, so exact methodology should be noted when comparing results.
Why This Benchmark Matters
HellaSwag’s primary significance is historical. It demonstrated three important principles that shaped subsequent benchmark design. First, it showed that adversarial filtering could create evaluations that were trivially easy for humans but hard for contemporary AI — a design philosophy that influenced GPQA, TruthfulQA, and other benchmarks. Second, it quantified the commonsense reasoning gap: in 2019, the best model (BERT) scored 47.3% on a task where humans scored 95.6%, revealing how far AI was from human-like understanding of everyday situations.
Third, HellaSwag’s rapid saturation — from 47.3% to 97.1% in six years — provides one of the clearest demonstrations of how quickly language model capabilities can improve. This trajectory has been cited in hundreds of AI progress analyses, policy documents, and research papers as evidence that benchmark-measured capabilities can advance much faster than initially expected.
For current model evaluation, HellaSwag serves as a commonsense floor test. A model scoring below 90% on HellaSwag likely has fundamental gaps in its understanding of the physical and social world that will affect its performance on a wide range of practical tasks. This makes HellaSwag useful as a quick sanity check, even though it can no longer differentiate frontier models.
Known Limitations and Criticisms
The most obvious limitation is saturation. With top models scoring 96-97%, HellaSwag provides essentially no signal for distinguishing between frontier systems. The benchmark has fulfilled its purpose and been surpassed by the models it was designed to test.
The adversarial filtering methodology, while innovative in 2019, was designed to fool models of that era (BERT, GPT-2). The distractors that were deceptively plausible to 2019-era models are transparent to current systems, which have much stronger commonsense understanding. An updated HellaSwag using adversarial filtering against GPT-4-class models would be significantly harder, but no such update has been produced.
HellaSwag’s scenarios are drawn from English-language sources (ActivityNet and WikiHow) and reflect culturally specific commonsense knowledge. Activities, social norms, and procedural sequences vary across cultures, meaning the benchmark may not accurately measure commonsense reasoning for non-Western contexts.
The likelihood-based evaluation method can sometimes be gamed by models that have strong calibration on surface-level text features (sentence length, vocabulary frequency) rather than genuine understanding of the scenario’s logical structure. Research has shown that some of the score improvement from 2019 to 2022 reflected better language modeling rather than better commonsense reasoning per se.
How Scores Have Changed Over Time
| Year | Top Model | Score | Key Insight |
|---|---|---|---|
| 2019 | BERT-Large | 47.3% | Humans scored 95.6%; massive gap exposed |
| 2020 | GPT-3 (175B) | 78.9% | Scale alone closed much of the gap |
| 2021 | TNLG v2 (530B) | 90.9% | First model to break 90% |
| 2023 | GPT-4 | 95.3% | Reached average human accuracy |
| 2024 | GPT-4o | 95.3% | No improvement; plateau confirmed |
| 2025 | GPT-4.1 | 97.1% | Exceeded human accuracy, benchmark solved |
The progression illustrates a classic benchmark lifecycle: initial scores far below human performance, rapid improvement driven by scale and architecture advances, convergence with human accuracy, and eventual saturation. HellaSwag’s lifecycle took approximately six years from launch to effective saturation — faster than many predicted in 2019 but slower than some highly specialized benchmarks.
HellaSwag vs Other Benchmarks
| Benchmark | Tests | Status | Human Baseline | Current Top Score |
|---|---|---|---|---|
| HellaSwag | Commonsense continuation | Saturated | 95.6% | 97.1% |
| WinoGrande | Coreference commonsense | Saturated | 94.0% | ~96% |
| PIQA | Physical commonsense | Saturated | 94.9% | ~96% |
| ARC-Challenge | Science commonsense | Saturated | ~95% | ~97% |
| SIQA | Social commonsense | Near-saturated | 88.1% | ~92% |
HellaSwag belongs to a family of commonsense reasoning benchmarks — including WinoGrande, PIQA, and ARC-Challenge — that have all been effectively saturated by frontier models. As a group, they confirm that commonsense reasoning in the specific formats these benchmarks test (multiple-choice selection among predefined options) is a solved capability for models at the GPT-4 level and above.
The remaining frontier for commonsense reasoning evaluation lies in open-ended scenarios where models must generate commonsense inferences without multiple-choice scaffolding, and in physical reasoning about novel situations not represented in training data. No widely-adopted benchmark yet fills this gap comprehensively.
Practical Implications
For users selecting a model, HellaSwag scores above 93% are effectively equivalent and should not influence the decision. Every frontier and near-frontier model has mastered commonsense continuation at human level or above. The benchmark is useful only as a red-flag detector: a model scoring below 90% on HellaSwag has fundamental commonsense gaps that will be noticeable in conversation, instruction following, and task completion.
For AI researchers, HellaSwag’s saturation illustrates why benchmark-driven development requires continuous creation of harder evaluations. The adversarial filtering methodology remains valuable and has been adapted for harder benchmarks, but the specific instances in HellaSwag no longer challenge modern systems.
For developers building applications that require commonsense understanding — chatbots, virtual assistants, content generators — HellaSwag confirms that this capability is no longer a bottleneck for frontier models. The practical failure modes in commonsense reasoning that users encounter in 2025 (e.g., models making physically implausible claims or misunderstanding social context) are more nuanced than what HellaSwag tests and require different evaluation approaches.
HellaSwag remains included in many evaluation suites (including the Open LLM Leaderboard) for historical continuity and because it still provides signal for smaller open-weight models in the 1B-7B parameter range, where commonsense reasoning is not yet a solved problem.
Frequently Asked Questions
Why was HellaSwag so hard for AI but easy for humans in 2019?
The adversarial filtering process created distractors that exploited the specific weaknesses of 2019-era language models — sensitivity to word co-occurrence patterns, inability to track causal chains, and poor physical reasoning. Humans easily recognized these distractors as absurd because they could draw on embodied experience and intuitive physics that models lacked. The gap exposed how much of what appeared to be “language understanding” in early models was actually sophisticated pattern matching without genuine comprehension.
Is HellaSwag still used in model evaluations?
Yes, but primarily for smaller models and as part of comprehensive evaluation suites. The Open LLM Leaderboard on Hugging Face includes HellaSwag as one of its standard benchmarks. For frontier model comparisons, it provides no useful signal. For evaluating 1B-13B parameter open-weight models, meaningful score differences still exist.
What replaced HellaSwag for commonsense evaluation?
No single benchmark has directly replaced HellaSwag. The field has shifted toward evaluating commonsense reasoning through harder, more integrated benchmarks like MMLU (which includes commonsense-adjacent subjects), GPQA (which requires commonsense as part of scientific reasoning), and through human preference evaluations like Chatbot Arena where commonsense errors are penalized by human judges.
Does high HellaSwag performance mean a model has common sense?
Not entirely. HellaSwag tests a specific form of commonsense reasoning — selecting plausible continuations of scenarios from a fixed set of options. Models can score near-perfectly on this task while still making commonsense errors in open-ended generation, such as claiming that humans have three arms or that ice sinks in water. The multiple-choice format provides scaffolding that compensates for reasoning weaknesses that emerge in free-form settings.
How does adversarial filtering work in HellaSwag?
A generator model creates candidate wrong answers for each scenario. A discriminator model then evaluates these candidates, and the ones that are most difficult for the discriminator to distinguish from the correct answer are selected. This process is iterated — each round produces harder distractors. Human annotators verify that the selected distractors remain obviously wrong to people, ensuring the benchmark tests AI-specific weaknesses rather than genuinely ambiguous situations.