HellaSwag

Overview

HellaSwag is a commonsense reasoning benchmark that tests whether models can predict the most plausible continuation of a scenario. Created by Rowan Zellers at AI2, it uses adversarial filtering to generate wrong answers that fool models while remaining obvious to humans.

What’s In It

HellaSwag contains approximately 70,000 multiple-choice questions where models select the correct ending to a scenario from four options. Scenarios are drawn from ActivityNet captions and WikiHow articles describing physical or social activities.

How It’s Used

HellaSwag evaluates commonsense physical and social reasoning. It was initially very challenging (GPT-2 achieved only 50%) but has been largely saturated by modern frontier models exceeding 95% accuracy.

Controversies

The adversarial generation process means wrong answers exploit specific model weaknesses rather than testing general commonsense. As models improve, the adversarially generated distractors become less effective.