Overview
SuperGLUE is the successor to GLUE, providing a more challenging suite of language understanding tasks. Created by a consortium including NYU and DeepMind, it tests diverse capabilities from coreference resolution to causal reasoning.
What’s In It
SuperGLUE comprises eight tasks: BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Each tests different language understanding capabilities with train, validation, and test splits.
How It’s Used
SuperGLUE served as the premier NLU benchmark from 2019-2021, motivating improvements in pre-training, architectures, and fine-tuning strategies. T5, DeBERTa, and GPT-3 all competed on its leaderboard.
Controversies
SuperGLUE was eventually saturated by large models achieving human-level performance. Some tasks have small evaluation sets making scores noisy. Its retirement left a gap that MMLU filled.