WebText

Overview

WebText is OpenAI’s proprietary dataset created for training GPT-2. It pioneered using social media signals as a quality filter, scraping text from URLs shared on Reddit with sufficient engagement. The dataset was never publicly released, motivating OpenWebText.

What’s In It

WebText contains approximately 40GB of text from 8 million web documents. URLs were collected from Reddit posts with at least 3 karma points, filtering for pages that received meaningful human engagement.

How It’s Used

WebText was used exclusively for training GPT-2, which demonstrated that sufficient scale and data quality could produce coherent long-form text generation. The design influenced numerous subsequent pre-training data curation strategies.

Controversies

OpenAI never released WebText, citing concerns about enabling malicious use of GPT-2. This closed approach contradicted open science norms and motivated EleutherAI and others to create open alternatives.