Overview
OpenWebText is an open-source recreation of OpenAI’s proprietary WebText dataset. It extracts and cleans text from URLs shared on Reddit with at least three upvotes, replicating the curation strategy that produced GPT-2’s training data.
What’s In It
The dataset contains approximately 38GB of deduplicated text extracted from 8 million web pages. URLs were sourced from Reddit submissions with a karma score of 3 or higher, using community upvotes as a quality signal.
How It’s Used
OpenWebText served as training data for RoBERTa and numerous other models needing a high-quality web text corpus. Its Reddit-based curation provides a natural quality filter that selects for engaging, readable content.
Controversies
Reddit’s demographic skew means OpenWebText over-represents content popular with young, English-speaking, male-dominated communities. Some extracted content may violate the original publishers’ terms of service.