Overview
The Pile is EleutherAI’s 825GB diverse English text dataset designed to train large language models. It was created to provide an open alternative to proprietary datasets, combining 22 high-quality sub-datasets spanning academic papers, books, code, web content, and specialized domains.
What’s In It
The Pile combines 22 component datasets including PubMed Central, ArXiv, GitHub, FreeLaw, Stack Exchange, USPTO patents, Project Gutenberg, Wikipedia, and OpenWebText2. Each component adds diversity and domain coverage. The dataset is deduplicated and quality-filtered, totaling approximately 825GB.
How It’s Used
The Pile served as training data for GPT-NeoX-20B, GPT-J-6B, and the Pythia model suite. Its transparent composition made it a cornerstone of open-source LLM development. Researchers use individual components for domain-specific training and the full dataset for studying data composition effects.
Controversies
Books3, a component containing pirated books, led to legal challenges. Some components contain personal information scraped from the web. EleutherAI provides tooling for copyright holders to check if their work is included.