Overview
Common Crawl is the largest open repository of web crawl data, maintained by a non-profit foundation since 2008. It has become the foundational data source for training virtually all major language models, providing petabytes of raw web text that other datasets clean and filter.
What’s In It
Common Crawl produces monthly web crawls containing billions of web pages stored in WARC format. Each monthly crawl contains roughly 3-5 billion pages, with the total archive spanning petabytes across hundreds of languages and every conceivable web domain.
How It’s Used
Almost every major language model traces its training data back to Common Crawl. C4, RefinedWeb, RedPajama, The Pile, and ROOTS all derive from it with various filtering pipelines. It also powers search engines and academic web research.
Controversies
As a raw web crawl, Common Crawl contains toxic content, misinformation, copyrighted material, personal information, and spam. The challenge of effectively filtering this raw data while maintaining diversity is an active research area. Websites cannot easily opt out of inclusion.