RedPajama

Overview

RedPajama is Together AI’s open reproduction of the LLaMA training data recipe. Created to give the open-source community full transparency into large model training data, it replicates the seven data sources Meta described in their LLaMA paper at a 1.2 trillion token scale.

What’s In It

RedPajama-v1 contains 1.2 trillion tokens from seven sources matching the LLaMA recipe: Common Crawl, C4, GitHub code, Wikipedia, books, ArXiv papers, and Stack Exchange. RedPajama-v2 expanded to over 30 trillion tokens with quality signals and deduplication metadata.

How It’s Used

RedPajama enabled training of fully open models where both weights and data are available. Together AI’s RedPajama-INCITE models used it directly. The v2 release lets researchers study filtering strategies without re-crawling the web.

Controversies

Like all web-derived datasets, RedPajama inherits copyright and quality concerns from its sources. The project aimed to address the transparency gap where models were released as open weights but trained on undisclosed data.