SlimPajama

Overview

SlimPajama is Cerebras’s cleaned and deduplicated version of RedPajama, reducing 1.2 trillion tokens to 627 billion high-quality tokens. It demonstrates that aggressive deduplication can significantly improve training efficiency.

What’s In It

SlimPajama contains 627 billion tokens derived through extensive MinHash LSH deduplication at the document level. The pipeline removes near-duplicates, filters short and low-quality documents, and rebalances domain proportions.

How It’s Used

SlimPajama is used for efficient pre-training where compute budgets are limited. The reduced size allows multiple training epochs on higher-quality data rather than single passes over larger but noisier corpora.

Controversies

The deduplication approach can sometimes remove legitimate repeated content like common phrases or standard document templates. Questions remain about optimal deduplication thresholds.