Plain-English Summary
LLaMA 2 was Meta’s second-generation open language model family, released with a permissive commercial license that allowed companies and developers to build products on top of the models. Available at 7B, 13B, and 70B parameter sizes, LLaMA 2 was trained on 2 trillion tokens and included both base models and chat-tuned variants aligned with RLHF.
The paper provided detailed descriptions of the safety training process, including red-teaming, iterative refinement of the reward model, and extensive safety evaluations. This level of transparency about the alignment process was unprecedented for a model release of this scale.
Key Innovation
The paper’s most significant contribution was demonstrating that alignment techniques could be effectively applied to open models and documenting the full process transparently. The rejection sampling and RLHF pipeline was described in sufficient detail for others to reproduce, creating a shared understanding of production-quality alignment.
Ghost Attention (GAtt), a technique for maintaining system prompt adherence across multi-turn conversations, was a practical contribution that improved the usability of chat models. The safety benchmarks and red-teaming methodology set a standard for responsible open-source releases.
Impact on the Field
LLaMA 2’s commercial license transformed the open-source AI economy. Companies could build products on LLaMA 2 without the legal ambiguity that surrounded the original LLaMA’s research-only license. This enabled an ecosystem of startups, enterprise deployments, and fine-tuned variants that could be commercialized.
The paper’s detailed safety documentation also established expectations for responsible open-weight releases. Subsequent releases from Mistral, Alibaba, and others followed similar patterns of publishing safety evaluations alongside model weights.
Models That Built on This
CodeLlama specialized LLaMA 2 for programming. Zephyr used DPO on LLaMA 2 to create an efficient chat model. Thousands of commercial applications were built on LLaMA 2’s foundation. LLaMA 3 extended the approach with larger scale and more training data. The permissive licensing model Meta established with LLaMA 2 became the standard for open-weight releases.