Research Papers

Used pure reinforcement learning to train language models to develop sophisticated reasoning chains, achieving performance competitive with OpenAI's o1 without supervised reasoning examples.

Combined multi-head latent attention with fine-grained mixture-of-experts to create a 236B parameter model with only 21B active parameters, achieving GPT-4-level performance at a fraction of the cost.

Introduced Anthropic's frontier model family with industry-leading safety practices, strong multilingual performance, and vision capabilities across three capability tiers.

Released a 314B parameter mixture-of-experts model as open weights, the largest open MoE model at the time, demonstrating xAI's competitive position after less than a year of operation.

Introduced Google's natively multimodal model family that processes text, images, audio, and video interleaved, achieving state-of-the-art results across 30+ benchmarks.

Introduced a selective state space model that matches Transformer-quality language modeling with linear scaling in sequence length, offering an alternative to quadratic attention.

Released a 7B parameter model that outperformed LLaMA 2 13B on all benchmarks through architectural innovations including grouped-query attention and sliding window attention.

Created a benchmark of 2,294 real GitHub issues and their verified solutions, measuring whether AI systems can autonomously resolve actual software engineering tasks.

Introduced Alibaba's competitive open-weight model family with strong multilingual capabilities, particularly Chinese-English performance, trained on 3 trillion tokens.

Released open-weight language models with commercially permissive licensing, including RLHF-aligned chat variants, establishing the template for responsible open-source AI deployment.

Systematically cataloged the open problems and fundamental limitations of RLHF, including reward hacking, feedback quality, and the challenges of scalable oversight.

Developed an activation-aware quantization method that identifies and preserves salient weight channels, achieving better quality than GPTQ at the same compression level.

Showed that a 1.3B parameter model trained on high-quality textbook-style synthetic data could match or exceed models 10x larger on code generation benchmarks.

Simplified RLHF by eliminating the need for a separate reward model, directly optimizing language models on human preferences using a simple classification loss.

Proposed a hybrid architecture combining the efficient linear-time inference of RNNs with the parallelizable training of Transformers, enabling competitive language modeling without quadratic attention.

Showed that language models can generate their own instruction-following training data, bootstrapping alignment from a small set of seed examples without expensive human annotation.

Extended chain-of-thought into a tree search over reasoning paths, allowing language models to explore, evaluate, and backtrack through multiple solution strategies.

Built a promptable image segmentation model trained on 1 billion masks, capable of segmenting any object in any image given points, boxes, or text prompts as input.

Demonstrated that fine-tuning a 7B model on 52K self-generated instructions could produce ChatGPT-like capabilities for under $600, democratizing instruction-tuned AI.

Documented the capabilities of OpenAI's most powerful model at the time, demonstrating human-level performance on professional exams and broad multimodal understanding.

Fine-tuned LLaMA on 70K user conversations shared from ChatGPT, achieving 90% of ChatGPT quality and establishing the Chatbot Arena evaluation framework.

Released a family of open-weight foundation models (7B to 65B parameters) that matched GPT-3 performance at a fraction of the size, catalyzing the open-source AI movement.

Introduced a method to accelerate inference by using a small draft model to propose tokens that a larger model verifies in parallel, achieving 2-3x speedups with no quality loss.

Demonstrated that language models can learn when and how to call external tools (calculators, search engines, APIs) by training on self-generated examples of tool use.

Introduced a method for training AI to be helpful and harmless using a set of written principles instead of relying entirely on human labelers, enabling more scalable and transparent alignment.

Trained a speech recognition model on 680,000 hours of weakly supervised web audio, achieving near-human accuracy across languages and accents without fine-tuning.

Developed a one-shot quantization method that compresses large language models to 3-4 bits per weight with minimal accuracy loss, enabling inference of 175B models on a single GPU.

Redesigned the attention computation to be aware of GPU memory hierarchy, achieving 2-4x speedups and enabling much longer context windows without approximation.

Showed that using a large frozen language model as the text encoder for diffusion models produces unprecedented photorealism, proving that language understanding is the key bottleneck in text-to-image generation.

Scaled a dense Transformer to 540 billion parameters using Google's Pathways system, demonstrating breakthrough performance on reasoning tasks and discontinuous capability jumps at scale.

Moved the diffusion process to a compressed latent space, making high-quality image generation 10-100x more efficient and enabling Stable Diffusion's open-source revolution.

Proved that most large language models were dramatically undertrained, showing that smaller models trained on more data outperform larger models trained on less data for the same compute budget.

Demonstrated how to align language models with human intent using reinforcement learning from human feedback, making models helpful and safe.

Augmented Transformer language models with a retrieval mechanism over a 2 trillion token database, improving performance with a fraction of the parameters of comparable models.

Showed that prompting language models to explain their reasoning step-by-step dramatically improves performance on math, logic, and complex reasoning tasks.

Introduced the Codex model and HumanEval benchmark of 164 hand-written programming problems, establishing the standard evaluation for AI code generation capabilities.

Introduced a parameter-efficient fine-tuning method that freezes model weights and injects small trainable matrices, reducing trainable parameters by 10,000x while matching full fine-tuning performance.

Introduced Rotary Position Embedding, which encodes position information through rotation of the query and key vectors, enabling better generalization to longer sequences than seen during training.

Demonstrated that a Transformer trained on text-image pairs could generate coherent images from arbitrary text descriptions, establishing the text-to-image generation paradigm.

Trained a model to connect images and text in a shared space using 400 million image-text pairs from the internet, enabling zero-shot visual classification and powering text-to-image generation.

Simplified mixture-of-experts routing to scale Transformers to a trillion parameters while keeping computation costs manageable by activating only a fraction of the model for each input.

Showed that a standard Transformer applied directly to sequences of image patches can match or exceed CNNs on image classification, unifying vision and language architectures.

Created a comprehensive benchmark spanning 57 academic subjects from elementary math to professional law, establishing the standard measure of language model knowledge breadth.

Pioneered training reward models from human comparisons and using them to optimize language model outputs via reinforcement learning, establishing the RLHF methodology later used in ChatGPT.

Introduced the modern diffusion model framework that generates images by learning to gradually remove noise, enabling high-quality image generation that eventually surpassed GANs.

Demonstrated that a 175B parameter language model can perform tasks from just a few examples in context, establishing in-context learning as a new paradigm and proving the scaling hypothesis.

Combined language models with an external knowledge retrieval system, allowing models to look up facts rather than memorizing everything, reducing hallucination and enabling updatable knowledge.

Established precise mathematical relationships between model size, dataset size, compute budget, and language model performance, providing a predictable roadmap for scaling AI.

Demonstrated that knowledge distillation could compress BERT to 60% of its size while retaining 97% of performance, establishing the template for efficient model deployment.

Introduced sparse attention patterns that reduce the quadratic cost of Transformer attention to O(N*sqrt(N)), enabling the processing of much longer sequences.

Showed that a 1.5B parameter language model trained on internet text could perform diverse NLP tasks without explicit fine-tuning, establishing the zero-shot learning paradigm.

Introduced bidirectional pre-training for language models, enabling a single model to be fine-tuned for virtually any NLP task.

Introduced the Transformer architecture, replacing recurrence and convolutions with self-attention mechanisms that now underpin virtually all modern language models.