The Transformer is a neural network architecture that processes sequences by allowing every element to attend to every other element simultaneously, rather than reading input one step at a time. Introduced in Google’s 2017 paper “Attention Is All You Need” by Vaswani et al., it replaced recurrent neural networks (RNNs) and LSTMs as the dominant architecture for language tasks within two years of publication — and has since expanded to dominate vision, audio, protein folding, robotics, and virtually every other domain where deep learning is applied.
Every frontier language model in production as of 2026 — GPT-4.1, Claude Opus 4, Gemini 2.5 Pro, Llama 4, Mistral Large, and DeepSeek-V3 — is built on the Transformer or a direct descendant. The architecture’s combination of parallelizable training, scalable parameter counts, and flexible attention patterns has proven so effective that billions of dollars in research have failed to find a fundamentally superior alternative for general-purpose language modeling.
How the Transformer works
The Transformer processes input through a series of layers, each applying two main operations: self-attention and feed-forward transformation. Understanding these components reveals why the architecture works so well and where its limitations come from.
Tokenization and input embedding
Raw text first passes through a tokenizer that splits it into subword units called tokens. Each token is mapped to a learned vector (an embedding) that captures its semantic properties. The model also adds positional information so it knows where each token appears in the sequence — the original paper used sinusoidal position encodings, though modern models have moved to rotary position embeddings (RoPE) or learned position encodings.
Self-attention: the core mechanism
Self-attention is what makes the Transformer work. For each token in the sequence, the model computes three vectors: a Query (what am I looking for?), a Key (what do I contain?), and a Value (what information do I carry?). The model then computes attention scores by taking the dot product of each token’s Query with every other token’s Key, applies a softmax to normalize the scores, and uses these weights to create a weighted sum of Value vectors.
This mechanism lets the word “bank” in “river bank” attend strongly to “river” and produce a different representation than “bank” in “bank account,” where it attends to “account.” The model learns these attention patterns from data — no linguistic rules are hardcoded.
Each Transformer layer contains multiple attention heads (typically 32-128 in frontier models), each learning different types of relationships. Research has shown that different heads specialize: some track syntactic dependencies (subject-verb agreement), others capture semantic relationships, and others handle positional patterns. A model like Claude Opus 4, with 128 attention heads per layer across 80+ layers, runs thousands of parallel attention computations per token.
Feed-forward networks
After attention, each token’s representation passes through a position-wise feed-forward network — typically two linear transformations with a nonlinear activation (GeLU or SwiGLU) between them. This is where much of the model’s factual knowledge is stored. Research from Anthropic and others has shown that feed-forward layers act as key-value memory stores, mapping input patterns to associated knowledge.
Layer stacking and residual connections
A full Transformer stacks dozens of these attention + feed-forward blocks. GPT-4-class models use 80-120 layers. Each layer has a residual connection (the input is added back to the output) and layer normalization, which prevents training instability as the network gets deeper. The residual stream acts as a highway that information flows along, with each layer reading from and writing to it.
The three Transformer variants
The original paper described an encoder-decoder architecture, but practice has converged on three variants optimized for different tasks:
| Variant | How it works | Notable models | Primary use case |
|---|---|---|---|
| Encoder-only | Processes full input bidirectionally; every token attends to every other token | BERT, RoBERTa, DeBERTa, ModernBERT | Classification, semantic search, NER, embeddings |
| Decoder-only | Processes tokens left-to-right; each token only attends to preceding tokens (causal masking) | GPT-4.1, Claude Opus 4, Llama 4, Mistral Large | Text generation, chat, reasoning, agents |
| Encoder-decoder | Encoder reads full input bidirectionally, decoder generates output attending to both encoder output and its own previous tokens | T5, FLAN-T5, UL2, mBART | Translation, summarization, structured transformation |
Decoder-only models have dominated since 2020 because they scale more efficiently and can be trained with a single, simple objective (next-token prediction) on massive unlabeled corpora. The encoder-only variant remains essential for embeddings and retrieval tasks, where bidirectional understanding outperforms causal models.
The Transformer scaling timeline
The Transformer’s impact is inseparable from its scaling properties. The architecture’s parallelism meant it could exploit GPU clusters far more efficiently than RNNs, unlocking a scaling trajectory that has driven the entire modern AI era:
| Year | Milestone model | Parameters | Key innovation |
|---|---|---|---|
| 2017 | Original Transformer | 65M | Self-attention replacing recurrence |
| 2018 | BERT | 340M | Bidirectional pre-training + fine-tuning |
| 2018 | GPT-1 | 117M | Decoder-only pre-training on books |
| 2019 | GPT-2 | 1.5B | Scaled unsupervised pre-training |
| 2020 | GPT-3 | 175B | In-context learning at scale |
| 2021 | PaLM | 540B | Pathway-based distributed training |
| 2023 | GPT-4 | ~1.8T (est.) | Mixture-of-experts, multimodal |
| 2024 | Llama 3.1 405B | 405B | Open-weights frontier model |
| 2024 | Claude 3.5 Sonnet | Undisclosed | Efficient mid-size frontier performance |
| 2025 | DeepSeek-V3 | 671B (37B active) | MoE with extreme efficiency |
| 2025 | Claude Opus 4 | Undisclosed | Agentic reasoning, extended thinking |
| 2025 | Llama 4 Maverick | 400B (17B active) | Open-weights MoE rivaling proprietary models |
This 10,000x increase in parameters over eight years was accompanied by roughly proportional increases in training data (from gigabytes to tens of terabytes) and compute (from single GPUs to clusters of tens of thousands of accelerators costing hundreds of millions of dollars per training run).
Seven innovations that transformed the Transformer
The 2017 architecture has been heavily modified. Seven post-publication innovations have been critical to making Transformers work at modern scale:
1. Rotary Position Embeddings (RoPE). Proposed by Su et al. in 2021, RoPE encodes position information by rotating the Query and Key vectors in pairs of dimensions. Unlike learned absolute positions, RoPE generalizes to sequence lengths longer than those seen in training, which is essential for models with context windows of 128K-1M tokens. Used by Llama, Mistral, and most modern open-weights models.
2. Grouped-Query Attention (GQA). Standard multi-head attention requires separate Key and Value matrices for every head, which becomes a memory bottleneck during inference. GQA, introduced by Ainslie et al. in 2023, shares Key-Value heads across groups of Query heads — typically using 8 KV heads for 32 Query heads. This reduces KV cache memory by 4x with minimal quality loss. Used in Llama 3/4, Mistral, and Gemma.
3. Flash Attention. Developed by Tri Dao at Stanford in 2022, Flash Attention restructures the attention computation to minimize reads and writes to GPU high-bandwidth memory (HBM). By tiling the computation and keeping intermediate results in fast SRAM, Flash Attention delivers 2-4x wall-clock speedups and enables training on longer sequences without running out of memory. Now the default attention implementation in virtually all Transformer training.
4. Mixture-of-Experts (MoE). Instead of passing every token through all parameters, MoE architectures route each token to a subset of specialized expert networks. A model might have 671B total parameters but only activate 37B per token (as in DeepSeek-V3). This decouples model capacity from inference cost, enabling much larger effective models at practical serving costs. GPT-4, Mixtral, DeepSeek-V3, and Llama 4 Maverick all use MoE.
5. SwiGLU activation. Replacing the original ReLU activation in feed-forward layers with SwiGLU (Shazeer, 2020) improves model quality by 1-3% on language modeling benchmarks at no additional computational cost. Adopted by essentially all modern Transformers.
6. RMSNorm. Replacing Layer Normalization with Root Mean Square Normalization simplifies the normalization computation and reduces training time. Combined with pre-normalization (applying normalization before rather than after each sub-layer), RMSNorm has become standard in all major open-weights models.
7. KV Cache optimization. During autoregressive generation, the model recomputes attention over all previous tokens at each step. Caching the Key and Value matrices eliminates redundant computation but requires significant memory. Techniques like paged attention (used in vLLM), quantized KV caches, and sliding window attention manage this tradeoff in production serving systems.
Why alternatives haven’t displaced the Transformer
Despite known limitations — quadratic attention complexity with sequence length, large KV caches, and difficulty with very long contexts — alternatives have struggled to match the Transformer’s overall performance:
State-space models (SSMs). Mamba (Gu and Dao, 2023) and its successors process sequences in linear time rather than quadratic, making them theoretically superior for long sequences. In practice, hybrid architectures that combine SSM layers with attention layers (Jamba by AI21, Zamba by Zyphra) outperform pure SSMs, suggesting that self-attention captures something SSMs do not. Pure SSMs underperform Transformers of equivalent size on tasks requiring precise information retrieval from long contexts.
Linear attention. Various approaches approximate the softmax attention operation with linear-complexity alternatives. These work well for simple tasks but degrade on complex reasoning and long-range dependency tasks that require the full softmax attention distribution.
RWKV. An RNN-like architecture that can be trained in parallel like a Transformer but runs like an RNN at inference time. Competitive on language modeling benchmarks at small scales (1-14B parameters) but has not demonstrated frontier-level performance.
The consensus in 2026 is that the Transformer’s core attention mechanism is likely necessary for high-performance language modeling, but will increasingly be combined with more efficient components (SSM layers, linear attention for less critical layers) in hybrid architectures.
Transformers beyond language
The Transformer has been adapted to nearly every domain in AI, often replacing domain-specific architectures that had been refined over decades:
Computer vision. Vision Transformer (ViT), introduced by Google in 2020, splits images into patches and treats them as tokens. ViT and its successors (DINOv2, SigLIP) now match or exceed convolutional neural networks (CNNs) on image classification, object detection, and segmentation — tasks where CNNs dominated for nearly a decade.
Audio and speech. OpenAI’s Whisper uses an encoder-decoder Transformer for speech recognition, achieving near-human accuracy across 99 languages. Meta’s MusicGen uses a decoder-only Transformer for music generation. Audio is converted to spectrograms or discrete audio tokens and processed identically to text.
Protein structure. DeepMind’s AlphaFold 2 uses a modified Transformer (the Evoformer) to predict 3D protein structures from amino acid sequences, solving a 50-year grand challenge in biology. AlphaFold 3 extended this to protein-ligand interactions, drug discovery, and molecular design.
Multimodal models. Modern frontier models like GPT-4o, Gemini 2.5, and Claude Opus 4 process text, images, audio, and video through unified Transformer architectures, using modality-specific tokenizers that feed into a shared Transformer backbone.
Robotics and control. Google DeepMind’s RT-2 and Gato use Transformers to map sensory inputs directly to robot actions. The same architecture that generates text can generate motor control sequences when trained on robotics data.
The computational economics of Transformers
Training and serving Transformers at scale defines the economics of the AI industry:
| Cost category | Approximate range (2025-26) |
|---|---|
| Pre-training a frontier model (GPT-4 class) | $50M-$500M in compute |
| Training cluster (10,000+ H100 GPUs) | $300M-$1B capital expenditure |
| Serving a frontier model per 1M tokens | $1-$15 input, $2-$75 output |
| Serving an efficient mid-size model per 1M tokens | $0.10-$3 input, $0.25-$15 output |
| Fine-tuning a 70B model | $500-$5,000 per run |
| Running a 7B model on a consumer GPU | ~$0.50/hour electricity |
The quadratic attention complexity means that doubling the context window roughly quadruples the compute needed for the attention operation (though Flash Attention and other optimizations reduce the practical impact). This is why context window expansion has proceeded more slowly than parameter scaling.
Frequently asked questions
What is the difference between a Transformer and a large language model? A Transformer is a neural network architecture — a blueprint for how the model is structured. A large language model (LLM) is what you get when you build a very large Transformer and train it on a massive text corpus. All modern LLMs use the Transformer architecture, but not all Transformers are LLMs — smaller Transformers are used for classification, search, embeddings, speech recognition, and other tasks.
Why is the Transformer called “Transformer”? The name comes from the model’s ability to transform input representations through layers of self-attention. Each layer transforms the representation of every token based on its relationship to all other tokens, progressively building richer, more contextualized representations. The name was chosen by the original Google Brain team of Vaswani et al.
Will Transformers be replaced? As of 2026, no alternative architecture has matched the Transformer’s performance at frontier scale across diverse tasks. Hybrid architectures that combine Transformer attention layers with more efficient components (state-space models, linear attention) are gaining traction for specific use cases, but pure replacements have not materialized. The Transformer will likely remain dominant for at least the next several years, though it may evolve into hybrid forms.
How much does it cost to train a Transformer model? Costs span many orders of magnitude depending on size. Training a 7B parameter model from scratch costs roughly $50K-$200K in cloud compute. A frontier model (hundreds of billions of parameters with MoE) costs $50M-$500M. Fine-tuning an existing model is far cheaper — typically $500-$5,000 for a 70B model using parameter-efficient methods like LoRA.
What is self-attention? Self-attention is the mechanism that lets each token in a sequence compute a weighted combination of all other tokens, based on learned relevance scores. It is the defining innovation of the Transformer. The “self” prefix distinguishes it from cross-attention, where tokens from one sequence attend to tokens from a different sequence (used in encoder-decoder models for tasks like translation).