Plain-English Summary
For decades, convolutional neural networks (CNNs) dominated computer vision. ViT asked a simple question: what if we just use a Transformer instead? The approach splits an image into a grid of 16x16 pixel patches, flattens each patch into a vector, and feeds the sequence of patch vectors to a standard Transformer encoder — the same architecture used for language. With sufficient training data, this approach matches or exceeds the best CNNs.
The simplicity was striking. No convolutions, no pooling, no vision-specific inductive biases. Just a standard Transformer applied to a sequence of image patches. When trained on enough data, the Transformer learns the visual features that CNNs have built-in.
Think of it like treating a photograph as a paragraph, where each “word” is a small patch of the image. The Transformer reads these visual “words” and understands the image the same way it understands text.
Key Innovation
ViT demonstrated that the Transformer’s general attention mechanism can discover visual features (edges, textures, objects) without the architectural priors that CNNs encode (locality, translation invariance). This requires more training data than CNNs but ultimately produces better representations. At sufficient scale, general architectures beat specialized ones.
The patch-based tokenization bridged vision and language — images and text become sequences of tokens processed by the same architecture, enabling unified multimodal models.
Impact on the Field
ViT collapsed the distinction between vision and language architectures. The same Transformer that processes text could process images, enabling unified multimodal models. This architectural unification made systems like CLIP, GPT-4V, and Gemini conceptually straightforward: just feed image and text tokens into the same Transformer.
The paper spawned DeiT, Swin Transformer, and BEiT families that refined ViT for different scales and applications. DINO and DINOv2 showed that ViT could learn excellent visual features through self-supervised learning alone.
Models That Built on This
CLIP uses a ViT image encoder. GPT-4V and Gemini use ViT-derived visual encoders. DINO and DINOv2 produce the best general-purpose visual features using ViT trained with self-supervision. Segment Anything (SAM) uses a ViT backbone. Most modern computer vision systems use Transformer-based architectures rather than CNNs, a shift initiated by this paper.