Plain-English Summary
SAM (Segment Anything Model) can identify and outline any object in any image based on simple prompts — click a point, draw a box, or describe what you want in words. Unlike previous segmentation models that could only identify objects from a fixed set of categories, SAM generalizes to any type of object it has never seen before. It works on photographs, medical images, satellite imagery, microscopy, and virtually anything else.
The model was trained on SA-1B, a dataset of over 1 billion segmentation masks collected using a human-AI annotation loop. This massive dataset gave SAM the breadth to handle arbitrary objects in arbitrary contexts.
Key Innovation
SAM was designed as a “foundation model for segmentation” — a model that can be prompted to perform any segmentation task rather than being trained for specific categories. The promptable design (accepting points, boxes, masks, or text as input) makes it a flexible building block that other systems can build upon.
The data engine was equally innovative: an iterative loop where SAM assisted human annotators, the annotations trained better SAM versions, and the improved model enabled faster annotation. This produced training data at unprecedented scale and diversity.
Impact on the Field
SAM established the foundation model paradigm for computer vision tasks beyond classification. Just as GPT-3 showed that a single pre-trained language model could handle many NLP tasks, SAM showed that a single pre-trained vision model could handle arbitrary segmentation tasks. This influenced how the field approaches other vision problems (detection, tracking, depth estimation).
The open release of SA-1B (the largest segmentation dataset ever created) provided the community with resources that accelerated research broadly.
Models That Built on This
SAM 2 extended the approach to video with temporal consistency. Grounding DINO combined SAM with text-based detection for open-vocabulary segmentation. SEEM unified segmentation, detection, and understanding in a single model. SAM has been integrated into Adobe products, medical imaging tools, autonomous driving systems, and hundreds of other applications that require precise object boundaries.