Segment Anything Model
The foundation model that changed computer vision forever. SAM can segment any object in any image with just a single click—without ever seeing that object type before.
Before April 2023, image segmentation was a closed-world problem. You trained a model to segment cats, and it could segment cats. Dogs? Train another model. Medical images? Another model. Every new domain required collecting thousands of labeled examples, training for days, and hoping the model generalized within that narrow slice of the visual world.
The Old World (Pre-SAM)
- ✗Train separate models for each object category
- ✗Need 10,000+ labeled masks per category
- ✗Models fail on unseen object types
- ✗Interactive segmentation was slow and clunky
- ✗No unified approach across domains
The SAM World
- ✓One model segments everything
- ✓Zero-shot transfer to new objects
- ✓Works on medical, satellite, microscopy images
- ✓Interactive: click, box, or describe
- ✓50ms per mask—real-time interaction
The GPT-3 Moment for Vision
SAM did for segmentation what GPT-3 did for language: it created a foundation model that could be prompted to perform tasks it was never explicitly trained for. Just as GPT-3 showed that scale + diverse data = emergent generalization, SAM showed that training on 1 billion masks creates a model that understands the concept of “object” at a fundamental level—regardless of what that object is.
Why This Matters Beyond Academia
Segment tumors, organs, cells—without domain-specific training
Segment any obstacle, even novel ones never seen in training
One-click object removal, background replacement, compositing
SAM wasn't just a larger model—it required three simultaneous breakthroughs: a new task definition, a new model architecture, and a new data engine. Any one of these alone wouldn't have been enough. The magic was in their combination.
Breakthrough 1: Promptable Segmentation Task
Previous segmentation tasks asked: “Segment all cats in this image.” SAM asks: “Given this prompt (a point, box, or mask), segment the object it refers to.” This reframing is crucial because:
- •The model doesn't need to know object categories—just object boundaries
- •Prompts provide the “what to segment” signal that replaces category labels
- •The same architecture handles clicks, boxes, and rough masks as input
Breakthrough 2: Efficient Architecture
SAM's architecture is designed for amortized inference: compute the expensive image embedding once, then generate multiple masks cheaply. This enables real-time interaction.
Breakthrough 3: The Data Engine
You can't train a foundation model without foundation-scale data. But 1 billion high-quality masks didn't exist. Meta built a data flywheel:
Professional annotators used early SAM to speed up labeling. SAM proposes, humans correct.
SAM auto-detects obvious objects; humans label the rest. Model improves, auto-detects more.
Grid of 32×32 points → SAM generates all plausible masks → filter by confidence and NMS.
The Virtuous Cycle
Better model → faster annotation → more data → better model. This cycle is why SAM's dataset is 400× larger than any previous segmentation dataset. The model literally helped create its own training data.
SAM's architecture is elegantly simple: three components, each with a clear responsibility. The design philosophy is separation of concerns—the expensive computation (understanding the image) is done once, while the cheap computation (responding to prompts) can be done many times.
Image Encoder
A ViT-H (Vision Transformer Huge) pre-trained with MAE (Masked Autoencoder). Processes the input image into a rich 64×64×256 feature map that captures all visual information needed for segmentation.
Prompt Encoder
Converts user prompts (points, boxes, masks) into embeddings that can interact with image features. Points and boxes become positional encodings; masks are convolved into the embedding space.
Mask Decoder
A lightweight transformer that fuses image and prompt embeddings, then upsamples to produce the final mask. Outputs multiple masks + confidence scores to handle ambiguity.
Why This Design is Genius
The image encoder is a ViT-H/16 (Vision Transformer Huge with 16×16 patches) that was pre-trained using MAE (Masked Autoencoder) on ImageNet. This combination—huge capacity + self-supervised pre-training—gives SAM its remarkable ability to understand diverse images.
Architecture Specifications
Why MAE Pre-training?
MAE randomly masks 75% of image patches, then trains the model to reconstruct them. This forces the model to learn:
- •Global context: Understanding whole objects from partial views
- •Local details: Reconstructing fine textures and edges
- •Spatial relationships: How parts relate to wholes
Windowed Attention
Global self-attention on 4,096 tokens would be O(n²) = 16.7M operations per layer. SAM uses:
- •14×14 local windows: Most layers attend within windows
- •4 global layers: Layers 5, 11, 17, 23 use global attention
- •Result: 5× faster than full global attention
The Output: 64×64×256 Feature Map
The encoder's 1,280-dim embeddings are projected down to 256 dimensions, giving us a dense spatial feature map where each of the 4,096 positions encodes rich information about a 16×16 pixel region. This map is:
Every 16×16 region has its own embedding—no information loss from pooling
Each embedding knows about the whole image through attention
Computed once, used for any number of prompts
Model Size Comparison
All numbers are for the image encoder only. The flagship model uses ViT-H.
The prompt encoder translates human intent into a language the mask decoder understands. It handles three types of prompts: sparse prompts (points and boxes) anddense prompts (masks). Each type is encoded differently, but all produce embeddings that can be combined with image features.
Prompt Encoding
Sparse tokens: 1 (1 foreground point)
How it works: Each point becomes a 256-dim embedding via Fourier positional encoding + a learned type embedding (foreground/background). Boxes are encoded as 2 corner points with special corner embeddings. All sparse prompts are concatenated into a sequence that the mask decoder attends to.
PE uses sin/cos at multiple frequencies
Two corners with top-left/bottom-right embeddings
Sparse Prompts: Points
A click at position (x, y) becomes a 256-dimensional embedding through:
- •PE(x, y): Fourier positional encoding of coordinates
- •type_embed: Learned embedding for “foreground” or “background”
- •Foreground points say “include this”; background points say “exclude this”
Sparse Prompts: Boxes
A bounding box is encoded as two points: top-left and bottom-right corners.
- •Each corner has its own learned type embedding
- •The decoder learns that boxes mean “segment something inside this region”
- •Boxes are more informative than single points—less ambiguity
Dense Prompts: Masks
When you have a rough mask (from a previous iteration or another source), SAM can refine it:
↓ 2× Conv layers
Output: 64×64×256 dense embedding
The mask is downsampled to match the image embedding resolution
Combining Multiple Prompts
SAM can handle multiple prompts simultaneously: several foreground points, background points, a box, and a mask—all at once. Sparse prompts are concatenated into a sequence; dense prompts are added to image features. This flexibility lets users iteratively refine selections: “include this, exclude that, refine this edge.”
The mask decoder is where image understanding meets user intent. It's a lightweight transformer (~4M parameters) that fuses image embeddings with prompt embeddings, then upsamples to produce high-resolution masks. Despite its simplicity, this component does the heavy lifting of actually producing the segmentation.
Image embeddings (64×64×256) + prompt embeddings + 4 output tokens (3 masks + 1 IoU)
Two-Way Attention
Unlike standard cross-attention, SAM's decoder has bidirectional flow: tokens attend to image, AND image attends to tokens. This lets the image “understand” the prompt.
Why Only 2 Layers?
The heavy lifting is done by the image encoder. The decoder just needs to route information between already-rich representations. More layers didn't improve quality.
Decoder Architecture
Image embeddings (64×64×256) + prompt embeddings + learned output tokens. Output tokens are like “queries” that will become the final masks.
Each layer performs: (1) self-attention on tokens, (2) cross-attention from tokens to image, (3) MLP, (4) cross-attention from image to tokens. This bidirectional flow is key.
Two transposed convolution layers upsample the image features 4× while the output tokens become per-mask MLP heads that predict mask logits.
Each output token's MLP produces weights that are dot-producted with the upsampled features to produce a 256×256 mask. Final resize to original resolution.
Why Two-Way Attention?
Standard cross-attention only goes one direction (queries attend to keys). SAM's bidirectional design lets image features also attend to prompts. This means:
- •Image features become “aware” of what the user wants
- •Prompt tokens gather relevant visual context
- •Both sides refine each other over two layers
IoU Prediction Head
Each output token also produces an IoU score—the model's confidence that this mask is correct. This is crucial for:
- •Ranking multiple mask proposals
- •Filtering low-confidence masks in automatic mode
- •Telling users “I'm not sure about this one”
Resolution Trade-off
The decoder outputs 256×256 masks, which are then bilinearly upsampled to the original resolution. This is a deliberate trade-off: higher internal resolution would be more accurate but slower. For most use cases, 256×256 is sufficient—edges are refined by the final resize, and the perceptual difference is minimal.
SA-1B (Segment Anything 1 Billion) is the largest segmentation dataset ever created, and its existence is inseparable from SAM itself. The dataset and model were built together in a co-evolutionary process where each improved the other.
Dataset Statistics
Phase 1: Assisted Manual
Professional annotators used a browser-based tool with an early SAM model. SAM predicted masks from clicks; annotators corrected errors. Each image took ~14 seconds to fully annotate.
Phase 2: Semi-Automatic
SAM now auto-detected “confident” masks. Annotators only labeled objects SAM missed. Average annotation time dropped to ~7 seconds per image as SAM caught more objects.
Phase 3: Fully Automatic
32×32 grid of points → SAM generates all plausible masks → NMS and confidence filtering. No human involvement. This is where the billion masks came from.
Comparison to Prior Datasets
| Dataset | Images | Masks | Masks/Image |
|---|---|---|---|
| COCO | 118k | 896k | 7.6 |
| LVIS | 100k | 1.3M | 13 |
| Open Images | 1.7M | 2.7M | 1.6 |
| SA-1B | 11M | 1.1B | 100 |
Geographic Diversity
Unlike ImageNet (heavily US/European), SA-1B explicitly sought geographic balance. Images come from all inhabited continents, with deliberate oversampling of underrepresented regions. This matters because visual concepts vary by culture—buildings, food, clothing, signs all look different globally.
Privacy & Ethics
All faces and license plates were automatically blurred before release. Images were licensed from stock photo providers (not scraped from the web). Meta also conducted extensive bias audits—SAM performs equally well across perceived gender, skin tone, and age groups.
Training SAM required solving a chicken-and-egg problem: you need a good model to generate training data, but you need training data to make a good model. The solution was iterative co-training—improving model and data in alternating cycles.
Training Progression
Train on COCO + LVIS + public datasets. Use this model for assisted manual annotation → 4.3M masks.
Retrain on 4.3M masks. Now good enough for semi-automatic mode → 5.9M additional masks.
Retrain on 10.2M masks. Quality high enough for fully automatic mode → 1.1B masks.
Final training on all 1.1B masks. This is the released SAM model.
Loss Functions
SAM uses a combination of losses to train the mask prediction:
Focal loss handles class imbalance; Dice loss ensures good overlap; IoU loss trains the confidence head.
Training Hyperparameters
Prompt Simulation During Training
Since users will provide diverse prompts, training must simulate this diversity:
- •Random points: Sample points uniformly within ground-truth masks (foreground) and outside (background)
- •Noisy boxes: Ground-truth boxes with random jitter (simulating imprecise human drawing)
- •Mask prompts: Previous iteration's output (train for iterative refinement)
- •Mixed prompts: Combinations of the above (most realistic)
The most remarkable property of SAM is zero-shot generalization: it segments objects it has never seen in domains it was never trained on. This isn't transfer learning (fine-tuning on target domain)—it's true zero-shot, using SAM exactly as released.
Domains SAM Handles (Zero-Shot)
- ✓Medical: X-rays, CT scans, MRI, microscopy
- ✓Satellite: Aerial imagery, building footprints, agriculture
- ✓Scientific: Cell cultures, astronomical objects, materials
- ✓Art: Paintings, drawings, digital art, historical documents
- ✓Underwater: Marine life, coral reefs, underwater structures
- ✓Industrial: Defect detection, part segmentation, quality control
Why Does This Work?
SAM learned the concept of “object boundary” rather than specific object categories:
- •Visual universals: Edges, textures, and depth cues work across domains
- •Prompt grounding: The click says “segment HERE” regardless of what “here” is
- •Scale of training: 1B masks include enough edge cases to generalize
Benchmark: 23 Diverse Datasets
Meta evaluated SAM on 23 datasets spanning different domains, image types, and segmentation tasks. Key finding: SAM matches or exceeds supervised baselines on most datasets—without any training on them.
Where Zero-Shot Struggles
SAM performs worse on: (1) extremely low-contrast images (infrared, certain medical modalities), (2) highly abstract boundaries (semantic segmentation like “road” vs “sidewalk”), and (3) very fine structures (hair strands, thin wires). These often need domain-specific fine-tuning.
A single click is inherently ambiguous. Click on a person's eye—do you want the eye, the face, or the whole person? SAM's solution: output multiple masks at different granularity levels, each with a confidence score. The user (or downstream system) picks the right one.
The Three Mask Scales
Largest valid object containing the click
A meaningful sub-component
Fine-grained detail
How It Works
The mask decoder has 3 output tokens instead of 1. Each token is trained to specialize in a different granularity level:
- •Token 0: Learns to predict whole objects
- •Token 1: Learns to predict parts
- •Token 2: Learns to predict subparts
During training, each token is matched to its closest ground-truth mask (Hungarian matching), which naturally causes specialization.
IoU Confidence Ranking
Each mask comes with a predicted IoU score (0 to 1). This lets you:
- •Default to highest: Just use the mask with highest IoU
- •Show all three: Let user pick the right granularity
- •Filter low confidence: Discard masks below threshold
Example: Click on a Car Wheel
Just the wheel hub cap
The entire wheel including tire
The whole car
All three are “correct” segmentations—they just answer different interpretations of “what did the user mean?”
When Ambiguity Disappears
With sufficient prompts (multiple points, or a box), ambiguity resolves. SAM then returns a single mask (single-mask mode). Boxes are especially unambiguous: “segment the primary object inside this region.”
Explore SAM's architecture components interactively. These demos illustrate the key design decisions that make SAM work.
Architecture Flow
Prompt Encoding
Prompt Encoding
Sparse tokens: 1 (1 foreground point)
How it works: Each point becomes a 256-dim embedding via Fourier positional encoding + a learned type embedding (foreground/background). Boxes are encoded as 2 corner points with special corner embeddings. All sparse prompts are concatenated into a sequence that the mask decoder attends to.
PE uses sin/cos at multiple frequencies
Two corners with top-left/bottom-right embeddings
Mask Decoder Mechanics
Image embeddings (64×64×256) + prompt embeddings + 4 output tokens (3 masks + 1 IoU)
Two-Way Attention
Unlike standard cross-attention, SAM's decoder has bidirectional flow: tokens attend to image, AND image attends to tokens. This lets the image “understand” the prompt.
Why Only 2 Layers?
The heavy lifting is done by the image encoder. The decoder just needs to route information between already-rich representations. More layers didn't improve quality.
SAM was a breakthrough, but not a solution to all problems. Understanding its limitations helps you know when to use it and when to look elsewhere.
SAM's Limitations
- ✗No video: Each frame is independent—no temporal consistency
- ✗Slow encoder: 150ms per image limits real-time video
- ✗Fine structures: Hair, fur, thin objects are often imprecise
- ✗Semantic confusion: Can't distinguish “all dogs” from “this dog”
- ✗No text prompts: Points and boxes only (text was experimental)
- ✗Memory intensive: ViT-H requires significant GPU RAM
SAM 2 Improvements
- ✓Video support: Temporal memory propagates masks across frames
- ✓6× faster: Hiera encoder replaces ViT for real-time video
- ✓Better boundaries: Improved training on fine structures
- ✓SA-V dataset: 51k videos with 600k+ masklets
- ✓Occlusion handling: Tracks objects through temporary occlusions
- ✓Streaming mode: Process video frame-by-frame efficiently
SAM vs SAM 2: When to Use Which
- • Working with single images
- • You need the extensively-tested original model
- • Memory is extremely constrained
- • Using existing SAM-based pipelines
- • Working with video or image sequences
- • Real-time performance is needed
- • Tracking objects across frames
- • Better boundary precision is required
The Bigger Picture
SAM demonstrated that foundation models work for vision, not just language. The pattern—massive data, promptable interface, zero-shot transfer—is now being applied to 3D (SAM3D), medical imaging (MedSAM), and beyond. The era of training task-specific vision models is ending; the era of prompting general vision models has begun.
🎯 SAM: The Complete Picture
SAM proved that with enough scale, diversity, and the right task formulation, a single model can learn to segment anything. It's not just a model—it's a new paradigm for computer vision.