SAM 3 Explained: Concept Segmentation, GPU Requirements & How to Run It

Introduction

There is a recurring debate right now: “Is computer vision finally a solved problem?” It is a tempting thought. We have models that can describe images with poetic detail and generators that can dream up video from scratch. Yet, until recently, if you wanted to ask a computer to “mask every single screw in this engine disassembly video,” you were out of luck. You had to click on them. One by one.

That changes with Segment Anything Model 3, or SAM 3.

Meta has just dropped the third iteration of their foundational vision model, and it represents a pivot from purely visual understanding to semantic understanding. While SAM 1 and SAM 2 were masterful at segmenting objects you explicitly pointed to, SAM 3 introduces a capability called Promptable Concept Segmentation. It does not just see pixels; it understands categories.

We are going to tear down the architecture, look at the benchmarks that matter, and show you how to run SAM 3 locally.

2026 update: Meta released SAM 3.1 Object Multiplex on March 27, 2026. It keeps SAM 3’s concept-segmentation capabilities but introduces shared-memory processing for much faster multi-object video tracking. Meta reports roughly a 7× speedup when tracking 128 objects on a single H100 compared with the original November 2025 SAM 3 release. The SAM 3 image examples below remain relevant, while users deploying SAM 3.1 should use the latest Meta repository code and checkpoints.

1. What is SAM 3? The Leap from “Visual” to “Concept” Segmentation

To understand why SAM 3 is a big deal, you have to look at the limitation of its predecessors. SAM 2 is an incredible piece of engineering, but it is fundamentally a reactive tool. You give it a click, a box, or a mask, and it gives you the object. It is a translator of user intent into pixel masks.

SAM 3 changes the game by accepting concepts as inputs.

This is Promptable Concept Segmentation (PCS). Instead of clicking on a specific car, you can prompt the model with the text “car” or “red vehicle.” The model then hunts down every instance of that concept across the image or video frame. It handles the logic of “finding” and “segmenting” simultaneously.

The unified architecture creates a strange new workflow where you can combine inputs. You can provide a text prompt (“cat”) and a visual exemplar (a bounding box around one specific cat), and the model generalizes from that combination to find all cats that look similar. It is a shift from “segment this” to “segment everything like this.”

2. Under the Hood: The “Presence Token” and Deep Encoder Architecture

The engineering team at Meta Superintelligence Labs didn’t just glue a language model to SAM 2. They rebuilt the pipeline to solve a specific headache in open-vocabulary segmentation: false positives.

When you ask a standard model to find “a unicorn” in a picture of a kitchen, standard detectors often hallucinate. They try to force the most “unicorn-like” blob (perhaps a blender) into a bounding box because they are optimized to localize, not to reject.

SAM 3 solves this with a Presence Head.

The architecture uses a shared Perception Encoder (PE) backbone that processes both vision and text. But here is the clever part: they decoupled the recognition (“is the object even here?”) from the localization (“where is the object?”).

A global Presence Token is responsible solely for predicting the probability that the concept exists in the frame. The localization queries then do the heavy lifting of drawing masks. The final score is a product of these two. If the Presence Token says “zero chance,” the localization queries get suppressed. This simple architectural tweak drastically reduces the hallucination rate for objects that aren’t there.

3. SAM 3 vs SAM 2: Key Differences and Improvements

If you are a developer deciding whether to upgrade, you need to know if the compute cost is worth it. SAM 3 is essentially a superset of SAM 2, meaning it retains all the visual prompting capabilities (clicks/boxes) while adding the semantic layer.

Here is the breakdown of the generational leap:

• Input Modality: SAM 2 relies on geometric prompts (points, boxes). SAM 3 accepts geometric prompts, text phrases, and image exemplars.
• Output Scope: SAM 2 is designed to segment a single target object per prompt. SAM 3 is designed to exhaustively segment all instances of a concept.
• Video Persistence: SAM 3 features a memory-based tracker that shares the backbone with the detector. It improves re-identification when objects disappear behind occlusions and re-emerge later—a notorious pain point in tracking.
• Disambiguation: SAM 3 employs a “hard mode-switch.” It can toggle between “concept mode” (find all cars) and “instance mode” (refine this specific car mask), preventing the logic errors that occur when models try to do both at once.

4. Breaking Down the Benchmarks: Is it Really SOTA?

Meta created a new benchmark specifically for this release called SA-Co (Segment Anything with Concepts). It is a beast of a dataset, containing over 207,000 unique concepts—roughly 50 times more than existing benchmarks.

The performance numbers suggest SAM 3 isn’t just a minor update.

In zero-shot evaluations on the LVIS dataset, SAM 3 achieved a mask AP (Average Precision) of 48.8. For context, the previous state-of-the-art was sitting at 38.5. That is a massive jump in the world of computer vision, where we usually fight for 1% gains.

SAM 3 benchmarks show particular strength in video consistency. By decoupling the detection and tracking, the model sustains near real-time performance for about 5 concurrent objects.

Here is a look at the comparative performance data:

The data shows SAM 3 doubling the accuracy of existing systems like OWLv2 on the new concept segmentation tasks. It approaches human performance in video tracking, which is arguably the hardest task in the suite.

5. Real-World Use Cases: Medical, Marine, and Creative

Synthesizing reddit discussions and early experiments from the community reveals where this model actually shines versus where it is just hype.

5.1 Medical Imaging:

Medical image segmentation AI is the “holy grail” application for tools like this. SAM 3 shows promise here because of its promptable nature. A radiologist could theoretically prompt “tumor” or “lesion.” Early tests show it is excellent for pre-labeling—generating a rough pass that a human expert refines. It does not replace the expert, as it can struggle with highly specific biological structures (like distinct intracranial arteries) without fine-tuning, but it speeds up the workflow massively.

5.2 Scientific Research:

Marine ecology researchers are already looking at SAM 3 for analyzing Scientific Research footage. The “Presence Token” is valuable here. If you are scanning hours of empty ocean floor for a specific starfish, you want a model that confidently says “nothing here” rather than hallucinating rocks as starfish.

5.3 Creative and VFX:

For video editors, the “concept segmentation” is a rotoscoping dream. You can type “person” and get a mask for every actor in the scene to apply color grading or effects. It removes the need to manually initialize trackers on every single person in a crowd shot.

6. SAM 3 GPU and VRAM Requirements: Can You Run It Locally?

Yes. SAM 3 can run locally, although Meta does not publish a fixed minimum VRAM requirement for inference. The current official repository requires Python 3.12 or newer, PyTorch 2.7 or newer, a CUDA-compatible GPU and CUDA 12.6 or newer.

6.1 SAM 3 VRAM Requirements

SAM 3 contains roughly 0.9 billion parameters, but model size alone does not determine VRAM usage. Runtime memory also depends on image resolution, precision, the number of prompts or objects being processed and whether you are working with images or video.

A 24GB RTX 3090 or RTX 4090 is therefore a practical high-end consumer GPU for experimenting with SAM 3, but 24GB should not be presented as Meta’s official minimum requirement. Meta does not currently specify one universal VRAM number.

The standard SAM 3 image pipeline is designed around approximately 1008×1008 input resolution. Hugging Face’s Transformers implementation also supports lower custom resolutions to reduce memory use and improve speed, although its documentation warns that changing the resolution can reduce accuracy.

6.2 SAM 3 Inference Speed

Meta reports that the original SAM 3 processes a single image with more than 100 detected objects in about 30 ms on an NVIDIA H200 GPU. That figure should not be treated as an RTX 4090 benchmark.

There is currently no official Meta benchmark giving one universal SAM 3 inference time for the RTX 4090. Consumer-GPU speed varies with resolution, implementation, precision, prompt type and workload, so an exact RTX 4090 latency should be measured on the intended pipeline rather than inferred from the H200 result.

6.3 SAM 3.1 and the Video Bottleneck

The original SAM 3 video implementation processes tracked objects independently, so inference cost grows as the number of objects increases. Meta addressed this in SAM 3.1 Object Multiplex, released March 27, 2026.

Object Multiplex groups multiple objects into shared-memory processing buckets. Meta reports roughly a 7× speedup at 128 tracked objects on a single H100 compared with the original November 2025 SAM 3 implementation, making SAM 3.1 the more relevant choice for demanding multi-object video workloads.

7. How to Use SAM 3: A Quick Start Guide

The code is out, and the barriers to entry are low. You have three primary ways to get this running.

7.1 Option 1: Direct Python Usage

This is for the engineers building pipelines. You will need to request access to the checkpoints via Hugging Face first.

import torch
from PIL import Image
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

# 1. Load the model (Requires CUDA)
model = build_sam3_image_model()
processor = Sam3Processor(model)

# 2. Load your image
image = Image.open("my_dataset_image.jpg")
inference_state = processor.set_image(image)

# 3. The Magic: Prompt with a concept
# SAM 3 will find ALL instances of this concept
output = processor.set_text_prompt(
    state=inference_state,
    prompt="red sports car"
)

# 4. Extract results
masks = output["masks"]
boxes = output["boxes"]
print(f"Found {len(masks)} instances.")

7.1B Using SAM 3 with Hugging Face Transformers

There are now two common SAM 3 Python APIs. Meta’s native repository uses build_sam3_image_model together with sam3.model.sam3_image_processor.Sam3Processor, as shown above. Hugging Face Transformers also provides Sam3Processor and Sam3Model directly.

import torch
from PIL import Image
from transformers import Sam3Model, Sam3Processor

device = "cuda" if torch.cuda.is_available() else "cpu"

model = Sam3Model.from_pretrained("facebook/sam3").to(device)
processor = Sam3Processor.from_pretrained("facebook/sam3")

image = Image.open("my_dataset_image.jpg").convert("RGB")

inputs = processor(
    images=image,
    text="red sports car",
    return_tensors="pt"
).to(device)

with torch.no_grad():
    outputs = model(**inputs)

results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)[0]

print(f"Found {len(results['masks'])} instances.")

Cannot import Sam3Processor from transformers?

If you see:

ImportError: cannot import name 'Sam3Processor' from 'transformers'

you are probably using an older Transformers installation that predates SAM 3 support. Upgrade Transformers and restart your Python or Jupyter environment:

pip install -U "transformers>=5.0.0" accelerate

Then use:

from transformers import Sam3Processor, Sam3Model

Do not confuse this with Meta’s native import:

from sam3.model.sam3_image_processor import Sam3Processor

Both classes are called Sam3Processor, but they belong to different APIs.

7.2 Option 2: Visualizing with Notebooks

The GitHub repository includes excellent Jupyter notebooks (sam3_image_predictor_example.ipynb). These are the best way to inspect the “Presence Score” and see how the model discriminates between positive and negative text prompts.

7.3 Option 3: Auto-Labeling Pipelines

The killer app for SAM 3 isn’t necessarily running it in production—it is using it to label data. You can use SAM 3 to auto-label a massive dataset of specific objects (e.g., “hard hats”) and then train a smaller, faster model (like YOLOv10 or RT-DETR) on those labels for edge deployment. Python scripts can automate this Auto-Labeling Pipelines process effectively.

8. Limitations and “Hallucinations”

We need to be honest about what this model cannot do. SAM 3 is not an AGI. It is a pattern matcher with a good vocabulary.

8.1 Spatial Reasoning:

It struggles with queries that require logic. If you ask for “the man behind the car,” SAM 3 often fails to understand the prepositional relationship and might just segment both the man and the car. It segments nouns well; it segments relationships poorly.

8.2 Ambiguity:

The English language is messy. If you prompt “bat,” SAM 3 relies on visual context to decide between the animal and the baseball equipment. If the image is ambiguous, the model’s guess is a coin toss. The “Ambiguity Head” in the architecture tries to mitigate this by predicting multiple valid masks, but user guidance is often still required.

8.3 Generalization:

While it is “open vocabulary,” it is not omniscient. It generalizes poorly to niche domains (like thermal imagery or specific industrial parts) without fine-tuning. The “concept” understanding breaks down when the visual features deviate too far from the training distribution.

9. Conclusion: The “GPT Moment” for Computer Vision?

Segment Anything Model 3 feels like a foundational shift. We are moving away from the era where computer vision models were just “eye” simulators that needed manual pointing. We are entering an era where they have a brain behind the eyes.

The decoupling of recognition and localization via the Presence Token is a technical insight that will likely be copied across the industry. It solves the fundamental problem of “don’t find things that aren’t there.”

For developers and researchers, the value of SAM 3 lies in its versatility. Whether you are building medical image segmentation AI pipelines or just trying to automate video editing, this model raises the baseline of what is possible out of the box.

The weights are available. The code is on GitHub. It is time to see what you can build with it.

Next Step: Would you like me to generate a specific Python script for batch-processing a folder of images using SAM 3 to auto-generate masks for a custom dataset?