You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

These weights are DERIVED from Meta's SAM3 (SAM License) and are provided for research/evaluation. Redistribution requires SAM License pass-through. Commercial/ production use requires a separate license from General Instinct, Inc.

InstinctSAM — Compressed SAM3 (commercial-clean components)

Compressed components for SAM3 open-vocabulary ("concept", text-prompted) instance segmentation. This repo ships only our own distilled weights — a compressed ViT-B vision trunk and commercial-clean GIText text towers — which you graft onto the (separately licensed, Meta-gated) SAM3 heads. No Meta vision/head weights and no Apple/MobileCLIP (apple-amlr) weights are redistributed here.

Metric: cgF1 on SA-Co/Gold = pmF1 × IL_MCC (official SAM3 leaderboard metric). Teacher (SAM3, 840M): cgF1 0.521 on our 6544-pair SA-1B/Gold eval.

The compression↔accuracy frontier (6544-pair SA-Co/Gold cgF1)

variant	what's compressed	params	vs teacher	cgF1	% teacher	commercial
teacher SAM3	—	840M	1.0×	0.521	100%	—
LiteText (GIText-large)	text 354M→63M	551M	1.5×	0.493	94.6%	✅ clean
LiteText (GIText-base)	text 354M→44M	530M	1.6×	0.489	93.9%	✅ clean
Hiera-L (vision-compressed)	vision 454M→150M (SAM2-Hiera-L)	537M	1.6×	0.431	82.7%	✅ clean

Read the in_res before trusting a number in this table. The Hiera trunk resizes internally, and that knob moves accuracy more than any other choice here. The 0.431 above is the original concept-distilled trunk at its native resolution. Drop it to in_res=768 for speed and that same trunk collapses to 0.277 — small objects are what break (recall on sub-16px instances falls 0.146 → 0.008). The multi-resolution trunk exists to fix exactly that: 0.348 @768 and 0.411 @1152, i.e. it trades ~0.02 of peak for +0.07 at the resolution you actually ship. If you run below native resolution, use the multires weights.

This is worth stating plainly because we shipped the wrong configuration for a while: a 6-frame spot check green-lit in_res=768 using a single person prompt on video frames, where every object is large. It was structurally blind to the failure it was meant to catch.

| vision-compressed (ViT-B, earlier) | vision 454M→107M | 493M | 1.7× | 0.353 | 68% | ✅ clean | | dual-compressed | vision + text | 182M | 4.6× | 0.246 | 47% | ✅ clean |

Key findings. (1) The text encoder is the "free" thing to compress — distilling it costs ~10% accuracy for a similar param saving, while vision compression costs ~32%. (2) Our commercial-clean GIText text tower (a from-scratch CLIP-BPE transformer, our code, no Apple dependency) matches and beats the apple-amlr MobileCLIP LiteText (0.493 vs 0.469) at 90–95% of teacher, distilled from SAM3's PE-text on a broad open-vocab + RefCOCO prompt set. All variants beat the released EfficientSAM3 (0.133 on this harness) by 1.9–3.7×.

Files

gitext_large_v4.pt — commercial-clean text tower, 63M (GIText-large; 0.493/94.6%).
gitext_base_v3.pt — commercial-clean text tower, 44M (GIText-base, 0.489/93.9%).
hiera_large_concept_trunk.pt — compressed vision trunk, SAM2-Hiera-L 150M (concept-distilled, 0.431/82.7% teacher) — the best vision-compression point; graft into model.backbone.vision_backbone.trunk. Preserves occlusion tracking (~teacher). Full pipeline (with the trunk TensorRT-compiled) measures 12.7 FPS on an idle A100 (vision+neck 29 ms / decode+mask 49 ms); the decode+mask head is the dominant floor, so ≥15 FPS at ≥80% needs a lighter decoder. Thor not yet measured — run scripts/bench_thor.py on-device. Reproduction recipe + edge-speed detail: docs/HIERA_BACKBONE.md in the GitHub repo.
concept_vitb_trunk_step6000.pt — earlier ViT-B vision trunk (concept-distilled, 0.353/68%).
vit_base_stageA.pt — ViT-B vision trunk (Stage-A feature distill).

Usage (sketch)

Build SAM3 (your own gated SAM3 checkpoint), swap in the GIText text tower for the recommended "LiteText" variant. Full reproduction, training recipe, and eval harness: https://github.com/william-Dic/InstinctSAM (see docs/MODEL_CARD.md, src/train_text.py, src/eval_saco_cgf1.py).

⚠️ Licensing

These weights are derived from SAM3 → SAM License with pass-through (research/eval OK; redistribute under the same terms + include the license).
The GIText text tower architecture is our own (standard CLIP-BPE transformer) — no apple-amlr / MobileCLIP dependency, so it is commercially usable where SAM3's own license permits.
Commercial/production use of the combined system requires a separate license from General Instinct, Inc. — guanming@general-instinct.com.
SA-1B (distillation data) — Meta research license; hold your own rights.

Hiera-L (vision-compressed) — read this before using it

The heads are not optional

hiera_large_concept_trunk.pt alone does not reproduce our numbers. Concept distillation co-trains the DETR decoder (175 tensors) and dot_prod_scoring (10 tensors) with the trunk; every one of them differs from stock SAM3. Loading only the trunk silently leaves stock SAM3 heads on a distilled trunk. Always pair a *_trunk.pt with its matching *_heads.pt.

Everything else (neck, fusion encoder, segmentation head, text tower) is bit-identical to official SAM3 and comes from sam3.pt.

Accuracy depends on `in_res` — always report it

The trunk wrapper resamples its input to an internal in_res (native 1152) before Hiera and re-interpolates the output to the fixed 72-grid. Lowering in_res is the main speed lever and costs real accuracy. cgF1 on the 6544-pair SA-1B/Gold subset, official standalone_cgf1.py, teacher = 0.521:

in_res	multires weights	original (1152-only) weights	% of teacher	Thor FPS
1152	0.411	0.431	78.9%	7.6
896	0.364	0.303	69.9%	9.5
768	0.348	0.277	66.8%	10.7
640	0.307	0.234	58.9%	11.6

The _multires_ weights were distilled with in_res randomised per step, so they hold up at every deployment resolution. The original weights are only valid at 1152 — at 768 they lose 30 points of teacher retention, almost entirely in small objects (sub-16px recall 0.146 → 0.008).

in_res must be a multiple of 64 (1008 fails).

Files

file	use
`hiera_large_multires_trunk.pt` + `hiera_large_multires_heads.pt`	recommended — robust across `in_res`
`hiera_large_concept_trunk.pt` + `hiera_large_concept_heads.pt`	original; 0.431 but only at `in_res=1152`

Assemble

import torch
from sam3.model_builder import build_sam3_image_model, _create_student_vision_backbone

m = build_sam3_image_model(checkpoint_path='sam3.pt', load_from_HF=False, device='cuda').eval()
trunk = _create_student_vision_backbone('hiera', 'large').trunk.cuda().eval()
trunk.load_state_dict(torch.load('hiera_large_multires_trunk.pt')['trunk'], strict=False)
m.backbone.vision_backbone.trunk = trunk

h = torch.load('hiera_large_multires_heads.pt')
m.transformer.decoder.load_state_dict(h['decoder'], strict=False)
m.dot_prod_scoring.load_state_dict(h['dot_prod_scoring'], strict=False)

# pick the speed/accuracy point
next(x for _, x in trunk.named_modules() if hasattr(x, 'in_res')).in_res = 768

Verified: this assembly scores 0.347 @768 and 0.411 @1152, matching the full training checkpoint. Method and reproduction: https://github.com/william-Dic/InstinctSAM/blob/main/docs/MULTIRES.md

SAM 3.0 vs SAM 3.1 — measured, and the answer is "just use them"

These weights were distilled against facebook/sam3 → sam3.pt. We measured what happens if you assemble them on facebook/sam3.1 → sam3.1_multiplex.pt instead.

cgF1 on the 6544-pair SA-Co/Gold subset, in_res 768, official standalone_cgf1.py:

base checkpoint	cgF1	pmF1	IL_MCC
SAM 3.0 `sam3.pt`	0.348	0.484	0.723
SAM 3.1 `sam3.1_multiplex.pt`	0.346	0.485	0.715

−0.002. Use either.

Why it is this small — per-module relative L2 between the two detector.* state dicts:

module	rel L2
PE-text text tower	4.3e-20 (bit-identical, frozen)
PE-ViT vision trunk	1.2e-03
neck `convs.0-2`	9.9e-03
DETR decoder	9.9e-03
`dot_prod_scoring`	1.3e-02
fusion encoder / seg head / geometry enc.	1.1–1.4e-02

SAM 3.1 is a ~1% continued fine-tune with the vision and text towers essentially frozen, not a retrain. config.json is byte-identical between the two releases (773 keys, zero differences), and the image model builds to the same 1134 tensors either way.

Loading note. build_sam3_image_model(checkpoint_path="sam3.1_multiplex.pt") reports 4 missing and 74 unexpected keys. Both are benign:

missing backbone.vision_backbone.convs.3.* — the 0.5-scale neck level, which is dead code: it is computed every frame and then thrown away, because SAM3VLBackbone(scalp=1) drops the last feature level (vl_combiner.py:91).
unexpected — the multiplex tracker (interactive_convs, propagation_convs, tracker.*), which the image path does not use.

Optional free speedup, verified bit-exact. You can delete that dead level, but you must change both settings together, because scalp drops the last level, not the 0.5 level:

# equivalent to the default, minus 0.8M params of conv computed and discarded every frame
Sam3DualViTDetNeck(..., scale_factors=[4.0, 2.0, 1.0])   # was [4.0, 2.0, 1.0, 0.5]
SAM3VLBackbone(..., scalp=0)                             # was scalp=1

Verified over 3 images: encoder input bit-identical (max abs diff 0.0), same instance count. Changing only scale_factors and leaving scalp=1 silently feeds the encoder the 144-grid instead of the 72-grid — do not do that.

On-device: Jetson AGX Thor (Blackwell SM110)

Measured on hardware, not projected. Multires Hiera-L trunk, in_res 768, TensorRT 10.16 FP16 (engine vs eager: rel L2 0.003, cosine 0.99999). Full method: https://github.com/william-Dic/InstinctSAM/blob/main/docs/SAM31_MIGRATION.md

Real-time text-prompted multi-object tracking, live USB webcam at 1280×720:

objects tracked	FPS
idle (no prompt)	15.5–16.3
1	10.4
6	10.5
11	11.0

Cost is flat in object count — that is the whole point, and it needs both releases at once:

backend	marginal cost per object	best for
SAM 3.0 per-object tracker	~38 ms	N = 1
SAM 3.1 multiplex (16-slot buckets)	~flat	N ≥ 2

SAM 3.1's multiplex tracker always runs a full 16-slot bucket, so it loses at N=1 (109 ms vs 97 ms) and wins from N=2 on (110 ms vs 134 ms), by 2.0–2.5× at N=6. We keep both predictors resident (7.7 GB, sharing one TRT trunk) and route on object count. Two caveats for anyone trying this:

SAM 3.1's multiplex path does not work out of the box. start_session forwards kwargs init_state rejects (sam3#544, open; Meta's own SAM 3.1 notebook raises TypeError), and a hand-rolled per-frame loop hits four further interface changes that are invisible through the stock session API.
The crossover does not transfer between trunk backends. We first measured it at N=4 with an eager trunk; with TensorRT it is N=2, because the engine shrinks the fixed cost that the bucket premium is compared against.

Compression and multiplex are orthogonal: compression cuts the fixed per-frame cost (279 → 181 ms at N=1), multiplex cuts the per-object slope. Neither substitutes for the other.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

You need to agree to share your contact information to access this model

InstinctSAM — Compressed SAM3 (commercial-clean components)

The compression↔accuracy frontier (6544-pair SA-Co/Gold cgF1)

Files

Usage (sketch)

⚠️ Licensing

Hiera-L (vision-compressed) — read this before using it

The heads are not optional

Accuracy depends on in_res — always report it

Files

Assemble

SAM 3.0 vs SAM 3.1 — measured, and the answer is "just use them"

On-device: Jetson AGX Thor (Blackwell SM110)

Accuracy depends on `in_res` — always report it