Upload README.md with huggingface_hub

README.md

@@ -36,18 +36,42 @@ This export was produced by hand-wrapping the three sub-modules of `Sam3Model` a
 
 ```
 sam3-text-onnx/
-├── vision_encoder.onnx          # graph (6.2 MB)
-├── vision_encoder.onnx.data     # weights (1.84 GB)
-├── text_encoder.onnx            # graph (3.0 MB)
-├── text_encoder.onnx.data       # weights (1.35 GB)
-├── decoder.onnx                 # graph + weights inline (96 MB)
-├── export_sam3_vision.py        # script that produced vision_encoder.onnx
-├── export_sam3_text.py          # script that produced text_encoder.onnx
-├── export_sam3_decoder.py       # script that produced decoder.onnx
-└── validate_sam3_e2e.py         # end-to-end Python validation harness
+│
+├─ fp32 reference (3.3 GB total) — bit-equivalent to PyTorch
+│  ├── vision_encoder.onnx               (6.2 MB graph)
+│  ├── vision_encoder.onnx.data          (1.84 GB weights)
+│  ├── text_encoder.onnx                 (3.0 MB graph)
+│  ├── text_encoder.onnx.data            (1.35 GB weights)
+│  └── decoder.onnx                      (96 MB, weights inline)
+│
+├─ int8 dynamic quantization (839 MB total) — production default
+│  ├── vision_encoder_int8.onnx          (473 MB)
+│  ├── text_encoder_int8.onnx            (340 MB)
+│  └── decoder_int8.onnx                 (26 MB)
+│
+├─ int4 MatMul quantization (654 MB total) — browser/mobile
+│  ├── vision_encoder_int4.onnx          (5.6 MB graph)
+│  ├── vision_encoder_int4.onnx.data     (279 MB weights)
+│  ├── text_encoder_int4.onnx            (2.7 MB graph)
+│  ├── text_encoder_int4.onnx.data       (348 MB weights)
+│  └── decoder_int4.onnx                 (19 MB)
+│
+├─ Export scripts (PyTorch → ONNX)
+│  ├── export_sam3_vision.py             # produces vision_encoder.onnx
+│  ├── export_sam3_text.py               # produces text_encoder.onnx
+│  └── export_sam3_decoder.py            # produces decoder.onnx
+│
+├─ Quantization scripts (ONNX fp32 → smaller)
+│  ├── quantize_sam3_fp16.py             # ⚠️ currently broken, see Known Caveats
+│  ├── quantize_sam3_int8.py
+│  └── quantize_sam3_int4.py
+│
+└─ Validation scripts
+   ├── validate_sam3_e2e.py              # end-to-end pipeline test
+   └── validate_all_variants.py          # compares fp32/int8/int4 side-by-side
 ```
 
-Total: **~3.3 GB at fp32**. Quantization is recommended for browser deployment — see [Quantization & next steps](#quantization--next-steps).
+Three precision variants, each independently usable. See [Which precision should I use?](#which-precision-should-i-use) for guidance.
 
 ## Architecture
 
@@ -191,73 +215,196 @@ const { input_ids, attention_mask } = await tokenizer("seed", {
 | **Out** | `pred_boxes` | `[batch, 200, 4]` | float32 | Boxes in xyxy format, **normalized to [0, 1]** |
 | **Out** | `pred_logits` | `[batch, 200]` | float32 | Apply sigmoid for [0, 1] confidence scores |
 
+## Which precision should I use?
+
+Three precision variants are published. They share the same input/output contracts above — pick whichever matches your constraints.
+
+| Need… | Use | Why |
+|---|---|---|
+| Reference accuracy / research baseline | **fp32** | Bit-exact match against PyTorch. Reproducible numbers. |
+| Production server inference (CPU or GPU > 6 GB VRAM) | **int8** | 4× smaller than fp32, same top detections, scores within 0.001 of baseline. AVX-512 VNNI and CUDA both accelerate int8 ops natively. |
+| Browser / mobile / GPU with ≤ 4 GB VRAM | **int4** | 5× smaller than fp32, keeps the right top-N detections, modest quality loss in low-confidence range. |
+| GPU inference on a 4 GB consumer card | **int8 or int4 (not fp32)** | fp32 exhausts VRAM via intermediate Softmax buffers (~1.7 GB). int8 and int4 both fit. |
+| Fastest CPU inference | **int8** | Modern CPUs accelerate int8 GEMMs via AVX-VNNI; ~3× faster than fp32 even on CPU. |
+
+**Default recommendation: int8.** It's the same quality as fp32 in practice but a quarter of the size and several times faster. Only use fp32 if you need bit-exact reproducibility for academic work. Only use int4 if you're shipping to a context where every megabyte matters (browsers, mobile, embedded).
+
+### Validation evidence
+
+On a microscope-style seed image with prompt `"seed"` (the test image used during development):
+
+| Variant | Detections > 0.5 | Max score | Top 5 scores |
+|---|---|---|---|
+| fp32 (baseline) | 12 | 0.926 | 0.926, 0.889, 0.878, 0.874, 0.868 |
+| int8 | 12 | 0.925 | 0.925, 0.905, 0.902, 0.898, 0.872 |
+| int4 | 12 | 0.898 | 0.898, 0.863, 0.844, 0.797, 0.784 |
+
+int8 is essentially indistinguishable from fp32. int4 keeps the right top detections but scores are ~3% lower with more noise in the low-confidence range.
+
 ## Performance
 
-Measured on CPU (Intel laptop), single image:
+Per-component timings for a single 1008×1008 image. Measured on a laptop with a quad-core Intel CPU and an RTX 3050 Laptop (4 GB VRAM):
 
-| Component | Time |
-|---|---|
-| vision_encoder | ~150 s |
-| text_encoder | ~7 s |
-| decoder | ~8 s |
-| **Total per image** | **~170 s** |
-| Re-run with different prompt (same image) | **~15 s** (vision cached) |
+| Variant | vision (CPU) | vision (GPU) | text | decoder (CPU) | Total (best path) |
+|---|---|---|---|---|---|
+| fp32 | 181 s | OOM on 4 GB | 7 s | 13 s | **201 s (CPU only)** |
+| int8 | 53 s | 27 s | 1 s | 14 s | **42 s (hybrid)** |
+| int4 | 166 s | 12 s | 0.1 s | 16 s | **28 s (hybrid)** |
 
-GPU (CUDA execution provider, RTX-class) should be roughly 10-15× faster. WebGPU in modern Chrome should fall between the two.
+### The GPU+CPU hybrid pattern
 
-The architecture is designed for prompt iteration: cache the vision encoder output, re-run text + decoder per prompt.
+On consumer GPUs with limited VRAM, the **vision encoder benefits massively from GPU** (it's a heavy ViT) but the **decoder runs better on CPU** because its attention layers need ~860 MB intermediate buffers that don't fit alongside the encoder in VRAM. The hybrid setup:
 
-## Validation
+```python
+vsess = ort.InferenceSession("vision_encoder_int4.onnx",
+                              providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
+tsess = ort.InferenceSession("text_encoder_int4.onnx",
+                              providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
+dsess = ort.InferenceSession("decoder_int4.onnx",
+                              providers=["CPUExecutionProvider"])   # ← decoder on CPU
+```
 
-Compared end-to-end against the original PyTorch `Sam3Model.forward()` on a microscope-style seed image with prompt `"seed"`:
+With this split, the int4 vision encoder runs in **~12 seconds on an RTX 3050** (vs. 166 seconds CPU-only) — a 14× speedup on the bottleneck. Total inference drops from ~3 minutes to ~30 seconds.
 
-| Metric | PyTorch SAM 3 | This ONNX export |
-|---|---|---|
-| Detections > 0.5 | 12 | 12 |
-| Top scores | 0.66–0.93 | 0.66–0.93 |
-| Box locations | clustered at seeds | clustered at seeds |
+On a server-class GPU (16+ GB VRAM) all three can run on GPU and inference is sub-second.
 
-Numerically equivalent within tracing noise. See `validate_sam3_e2e.py` to reproduce.
+### Prompt-iteration architecture
 
-## Quantization & next steps
+The decoder uses pre-computed vision features as inputs, so swapping the prompt while keeping the image only re-runs the cheap text + decoder path (`~15 s` end-to-end, vs ~30+ seconds for a fresh image). This makes interactive use viable.
 
-The fp32 export totals ~3.3 GB which is impractical for browser deployment. The natural follow-up is per-component quantization, mirroring what `onnx-community/sam3-tracker-ONNX` does for the tracker variant:
+## Validation
 
-```js
-dtype: {
-  vision_encoder: "q4",                    // ~470 MB
-  prompt_encoder_mask_decoder: "fp32",     // kept at full precision
-}
+The fp32 ONNX export was compared end-to-end against the original PyTorch `Sam3Model.forward()` on a microscope-style seed image with prompt `"seed"`. Detection count, top scores, and box locations matched bit-for-bit within tracing noise.
+
+The quantized variants (int8, int4) were then compared against the fp32 ONNX baseline — see the [precision selection table](#which-precision-should-i-use) for results.
+
+To reproduce on your own image:
+
+```powershell
+python validate_sam3_e2e.py path/to/image.png "seed"             # single variant
+python validate_all_variants.py path/to/image.png "seed"         # all 3 side-by-side
 ```
 
-Estimated sizes after quantization:
+## Quantization results
 
-| Component | fp32 | fp16 | q8 | q4 |
-|---|---|---|---|---|
-| vision_encoder | 1.84 GB | ~900 MB | ~470 MB | ~240 MB |
-| text_encoder | 1.35 GB | ~675 MB | ~340 MB | ~170 MB |
-| decoder | 96 MB | ~48 MB | ~25 MB | ~13 MB |
-| **Total** | **3.3 GB** | **~1.6 GB** | **~830 MB** | **~420 MB** |
+Three precision variants are published in this repo. Actual measured sizes:
+
+| Component | fp32 | int8 (dynamic) | int4 (MatMul block) |
+|---|---|---|---|
+| vision_encoder | 1.84 GB | **473 MB** (3.7×) | **285 MB** (6.5×) |
+| text_encoder | 1.35 GB | **340 MB** (4.0×) | **350 MB** (3.9×) |
+| decoder | 96 MB | **26 MB** (3.6×) | **19 MB** (4.9×) |
+| **Total** | **3.3 GB** | **839 MB** (4.0×) | **654 MB** (5.0×) |
+
+### Notes on each variant
+
+**int8 — `quantize_sam3_int8.py`** uses `onnxruntime.quantization.quantize_dynamic` with `QuantType.QInt8`. Per-tensor symmetric quantization of MatMul/Gemm weights. No calibration data needed. Outputs are bit-equivalent to fp32 in practice.
+
+**int4 — `quantize_sam3_int4.py`** uses `MatMulNBitsQuantizer` with `block_size=128, bits=4, is_symmetric=True`. Only quantizes MatMul ops; other ops stay fp32. This is the same scheme HuggingFace uses with `dtype: "q4"` in `transformers.js`, and what `onnx-community/sam3-tracker-ONNX` uses for its vision encoder.
+
+**Note on text encoder size**: int4 only saves a small amount over int8 for the text encoder because most of its size is in the **CLIP token embedding table** (a lookup matrix, not a MatMul). The block-wise int4 scheme targets MatMul weights specifically, so embeddings stay fp32 in both.
+
+### fp16 status
+
+A fully-working **fp16** variant is not currently published. `onnxconverter_common.float16.convert_float_to_float16` produces type-mismatch errors when applied to the decoder — PyTorch's traced `Cast` ops create incompatible type boundaries that the converter doesn't propagate cleanly through. Blocking `Cast`/`CastLike` from conversion fixes the vision encoder but the decoder has additional `Mul` ops with the same problem.
 
-I haven't published quantized versions yet. If you build them, PRs welcome.
+For practical purposes, **int8 dominates fp16 anyway**: int8 is half the size with equivalent quality. fp16 has no real use case here. See `quantize_sam3_fp16.py` for the current (partially working) attempt if you want to try fixing it.
 
 ## Known caveats and TODOs
 
 - **Fixed input size**: The vision encoder's positional embeddings are precomputed for 1008×1008 input. Other sizes will produce shape mismatches. Use the SAM 3 image processor to handle resizing/padding.
 - **Geometry prompts not supported**: This export covers the text-only path. SAM 3's optional box/point prompts are not wired in — would need a separate decoder variant.
 - **Dynamic batch is dynamic but untested**: The export uses symbolic batch dim but I've only validated batch=1. Higher batch sizes should work but no guarantees.
+- **fp16 export is broken**: See [fp16 status](#fp16-status). int8 is the recommended half-precision alternative.
+- **fp32 on 4 GB VRAM**: The fp32 vision encoder's Softmax allocates a ~1.7 GB intermediate buffer; on consumer GPUs this exhausts VRAM. Use int8/int4 for GPU, or fp32 on CPU only.
 - **Tracer warnings during export**: A few `TracerWarning: Converting a tensor to a Python boolean...` were emitted. These bake config flags (e.g. `is_causal`, attention backend selection) into the graph at export time. Fine for inference with the same model config, but means the ONNX isn't reusable across configs.
 - **No `transformers.js` integration**: At time of writing, `transformers.js` only has `Sam3TrackerModel` (point/box) but not `Sam3Model` (text-promptable image). Until that lands, use `onnxruntime-web` directly.
+- **Tracer optimized out unused inputs**: The decoder ONNX accepts only `fpn_position_encoding_2` (not `_0` or `_1`) because those weren't actually used in the traced forward path. Inspect `decoder_sess.get_inputs()` for the canonical input names.
 
 ## How this was built
 
 Three Python scripts, one per sub-module, each calling `torch.onnx.export` on a thin wrapper around the SAM 3 component:
 
 1. `export_sam3_vision.py` — wraps `Sam3VisionModel`, flattens `Sam3VisionEncoderOutput` into a tuple of tensors. Uses the default (dynamo) exporter.
-2. `export_sam3_text.py` — wraps `CLIPTextModelWithProjection` + the `text_projection` Linear layer.
-3. `export_sam3_decoder.py` — wraps `detr_encoder` + `detr_decoder` + `mask_decoder` + `dot_product_scoring` + box head as a single pipeline. **Uses `dynamo=False`** (legacy tracer) — the dynamo exporter crashes on the attention reshape-after-transpose pattern.
+2. `export_sam3_text.py` — wraps `CLIPTextModelWithProjection` + the `text_projection` Linear layer. Key gotcha: SAM 3's text config has `max_position_embeddings=32` (shorter than standard CLIP's 77).
+3. `export_sam3_decoder.py` — wraps `detr_encoder` + `detr_decoder` + `mask_decoder` + `dot_product_scoring` + box head as a single pipeline. **Uses `dynamo=False`** (legacy tracer) — the dynamo exporter crashes on the attention reshape-after-transpose pattern (`Cannot view a tensor with shape (1, 201, 8, 32) ... as a tensor with shape (s2, 201, 256)`).
+
+Quantization scripts (added after the initial export):
+
+- `quantize_sam3_int8.py` — `onnxruntime.quantization.quantize_dynamic` with `QInt8` weight type. Works on all three components out of the box.
+- `quantize_sam3_int4.py` — `MatMulNBitsQuantizer` from `onnxruntime.quantization.matmul_nbits_quantizer`. `block_size=128`, `bits=4`, symmetric. Only quantizes MatMul ops; other ops stay fp32.
+- `quantize_sam3_fp16.py` — uses `onnxconverter_common.float16.convert_float_to_float16` with `op_block_list=["Cast", "CastLike"]`. Fixes the vision encoder but the decoder still has type mismatches at `Mul` ops; partial fix only.
+
+See `validate_sam3_e2e.py` and `validate_all_variants.py` for the end-to-end pipeline tests.
+
+## Reproducing this export
+
+If you want to redo the export from scratch (e.g. to target a different opset, change the input size, or add geometry prompts), here's the full sequence.
+
+### Prerequisites
+
+You need **transformers v5** (which is in pre-release as of writing) and a build of `optimum-onnx` from source. The stable `optimum` releases require transformers v4 and don't know about SAM 3. Install from source with strict dependency control:
+
+```powershell
+py -3.11 -m venv .venv
+.\.venv\Scripts\Activate.ps1
+pip install --upgrade pip
+
+# Clone transformers from source — v5 is required for Sam3Model class
+git clone https://github.com/huggingface/transformers.git
+cd transformers
+pip install .[torch]
+cd ..
+
+# Upgrade huggingface_hub to v1+ (transformers v5 requires it; --no-deps avoids
+# pulling old optimum back down)
+pip install --upgrade --no-deps "huggingface_hub>=1.0"
+
+# Install optimum-onnx from source with --no-deps to preserve transformers v5
+pip install git+https://github.com/huggingface/optimum-onnx.git --no-deps
+
+# Remaining deps
+pip install onnx onnxruntime onnxscript onnx_ir pillow torchvision opencv-python --no-cache-dir
+```
+
+Verify the install:
+
+```powershell
+python -c "from transformers import Sam3Model, Sam3Processor; print('SAM3 OK')"
+python -c "from optimum.exporters.onnx import main_export; print('optimum OK')"
+```
+
+You'll also need a HuggingFace token to download `facebook/sam3`:
+
+```powershell
+$env:HF_TOKEN = "hf_xxx"
+```
+
+### Run the export
+
+```powershell
+# fp32 export (~15 min total on CPU)
+python export_sam3_vision.py          # ~5-10 min, produces vision_encoder.onnx + .data
+python export_sam3_text.py            # ~2 min, produces text_encoder.onnx + .data
+python export_sam3_decoder.py         # ~3 min, produces decoder.onnx
+
+# Validate against PyTorch (~3 min, only do once)
+python validate_sam3_e2e.py path/to/test-image.png "your prompt"
+```
+
+### Quantize (optional but recommended)
+
+```powershell
+python quantize_sam3_int8.py          # ~2 min, produces *_int8.onnx
+python quantize_sam3_int4.py          # ~3 min, produces *_int4.onnx
+python validate_all_variants.py path/to/test-image.png "your prompt"
+```
+
+### Hardware notes
 
-All three are in this repo. See `validate_sam3_e2e.py` for the end-to-end pipeline.
+- **CPU is sufficient** for both export and inference. GPU helps for fast iteration during inference but isn't needed for the export itself.
+- **At least 8 GB system RAM** is needed during the vision encoder export — the model is ~3.4 GB and the tracer holds intermediate state.
+- **`onnxruntime-gpu` requires CUDA 12.x runtime libraries**. If you have an NVIDIA driver supporting CUDA 12.x but no CUDA toolkit installed, `pip install nvidia-cudnn-cu12 nvidia-cublas-cu12 nvidia-cuda-runtime-cu12 nvidia-cufft-cu12` brings the needed DLLs.
 
 ## License