Instructions to use nvidia/Nemotron-3.5-Parse with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/Nemotron-3.5-Parse with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="nvidia/Nemotron-3.5-Parse", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("nvidia/Nemotron-3.5-Parse", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/Nemotron-3.5-Parse with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/Nemotron-3.5-Parse"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Nemotron-3.5-Parse",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/Nemotron-3.5-Parse

SGLang

How to use nvidia/Nemotron-3.5-Parse with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/Nemotron-3.5-Parse" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Nemotron-3.5-Parse",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/Nemotron-3.5-Parse" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Nemotron-3.5-Parse",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use nvidia/Nemotron-3.5-Parse with Docker Model Runner:
```
docker model run hf.co/nvidia/Nemotron-3.5-Parse
```

Nemotron 3.5 Parse on AMD (RX 7800 XT / RDNA3) with vLLM 0.33 ROCm: working flags and speed

by evanccjones - opened 7 days ago

Discussion

evanccjones

7 days ago

vLLM 0.33 serves Nemotron 3.5 Parse natively on a consumer RDNA3 card, with no plugin. Sharing the settings and speed in case they save someone time. These are speed and configuration results only. I'm still checking accuracy, so please compare on your own documents.

Setup: RX 7800 XT (gfx1101, 16 GB), Ubuntu 26.04.1, kernel 7.0.0-42 with the in-kernel amdgpu driver (no ROCm installed on the host). Docker image vllm/vllm-openai-rocm:v0.33.0 (vLLM 0.33.0, PyTorch 2.13.0+rocm10.0.0, HIP 7.15), native nemotron_parse3_5 support.

docker run -d --name vllm --device /dev/kfd --device /dev/dri --group-add video --ipc=host
--security-opt seccomp=unconfined -p 127.0.0.1:8000:8000 -v :/models:ro
-e HIP_VISIBLE_DEVICES=0 -e TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
--entrypoint python3 vllm/vllm-openai-rocm:v0.33.0 -m vllm.entrypoints.openai.api_server
--model /models/Nemotron-3.5-Parse --served-model-name nemotron-parse
--trust-remote-code --dtype bfloat16 --max-model-len 16384 --max-num-seqs 64
--max-num-batched-tokens 8192 --gpu-memory-utilization 0.90 --no-enable-prefix-caching
--limit-mm-per-prompt '{"image":1}'
--chat-template /models/Nemotron-3.5-Parse/chat_template.jinja \ --speculative-config '{"method":"mtp","num

This is the 1280 px server. The 1280 px resu the client (long side 1280 px) beforesending; the server does not downscale. For full-size pages I used a different server: drop the --speculative-config line (MTP was slower at full size) and set - My full-size row below ran with--enforce-eager; a later paired run on 0.33 found no measurable difference between eager and GPU graphs at full size (23.7 vs 24.9 pages/min on 120 pages).

What mattered on RDNA3:

FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE lets the image encoder use Triton flash attention.
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 t enables PyTorch's efficient attention onRDNA3); I haven't tested vLLM without it.
PYTORCH_HIP_ALLOC_CONF=expandable_segments.33 (normal start, same KV cache size). On0.32 with NVIDIA's plugin it coincided with a misleading out-of-memory, so if you are on an older stack, try without it.
--max-num-batched-tokens: at 1280 px, 4096, 8192 and 16,384 all ran on 0.33, with 8192 fastest (within noise). At
full size I used 4096; 16,384 ran out of mg on 0.32 and I haven't retried it on 0.33.
MTP (one step) helps at reduced image size and slowed things down at full size.
GPU graphs on 0.33: no measurable differenter at 1280 px (single paired runs, withinnoise). On 0.32 with NVIDIA's plugin they were 29% slower at full size.

Speed (364 mixed pages: text, tables, figures, scans; requests at temperature 0):

┌─────────────────────────────────────────────────┬──────────────────────────────────────┬───────────────────────┐
│ Image size s │ Pages/min │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼───────────────────────┤
│ Full (3,500 px long side, ~2,700 prompt current, budget │ 28.8 │
│ tokens/page) │ 4096 │ │
├───────────────────────────────────────────────────────────┼───────────────────────┤
│ 1280 px long side (1,200 prompt tokens/page) │ MTP, graphs, 64 concurrent, budget │ 59.0 │
│ │ │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼───────────────────────┤
│ 1280 px, sustained 2 hours │ 62–69, no downward │
│ │ │ trend │
└───────────────────────────────────────────────────────────┴───────────────────────┘

Caveats:

One card, one run per configuration; differences described as "within noise" are single paired runs.
At 1280 px, 1–2% of pages hit the output lr pass, and which pages varies between runs. Catch these rather than trusting them.
Downscaling changes the output's layout ancy at 1280 px on your own material before adopting it.
A 200 W power cap (default 220 W) made no

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment