Sherlock-Comms/Ornith-1.5-122B-NVFP4-DFlash2-RTX-Spark

Ornith 1.5 122B NVFP4, DFlash2-ready, for RTX Spark laptops

This is ornith-ai/Ornith-1.5-122B-NVFP4 with the lm_head stored in BF16 instead of NVFP4. Nothing else is changed.

vLLM only runs the incoai/Ornith-1.5-122B-DFlash2 draft model if the target's lm_head is unquantized. The official NVFP4 checkpoint quantizes it, so vLLM rejects the draft. This checkpoint fixes that.

Built and tested for RTX Spark N1X laptops with 128 GB of memory. It also works with vLLM on other GPUs.

The BF16 head is dequantized from the official NVFP4 head, so its outputs match the official NVFP4 model.

Not affiliated with Ornith AI or Inco AI.

Changes from the official checkpoint

	Official NVFP4	This repo
lm_head.weight	NVFP4 packed, FP8 block scales, FP32 global scale	BF16 [248320, 3072]
Shards 1–7		identical
Shard 8		lm_head replaced, its two scale tensors removed, other tensors identical
config.json	lm_head quantized	lm_head in ignore
Size	77.6 GB	~78.7 GB

convert_lmhead_bf16.py in this repo does the conversion. It unpacks each FP4 (E2M1) value, applies the per-16-value FP8 scale and the global scale, and writes BF16.

Check: sampled rows match the original BF16 head from ornith-ai/Ornith-1.5-122B at cosine 0.995. The 9.6% relative error is the NVFP4 quantization error already in the official checkpoint.

Running it with vLLM 0.32.0 and DFlash2

vllm serve <this-repo> --served-model-name ornith-1.5-122b \
  --max-model-len 262144 --max-num-seqs 1 --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8 --kv-cache-memory-bytes 12884901888 \
  --enable-prefix-caching --prefix-cache-retention-interval 17344 \
  --speculative-config '{"method":"dflash","model":"incoai/Ornith-1.5-122B-DFlash2","num_speculative_tokens":7}' \
  --moe-backend marlin --no-async-scheduling \
  --mm-encoder-attn-backend TORCH_SDPA --limit-mm-per-prompt '{"image":4,"video":0}' \
  --mm-processor-cache-gb 1 --skip-mm-profiling \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --trust-remote-code

• Prefix caching needs --prefix-cache-retention-interval. With the default (0) it gets no hits on this model.
  • The DFlash2 draft's sliding-window layers mean cached prefixes can only be reused in 17,344-token steps (7 draft tokens; use 8640 for 6, 17472 for 8).
  • In a 5-turn test on a 61K-token conversation, each follow-up turn reused 52,032 tokens and started answering in 9 s instead of 38–41 s. All cached answers were correct.
  • The cache only helps when the start of the prompt stays the same between requests, as in an agent conversation. Another long prompt in between can push it out.
• Use --no-async-scheduling. Async scheduling doesn't work with DFlash.
• --moe-backend marlin keeps vLLM off the b12x MoE kernel, which hung the GPU on this chip.
• --skip-mm-profiling saves about 13 s at startup, because the KV cache size is set explicitly. Vision still works.
• TORCH_SDPA for the vision encoder, because flash-attn has no sm_121 build.
• KV cache: a full 262,144-token request needs 11.4 GiB of FP8 KV cache, hence the 12 GiB setting.
• The experts are W4A16 NVFP4, so vLLM uses the Marlin kernel. There is no native FP4 path for this checkpoint.

Results on an RTX Spark laptop

Hardware and software:

• ASUS ProArt P16: RTX Spark N1X (6,144-core GPU), 128 GB, GPU memory set to 96 GB.
• Windows 11 on Arm, driver 617.42.
• vLLM 0.32.0 in Docker under WSL 2.

	This checkpoint + DFlash2 (7 tokens), vLLM	Q4_K_M GGUF + MTP (1 token), llama.cpp on Windows
Decode	~50 tok/s (46–54 across runs, 23–93 per prompt)	36.6 tok/s
Prefill at 33K / 136K / 256K tokens	2,323 / 1,209 / 761 tok/s	609 / 504 / 434 tok/s
Needle-in-a-haystack at 33K / 136K / 256K	all found	all found
Tool-calling test (54 runs)	53/54, no invalid calls	54/54
Vision (reading a chart)	correct, 1.1 s	correct, 1.6 s
GPU memory	~94 GB, ~97 GB peak at startup	~84 GB

• GPU memory: the laptop's GPU memory has to be set to 96 GB, so 64 GB models can't run this. That leaves Windows about 30 GB.
• Draft length: 7 draft tokens averaged ~50 tok/s, 4 averaged ~47. About 3 of the 7 are accepted per step.

Licence

MIT, the same as ornith-ai/Ornith-1.5-122B.