keptn/ornith-1.5-35b-a3b-w4a8

Ornith 1.5 35B-A3B · w4a8 · for radiance

ornith-ai/Ornith-1.5-35B-A3B in one 19.7 GiB file for the radiance inference engine on AMD RDNA4. 4-bit experts make it fit on one 32 GB card, without host memory.

• Close to the original: KL divergence 0.064 to the bf16 original on wikitext-2; top token the same in 89.7%
• Fast: 229 tok/s BetterBench combined on one card, 306 tok/s on two, 395 tok/s on four
• Long context: 262,144 tokens; one card holds 4 sequences at full length
• Vision: images and video through the original bf16 vision tower
• Tools and thinking: OpenAI-style tool calls, thinking on or off per request

File	ornith-1.5-35b-a3b-w4a8.rad, 19.74 GiB
Needs	radiance 1.11.0 or newer (4-bit experts for the qwen35moe architecture); 1, 2 or 4 cards with 32 GB (tested on Radeon AI PRO R9700)
Weights	experts: 4-bit codebook, one FP8 scale per 64 weights, Hadamard rotation over 128, GPTQ; all other linears: FP8, one bf16 scale per 128×128 block
Activations	FP8
lm_head	int8, one bf16 scale per 128 columns
Speculation (MTP)	the model's own MTP head, 3 draft tokens
Vision tower	the original, bf16
Context	262,144 tokens
Version	1.0 (2026-10-20)

Quick start

With the compose files from the radiance repository (1.11.0 or newer): ornith-1.5-35b-a3b-w4a8.yaml (two cards) or ornith-1.5-35b-a3b-w4a8-tp1.yaml (one card). Set RADIANCE_MODELS, RADIANCE_API_KEY and the group ids in .env, see radiance's README.

Or directly (one card):

hf download keptn/ornith-1.5-35b-a3b-w4a8 ornith-1.5-35b-a3b-w4a8.rad --local-dir ~/models
docker run -d --init --name ornith --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e ROCR_VISIBLE_DEVICES=0 \
    stilldeadcode/radiance:1.11.0 --model /models/ornith-1.5-35b-a3b-w4a8.rad --host 0.0.0.0 --port 8000 \
    --kv-cache-dtype fp8 --max-model-len 262144 --max-num-seqs 8 \
    --max-num-batched-tokens 2048 --num-speculative-tokens 3 \
    --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5

	change
two cards	ROCR_VISIBLE_DEVICES=0,1, add --tp 2 --p2p on
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, add --tp 4 --p2p on
coding	--temp 0.6 --presence-penalty 0 (Ornith's setting for precise coding)
thinking	on by default; per request "chat_template_kwargs": {"enable_thinking": false} turns it off

Quality

Distance to the bf16 original. KL divergence on wikitext-2 (test), llama.cpp method: 100 chunks of 512 tokens, the second half of each chunk scored, 25,500 tokens. Reference: the bf16 original in HF transformers, fp32 math.

	mean KL	median KL	99th percentile	top token the same	perplexity (bf16: 7.94)
this file	0.064	0.027	0.60	89.7%	8.08
FP8 version (keptn/ornith-1.5-35b-a3b-fp8)	0.043	0.019	0.41	91.1%	8.01

Benchmarks, greedy. The thinking-off column is the one to compare with other cards.

benchmark	thinking off	thinking on
GSM8K (math, 5-shot)	88.9%	96.0%
IFEval (following instructions, strict)	76.9%	78.2%
LiveCodeBench v6 (175 tasks, pass@1)	30.9%	57.7%

• LiveCodeBench v6 by difficulty (off and on): easy 35/43 and 42/43, medium 14/52 and 39/52, hard 5/80 and 20/80.
• Thinking on uses output limits of 8,192 (GSM8K), 16,384 (IFEval) and 32,768 tokens (LiveCodeBench).
• At these limits 66 LiveCodeBench tasks, 8 IFEval prompts and 6 GSM8K questions stop before the answer. They count as wrong.

Speed

Radeon AI PRO R9700 at 215 W, radiance 1.8.1 + PR #144, fp8 KV, 3 draft tokens, thinking off. On 1.9.0 + PR #144 the same file gives 311.0 combined on two cards and the same KL divergence.

BetterBench 0.6.1, full run: 8 categories, 20 runs each after 3 warm-up runs, temperature 0.7, one stream. Then prefill and parallel streams. Decode tok/s (median):

	1 card	2 cards	4 cards
combined	228.8	305.7	395.4
chat	217.6	298.3	376.6
code	245.4	328.3	425.2
file_edit	279.5	371.7	488.1
json	280.7	382.5	487.0
math	262.6	362.8	443.2
prose	159.4	211.1	265.7
reasoning	198.8	263.2	344.3
summarization	215.0	284.3	372.8
prefill at 24k tokens	10,323	14,060	15,223
prefill at 47k tokens	8,562	11,964	13,951
4 streams, total	387	587	808
8 streams, total	527	787	1,062

All weights stay on the card, also on one card. The FP8 version needs host memory there and runs at 68.5 tok/s.

Vision

The original Ornith vision tower (bf16) reads images and video in chat requests (image_url, video_url).

benchmark (lmms-eval 0.7.3, LMMs-Eval-Lite, greedy, thinking off)	result
DocVQA val lite (500 questions)	93.5 (ANLS)
AI2D lite (500 questions)	46.6%

On AI2D the model often explains its choice instead of giving only the letter: 241 of 500 answers, against 136 for the FP8 version. The strict scorer counts these answers as wrong, also when the letter is correct. If only the first letter counts, the result is 71.6% (FP8 version: 71.2%).

Memory

KV cache, fp8 KV, --max-model-len 262144, from radiance's startup report:

	2 sequences	4 sequences	8 sequences
2 cards	524,320 tokens	1,048,640 tokens	2,097,280 tokens
4 cards	524,384 tokens	1,048,608 tokens	2,097,216 tokens

Tokens of context the KV pool holds in total, shared by all sequences. Where a value equals sequences × 262,144, every sequence can have the full context. One card holds 1,057,824 tokens with 8 sequences: 4 sequences at full length.

How it was made

1. Started from Ornith's FP8 release ornith-ai/Ornith-1.5-35B-A3B-FP8 (revision fab11c26).
2. Built a copy with 4-bit experts, rounded to nearest. It served only the next step.
3. Recorded the input statistics (Hessians) of every expert layer with this copy on one card: 1.26 million tokens of chat, documents, code and tool calls in Ornith's chat template.
4. Quantised the experts with GPTQ against these statistics, then three rounds of coordinate descent. All other weights came from the copy of step 2.

CALIB=calib rad-convert Ornith-1.5-35B-A3B-FP8 --tokenizer Ornith-1.5-35B-A3B-FP8/tokenizer.json \
    --recipe ornith-1.5-35b-a3b-w4a8.recipe --reuse ornith-w4a8-step2.rad -o ornith-1.5-35b-a3b-w4a8.rad

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs radiance 1.11.0 or newer. radiance 1.10.0 and older do not read 4-bit experts for this architecture.
• The experts of the MTP layer have no recorded statistics. They are rounded to nearest.
• With thinking on, hard tasks can need more than 32k output tokens.

Credits

• Model: the Ornith team (github.com/ornith-ai/Ornith-1), built on Qwen3.5 (Apache 2.0).
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance). The rule for the experts comes from radiance's recipe qwen4exp-w4nl64-i8-hc8m.recipe.
• Test sets: GSM8K, IFEval, LiveCodeBench, LMMs-Eval-Lite, wikitext-2.

License

MIT, the license of the base model. See LICENSE (from github.com/ornith-ai/Ornith-1).