keptn/ornith-1.5-35b-a3b-fp8

Ornith 1.5 35B-A3B · FP8 · for radiance

ornith-ai/Ornith-1.5-35B-A3B in one 34.6 GiB file for the radiance inference engine on AMD RDNA4. A mixture-of-experts reasoning model with 3B active parameters, tuned for coding and agent work.

File	ornith-1.5-35b-a3b-fp8.rad, 34.63 GiB
Needs	radiance 1.8.1 or newer; 2 or 4 cards with 32 GB, or 1 card with 16 GiB of pinned host memory (tested on Radeon AI PRO R9700)
Format	weights FP8 (E4M3), one bf16 scale per 128×128 block; activations FP8; lm_head FP8, 128×128 blocks
Speculation	the model's own MTP head, 3 draft tokens
Inputs	text · image
Context	262,144 tokens
Version	1.1 (2026-10-20)

Speed

Radeon AI PRO R9700 at 215 W, radiance 1.8.1, fp8 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

	1 card ¹	2 cards	4 cards
decode, one stream	68.5	276.5	364.0
prefill at 24k tokens	9,073	13,777	14,750
4 streams, total	110	513	728

¹ Experts in 16 GiB of pinned host memory.

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,305 tokens).

	mean KL	99th percentile KL	top token the same	perplexity ratio
this file	0.0104	0.113	97.0%	×1.001
keptn/ornith-1.5-35b-a3b-w4a8	0.0165	0.182	96.2%	×1.003

Quick start

With the compose files from the radiance repository (1.9.0 or newer): ornith-1.5-35b-a3b.yaml.

Or directly (two cards):

hf download keptn/ornith-1.5-35b-a3b-fp8 ornith-1.5-35b-a3b-fp8.rad --local-dir ~/models
docker run -d --init --name ornith --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e ROCR_VISIBLE_DEVICES=0,1 \
    stilldeadcode/radiance:1.8.1 --model /models/ornith-1.5-35b-a3b-fp8.rad --host 0.0.0.0 --port 8000 \
    --tp 2 --p2p on --kv-cache-dtype fp8 --max-model-len 262144 --max-num-seqs 8 \
    --max-num-batched-tokens 2048 --num-speculative-tokens 3 \
    --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5

	change
one card	ROCR_VISIBLE_DEVICES=0, drop --tp 2 --p2p on, add --placement expert_tiered --host-pool-mib 16384
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, --tp 4
coding	--temp 0.6 --presence-penalty 0 (Ornith's setting for precise coding)
thinking	on by default; per request "chat_template_kwargs": {"enable_thinking": false} turns it off

Memory and context

KV pool at --max-model-len 262144, fp8 KV, radiance 1.8.1, from radiance's startup report. Tokens of context in total, shared by all sequences:

	2 sequences	4 sequences	8 sequences
1 card	524,320	1,048,640	1,149,520
2 cards	524,320	1,048,640	2,097,280
4 cards	524,384	1,048,608	2,097,216

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance 1.15.1 (release image). There is one measurement for each vision or audio encoder, made with ornith-1.5-35b-a3b-fp8-block.rad.

input	cases	same answer	mean KL	top token the same
images (DocVQA and AI2D, fixed selection)	20	10/20	0.210	181/199

How it was made

1. Start from Ornith's FP8 release ornith-ai/Ornith-1.5-35B-A3B-FP8 (revision fab11c26). It has one FP8 scale per row.
2. Requantise every linear to FP8 with one scale per 128×128 block, round to nearest. radiance's expert kernels read only this form. The delta-net and MTP layers are bf16 in the release and get the same form.
3. Add a 2-bit copy of the lm_head for the draft rounds, as radiance's recipe for Qwen3.6-35B-A3B does.

rad-convert Ornith-1.5-35B-A3B-FP8 --tokenizer Ornith-1.5-35B-A3B-FP8/tokenizer.json \
    --recipe ornith-1.5-35b-a3b-fp8.recipe -o ornith-1.5-35b-a3b-fp8.rad

Calibration: no calibration data.

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

• radiance runs on AMD RDNA4 cards.
• The FP8 weights of the release are rounded a second time, into blocks.
• With thinking on, hard tasks can need more than 32k output tokens.
• On one card the experts come from host memory. This makes one card slow.

Credits

• Model: the Ornith team (github.com/ornith-ai/Ornith-1), built on Qwen3.5 (Apache 2.0).
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
• Test sets: MT-Bench, DocVQA, AI2D.

License

MIT, the license of the base model. See LICENSE (from github.com/ornith-ai/Ornith-1).