doth4580 commited on
Commit
8067fca
·
verified ·
1 Parent(s): 3e7112f

Model card: point serving at the 2x DGX Spark recipe

Browse files
Files changed (1) hide show
  1. README.md +64 -102
README.md CHANGED
@@ -10,134 +10,96 @@ tags:
10
  - vllm
11
  - moe
12
  - long-context
 
13
  - naive-n0.5-flash
14
  ---
15
 
16
- # Naive-N0.5-Flash — EXL3 3.5 bpw (vLLM)
17
 
18
  > **Original model:** [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) by the
19
- > NaiveAI Team, released under the [MIT License](https://huggingface.co/NaiveAI/Naive-N0.5-Flash/blob/main/LICENSE).
20
  > This artifact is quantized from the public FP8 release
21
  > [NaiveAI/Naive-N0.5-Flash-FP8](https://huggingface.co/NaiveAI/Naive-N0.5-Flash-FP8).
22
  > It contains **modified weights** (EXL3 trellis quantization); the original model is © its authors.
23
- > The tokenizer, chat template, and model code shipped here are unchanged from the original.
24
 
25
- An **EXL3 (ExLlamaV3 trellis) 3.5 bpw** quantization of **Naive-N0.5-Flash**, a 309B-parameter
26
- mixture-of-experts model with 15.5B active parameters and a native 1M-token context window. The pack
27
- is **~137 GB on disk versus ~315 GB for the FP8 original**, which is what makes this model fit on a
28
- pair of 128 GB unified-memory machines.
29
 
30
- The quantization was produced with the [exllamav3](https://github.com/turboderp-org/exllamav3)
31
- tooling, adapted for this architecture, and was run on 8×H100 for speed of quantization.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
 
33
  ## Specifications
34
 
35
  | | |
36
  |---|---|
37
  | Base model | [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) (FP8 release as the direct source) |
38
- | Architecture | Mixture-of-Experts, 309B total / 15.5B active parameters, 48 layers |
39
- | Attention | hybrid Sliding-Window + DeepSeek Sparse Attention (no full-attention layers): 39 SWA + 9 DSA layers, SWA window 128, DSA top-2048 token selection, GQA with 4 KV groups |
40
- | MoE | 256 routed experts, 8 per token, expert intermediate size 2048 |
41
- | Hidden / heads | hidden 4096, 64 attention heads, 4 KV heads, head_dim 192 |
42
- | Context | 1,048,576 tokens (native), vocabulary 152,576 |
43
- | Quantization | EXL3 trellis, `mul1` codebook — 3.5 bpw average, 6-bit `lm_head`, unquantized embeddings |
44
- | ExLlamaV3 tooling | 1.5.1.post1 |
45
- | Format | `exl3` (`--quantization exl3`) |
46
- | On-disk size | ~137 GB across 24 shards |
47
  | License | MIT (inherited from the original model) |
48
 
49
- ### What is quantized
50
-
51
- - **Routed experts** (`mlp.experts.*`) — EXL3 trellis, 3.5 bpw.
52
- - **Non-routed linears** — 3 bpw: attention `q/k/v/o_proj`, the DSA indexer linears
53
- (`indexer.wq`, `indexer.wk`, `indexer.weights_proj`), and the dense `mlp.gate/up/down_proj`.
54
- `lm_head` is kept at 6 bpw.
55
- - **Unquantized (bfloat16)** — token embeddings and all norms.
56
- - Calibration: 250 rows × 2,048 columns, output scales always stored, `activation_scheme: dynamic`
57
- carried over from the FP8 source.
58
-
59
  ## Files
60
 
61
  | File | Notes |
62
  |---|---|
63
- | `model-00001..00024-of-00024.safetensors` | the quantized weights, ~137 GB total |
64
- | `model.safetensors.index.json` | tensor → shard index for the vLLM pack |
65
- | `quantization_config.json` | full EXL3 metadata (per-tensor trellis and scale information) |
66
- | `config.json` | model config carrying the EXL3 `quantization_config`, incl. the `non_routed_exl3` module map |
67
- | `config.json.native`, `model.safetensors.index.json.native` | the equivalent **native ExLlamaV3** pack metadata (see below) |
68
  | `modeling_naive_n05_flash.py`, `configuration_naive_n05_flash.py` | model code, loaded via `trust_remote_code` |
69
- | `chat_template.jinja`, `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `special_tokens_map.json`, `vocab.README.md` | tokenizer and chat template, unchanged from the original |
70
- | `generation_config.json` | recommended sampling (temperature 1.0, top-p 0.95) |
71
-
72
- The same weights serve two pack layouts: **vLLM** reads `config.json` (the `non_routed_exl3` map),
73
- while **ExLlamaV3** users can use the `.native` pair, which carries the original pack metadata for
74
- this artifact.
75
-
76
- ## Serving with vLLM
77
-
78
- EXL3 weights are **not** loadable by stock vLLM or `transformers`: this pack needs the
79
- [`vllm-exl3`](https://github.com/vcruz305/vllm-exl3) out-of-tree plugin plus an ExLlamaV3 build,
80
- validated here with **vLLM 0.29.0** and **exllamav3 1.5.1.post1**.
81
-
82
- The configuration below was validated on **2× DGX Spark (GB10)** across two nodes, TP=2 + EP=2 over
83
- Ray. Speculative decoding uses the DSpark drafter
84
- [NaiveAI/Naive-N0.5-Flash-FP8-Draft](https://huggingface.co/NaiveAI/Naive-N0.5-Flash-FP8-Draft):
85
-
86
- ```bash
87
- vllm serve /path/to/Naive-N0.5-Flash-exl3-3.5bpw-vllm \
88
- --quantization exl3 \
89
- --tensor-parallel-size 2 \
90
- --enable-expert-parallel \
91
- --distributed-executor-backend ray \
92
- --host 0.0.0.0 --port 8000 \
93
- --max-model-len 131072 \
94
- --max-num-seqs 1 \
95
- --gpu-memory-utilization 0.80 \
96
- --dtype bfloat16 \
97
- --moe-backend triton \
98
- --no-enable-flashinfer-autotune \
99
- --kernel-config '{"enable_cutedsl_warmup": false, "enable_jit_warmup": false}' \
100
- --speculative-config '{"method":"dspark","model":"/path/to/Naive-N0.5-Flash-FP8-Draft","num_speculative_tokens":3}'
101
- ```
102
-
103
- Drop `--speculative-config` to serve without the drafter. Startup takes several minutes on this
104
- hardware: loading is I/O-bound and the first run also compiles kernels.
105
-
106
- ## Measured (preliminary)
107
-
108
- On 2× DGX Spark (GB10), concurrency 1, greedy decoding, 2,048-token context, mixed chat prompts.
109
- **These numbers come from an early, still-developing serving port and will change** — treat them as
110
- a starting reference, not as a tuned result:
111
-
112
- | Configuration | Decode |
113
- |---|---:|
114
- | No speculation, eager | ~21.5 tok/s |
115
- | No speculation, CUDA graphs | ~23.5 tok/s |
116
- | CUDA graphs + DSpark, 3 speculative tokens | **~24.4 tok/s** |
117
-
118
- Prefill runs at roughly 950 tok/s at 2K tokens (about 2.1 s TTFT). Long context is not yet
119
- characterized. Where a decode step goes (no speculation, eager): the routed-expert MoE kernel is
120
- ~55% of the step, NCCL all-reduce ~17%, and the dense EXL3 GEMV ~14%.
121
 
122
- ## Notes and caveats
123
 
124
- - **Sparse DSA attention is not implemented in this port yet.** The 9 DSA layers currently run
125
- *dense* attention, which is equivalent to the reference model only up to `index_top_k = 2048`
126
- tokens. Beyond that the served model attends to every token instead of the reference top-2,048
127
- selection, so **long-context behaviour is not bit-comparable to the original model**.
128
- - **EXL3 at 3.5 bpw falls back to generic kernels** in existing ExLlamaV3 MoE paths — a
129
- fractional-bitrate-specific MoE kernel is still an open performance item.
130
- - `v_proj` is stored per-head padded and `o_proj` has the 0.707 value scale baked in; these are
131
- properties of the pack, not errors.
132
- - Recommended sampling, per the original model: `temperature=1.0`, `top_p=0.95`.
133
 
134
  ## License and credits
135
 
136
  - **Original model:** [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) —
137
- © the NaiveAI Team, [MIT License](https://huggingface.co/NaiveAI/Naive-N0.5-Flash/blob/main/LICENSE).
138
- A copy of the license is included in this repository (`LICENSE`). The model builds on Xiaomi's
139
- [MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Base) base model and on DeepSeek Sparse
140
- Attention; see the original model card for the full architecture and evaluation details.
141
- - **This artifact:** EXL3 3.5 bpw quantization of the public FP8 release, produced with the
142
- [exllamav3](https://github.com/turboderp-org/exllamav3) tooling. Weights are modified relative to
143
- the original; tokenizer and chat template are unchanged.
 
 
 
 
10
  - vllm
11
  - moe
12
  - long-context
13
+ - dgx-spark
14
  - naive-n0.5-flash
15
  ---
16
 
17
+ # Naive-N0.5-Flash — EXL3 3.5 bpw
18
 
19
  > **Original model:** [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) by the
20
+ > NaiveAI Team, [MIT License](https://huggingface.co/NaiveAI/Naive-N0.5-Flash/blob/main/LICENSE).
21
  > This artifact is quantized from the public FP8 release
22
  > [NaiveAI/Naive-N0.5-Flash-FP8](https://huggingface.co/NaiveAI/Naive-N0.5-Flash-FP8).
23
  > It contains **modified weights** (EXL3 trellis quantization); the original model is © its authors.
24
+ > Tokenizer and chat template are unchanged from the original.
25
 
26
+ An **EXL3 (ExLlamaV3 trellis) 3.5 bpw** quantization of **Naive-N0.5-Flash** — 309B parameters,
27
+ 15.5B active, native 1M-token context. The pack is **~137 GB (~128 GiB) against ~315 GB for the FP8
28
+ original**, which is what lets it fit across **two NVIDIA DGX Spark (GB10)** machines.
 
29
 
30
+ Quantized with the [exllamav3](https://github.com/turboderp-org/exllamav3) tooling, adapted for this
31
+ architecture (the patch that adds it ships with the recipe), run on **8×H100** for speed.
32
+
33
+ ## Serving
34
+
35
+ Serve it with the recipe:
36
+ **[sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark](https://github.com/sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark)**.
37
+
38
+ EXL3 weights do not load in stock vLLM or `transformers`, and this is a uniform fractional-bitrate
39
+ (3.5 bpw) pack, so it needs the exact pinned stack the recipe installs on both nodes: **vLLM 0.29.0**
40
+ + [vcruz305/vllm-exl3](https://github.com/vcruz305/vllm-exl3) + a
41
+ [vcruz305/exllamav3](https://github.com/vcruz305/exllamav3) build with fractional-K kernels + the
42
+ recipe's `vllm_naive_n05` model plugin (attention TP2, experts EP2, Ray over RoCE).
43
+
44
+ ```bash
45
+ cp config/cluster.env.example config/cluster.env && $EDITOR config/cluster.env
46
+ bash scripts/install.sh # on BOTH nodes
47
+ bash scripts/download.sh # on BOTH nodes — fetches this pack + the DSpark drafter
48
+ bash scripts/start.sh # Ray head + worker, then the OpenAI-compatible API on port 8000
49
+ ```
50
+
51
+ `download.sh` pulls this repository and
52
+ [NaiveAI/Naive-N0.5-Flash-FP8-Draft](https://huggingface.co/NaiveAI/Naive-N0.5-Flash-FP8-Draft) for
53
+ speculative decoding. The server is OpenAI-compatible at `http://<head-ip>:8000/v1`, model id
54
+ `naive-n0.5-flash`; loading takes about 7.5 minutes. Reasoning (`<think>`) and tool calls parse
55
+ correctly with the recipe's chat template and parsers — see its README for the details.
56
 
57
  ## Specifications
58
 
59
  | | |
60
  |---|---|
61
  | Base model | [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) (FP8 release as the direct source) |
62
+ | Architecture | MoE, 309B total / 15.5B active, 48 layers, hidden 4096, 64 heads / 4 KV heads, head_dim 192 |
63
+ | Attention | hybrid 39 SWA layers (window 128) + 9 DSA layers (top-2048 selection), GQA4, no full attention |
64
+ | MoE | 256 routed experts, 8 per token, expert intermediate 2048 |
65
+ | Context / vocab | 1,048,576 tokens / 152,576 |
66
+ | Quantization | EXL3 trellis, `mul1` codebook — uniform 3.5 bpw, 6-bit `lm_head`, embeddings unquantized, calibration 250×2048 |
67
+ | Format / size | `exl3` (`--quantization exl3`) — ~137 GB (~128 GiB), 24 shards |
 
 
 
68
  | License | MIT (inherited from the original model) |
69
 
 
 
 
 
 
 
 
 
 
 
70
  ## Files
71
 
72
  | File | Notes |
73
  |---|---|
74
+ | `model-00001..00024-of-00024.safetensors` | the quantized weights |
75
+ | `model.safetensors.index.json`, `quantization_config.json` | tensor index and the full EXL3 metadata |
76
+ | `config.json` | carries the EXL3 `quantization_config` incl. the `non_routed_exl3` map (the vLLM-rewritten pack config) |
77
+ | `config.json.native`, `model.safetensors.index.json.native` | the equivalent native **ExLlamaV3** pack metadata |
 
78
  | `modeling_naive_n05_flash.py`, `configuration_naive_n05_flash.py` | model code, loaded via `trust_remote_code` |
79
+ | tokenizer + `chat_template.jinja`, `generation_config.json` | unchanged from the original |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
 
81
+ ## Notes
82
 
83
+ - **Beyond 2,048 tokens the DSA layers are approximate.** The model's 9 DSA layers attend to the
84
+ top-2,048 tokens picked by a learned indexer; the recipe runs them as *dense* attention, which is
85
+ exact up to 2,048 tokens but past that sees more than the reference model does. Long-context
86
+ answers may differ from the original; the 1M native context needs a sparse attention backend that
87
+ is not implemented yet.
88
+ - **This is an early recipe, not a tuned one.** Measured decode on 2× DGX Spark: ~34.9 tok/s on pure
89
+ code, ~23.9 mixed, ~19.5 reasoning (~23.5 without speculation). Performance tables, the profile,
90
+ and the limitations are in the recipe's README.
91
+ - **One request at a time**, bf16 KV only, and keep the Sparks otherwise idle (`GPU_MEM_UTIL=0.80`).
92
 
93
  ## License and credits
94
 
95
  - **Original model:** [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) —
96
+ © the NaiveAI Team, [MIT](https://huggingface.co/NaiveAI/Naive-N0.5-Flash/blob/main/LICENSE); a copy
97
+ ships here as `LICENSE`. Built on Xiaomi's
98
+ [MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Base) and DeepSeek Sparse Attention.
99
+ - **This artifact:** EXL3 3.5 bpw quantization produced with
100
+ [exllamav3](https://github.com/turboderp-org/exllamav3) via the
101
+ [vcruz305](https://github.com/vcruz305/exllamav3) fork. Weights are modified; tokenizer and chat
102
+ template are not.
103
+ - **Recipe:** [sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark](https://github.com/sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark)
104
+ (MIT). Note that [`vllm-exl3`](https://github.com/vcruz305/vllm-exl3) is **AGPL-3.0** — relevant if
105
+ you expose the server to others.