Trellis
Safetensors
naive_n05_flash
exl3
exllamav3
3.5bpw
vllm
Mixture of Experts
long-context
dgx-spark
naive-n0.5-flash
custom_code
3-bit
Instructions to use doth4580/Naive-N0.5-Flash-EXL3-3.5bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Trellis
How to use doth4580/Naive-N0.5-Flash-EXL3-3.5bpw with Trellis:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Model card: point serving at the 2x DGX Spark recipe
Browse files
README.md
CHANGED
|
@@ -10,134 +10,96 @@ tags:
|
|
| 10 |
- vllm
|
| 11 |
- moe
|
| 12 |
- long-context
|
|
|
|
| 13 |
- naive-n0.5-flash
|
| 14 |
---
|
| 15 |
|
| 16 |
-
# Naive-N0.5-Flash — EXL3 3.5 bpw
|
| 17 |
|
| 18 |
> **Original model:** [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) by the
|
| 19 |
-
> NaiveAI Team,
|
| 20 |
> This artifact is quantized from the public FP8 release
|
| 21 |
> [NaiveAI/Naive-N0.5-Flash-FP8](https://huggingface.co/NaiveAI/Naive-N0.5-Flash-FP8).
|
| 22 |
> It contains **modified weights** (EXL3 trellis quantization); the original model is © its authors.
|
| 23 |
-
>
|
| 24 |
|
| 25 |
-
An **EXL3 (ExLlamaV3 trellis) 3.5 bpw** quantization of **Naive-N0.5-Flash**
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
pair of 128 GB unified-memory machines.
|
| 29 |
|
| 30 |
-
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
## Specifications
|
| 34 |
|
| 35 |
| | |
|
| 36 |
|---|---|
|
| 37 |
| Base model | [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) (FP8 release as the direct source) |
|
| 38 |
-
| Architecture |
|
| 39 |
-
| Attention | hybrid
|
| 40 |
-
| MoE | 256 routed experts, 8 per token, expert intermediate
|
| 41 |
-
|
|
| 42 |
-
|
|
| 43 |
-
|
|
| 44 |
-
| ExLlamaV3 tooling | 1.5.1.post1 |
|
| 45 |
-
| Format | `exl3` (`--quantization exl3`) |
|
| 46 |
-
| On-disk size | ~137 GB across 24 shards |
|
| 47 |
| License | MIT (inherited from the original model) |
|
| 48 |
|
| 49 |
-
### What is quantized
|
| 50 |
-
|
| 51 |
-
- **Routed experts** (`mlp.experts.*`) — EXL3 trellis, 3.5 bpw.
|
| 52 |
-
- **Non-routed linears** — 3 bpw: attention `q/k/v/o_proj`, the DSA indexer linears
|
| 53 |
-
(`indexer.wq`, `indexer.wk`, `indexer.weights_proj`), and the dense `mlp.gate/up/down_proj`.
|
| 54 |
-
`lm_head` is kept at 6 bpw.
|
| 55 |
-
- **Unquantized (bfloat16)** — token embeddings and all norms.
|
| 56 |
-
- Calibration: 250 rows × 2,048 columns, output scales always stored, `activation_scheme: dynamic`
|
| 57 |
-
carried over from the FP8 source.
|
| 58 |
-
|
| 59 |
## Files
|
| 60 |
|
| 61 |
| File | Notes |
|
| 62 |
|---|---|
|
| 63 |
-
| `model-00001..00024-of-00024.safetensors` | the quantized weights
|
| 64 |
-
| `model.safetensors.index.json` | tensor
|
| 65 |
-
| `
|
| 66 |
-
| `config.json`
|
| 67 |
-
| `config.json.native`, `model.safetensors.index.json.native` | the equivalent **native ExLlamaV3** pack metadata (see below) |
|
| 68 |
| `modeling_naive_n05_flash.py`, `configuration_naive_n05_flash.py` | model code, loaded via `trust_remote_code` |
|
| 69 |
-
|
|
| 70 |
-
| `generation_config.json` | recommended sampling (temperature 1.0, top-p 0.95) |
|
| 71 |
-
|
| 72 |
-
The same weights serve two pack layouts: **vLLM** reads `config.json` (the `non_routed_exl3` map),
|
| 73 |
-
while **ExLlamaV3** users can use the `.native` pair, which carries the original pack metadata for
|
| 74 |
-
this artifact.
|
| 75 |
-
|
| 76 |
-
## Serving with vLLM
|
| 77 |
-
|
| 78 |
-
EXL3 weights are **not** loadable by stock vLLM or `transformers`: this pack needs the
|
| 79 |
-
[`vllm-exl3`](https://github.com/vcruz305/vllm-exl3) out-of-tree plugin plus an ExLlamaV3 build,
|
| 80 |
-
validated here with **vLLM 0.29.0** and **exllamav3 1.5.1.post1**.
|
| 81 |
-
|
| 82 |
-
The configuration below was validated on **2× DGX Spark (GB10)** across two nodes, TP=2 + EP=2 over
|
| 83 |
-
Ray. Speculative decoding uses the DSpark drafter
|
| 84 |
-
[NaiveAI/Naive-N0.5-Flash-FP8-Draft](https://huggingface.co/NaiveAI/Naive-N0.5-Flash-FP8-Draft):
|
| 85 |
-
|
| 86 |
-
```bash
|
| 87 |
-
vllm serve /path/to/Naive-N0.5-Flash-exl3-3.5bpw-vllm \
|
| 88 |
-
--quantization exl3 \
|
| 89 |
-
--tensor-parallel-size 2 \
|
| 90 |
-
--enable-expert-parallel \
|
| 91 |
-
--distributed-executor-backend ray \
|
| 92 |
-
--host 0.0.0.0 --port 8000 \
|
| 93 |
-
--max-model-len 131072 \
|
| 94 |
-
--max-num-seqs 1 \
|
| 95 |
-
--gpu-memory-utilization 0.80 \
|
| 96 |
-
--dtype bfloat16 \
|
| 97 |
-
--moe-backend triton \
|
| 98 |
-
--no-enable-flashinfer-autotune \
|
| 99 |
-
--kernel-config '{"enable_cutedsl_warmup": false, "enable_jit_warmup": false}' \
|
| 100 |
-
--speculative-config '{"method":"dspark","model":"/path/to/Naive-N0.5-Flash-FP8-Draft","num_speculative_tokens":3}'
|
| 101 |
-
```
|
| 102 |
-
|
| 103 |
-
Drop `--speculative-config` to serve without the drafter. Startup takes several minutes on this
|
| 104 |
-
hardware: loading is I/O-bound and the first run also compiles kernels.
|
| 105 |
-
|
| 106 |
-
## Measured (preliminary)
|
| 107 |
-
|
| 108 |
-
On 2× DGX Spark (GB10), concurrency 1, greedy decoding, 2,048-token context, mixed chat prompts.
|
| 109 |
-
**These numbers come from an early, still-developing serving port and will change** — treat them as
|
| 110 |
-
a starting reference, not as a tuned result:
|
| 111 |
-
|
| 112 |
-
| Configuration | Decode |
|
| 113 |
-
|---|---:|
|
| 114 |
-
| No speculation, eager | ~21.5 tok/s |
|
| 115 |
-
| No speculation, CUDA graphs | ~23.5 tok/s |
|
| 116 |
-
| CUDA graphs + DSpark, 3 speculative tokens | **~24.4 tok/s** |
|
| 117 |
-
|
| 118 |
-
Prefill runs at roughly 950 tok/s at 2K tokens (about 2.1 s TTFT). Long context is not yet
|
| 119 |
-
characterized. Where a decode step goes (no speculation, eager): the routed-expert MoE kernel is
|
| 120 |
-
~55% of the step, NCCL all-reduce ~17%, and the dense EXL3 GEMV ~14%.
|
| 121 |
|
| 122 |
-
## Notes
|
| 123 |
|
| 124 |
-
- **
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
-
|
| 133 |
|
| 134 |
## License and credits
|
| 135 |
|
| 136 |
- **Original model:** [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) —
|
| 137 |
-
© the NaiveAI Team, [MIT
|
| 138 |
-
|
| 139 |
-
[MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Base)
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
[
|
| 143 |
-
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
- vllm
|
| 11 |
- moe
|
| 12 |
- long-context
|
| 13 |
+
- dgx-spark
|
| 14 |
- naive-n0.5-flash
|
| 15 |
---
|
| 16 |
|
| 17 |
+
# Naive-N0.5-Flash — EXL3 3.5 bpw
|
| 18 |
|
| 19 |
> **Original model:** [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) by the
|
| 20 |
+
> NaiveAI Team, [MIT License](https://huggingface.co/NaiveAI/Naive-N0.5-Flash/blob/main/LICENSE).
|
| 21 |
> This artifact is quantized from the public FP8 release
|
| 22 |
> [NaiveAI/Naive-N0.5-Flash-FP8](https://huggingface.co/NaiveAI/Naive-N0.5-Flash-FP8).
|
| 23 |
> It contains **modified weights** (EXL3 trellis quantization); the original model is © its authors.
|
| 24 |
+
> Tokenizer and chat template are unchanged from the original.
|
| 25 |
|
| 26 |
+
An **EXL3 (ExLlamaV3 trellis) 3.5 bpw** quantization of **Naive-N0.5-Flash** — 309B parameters,
|
| 27 |
+
15.5B active, native 1M-token context. The pack is **~137 GB (~128 GiB) against ~315 GB for the FP8
|
| 28 |
+
original**, which is what lets it fit across **two NVIDIA DGX Spark (GB10)** machines.
|
|
|
|
| 29 |
|
| 30 |
+
Quantized with the [exllamav3](https://github.com/turboderp-org/exllamav3) tooling, adapted for this
|
| 31 |
+
architecture (the patch that adds it ships with the recipe), run on **8×H100** for speed.
|
| 32 |
+
|
| 33 |
+
## Serving
|
| 34 |
+
|
| 35 |
+
Serve it with the recipe:
|
| 36 |
+
**[sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark](https://github.com/sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark)**.
|
| 37 |
+
|
| 38 |
+
EXL3 weights do not load in stock vLLM or `transformers`, and this is a uniform fractional-bitrate
|
| 39 |
+
(3.5 bpw) pack, so it needs the exact pinned stack the recipe installs on both nodes: **vLLM 0.29.0**
|
| 40 |
+
+ [vcruz305/vllm-exl3](https://github.com/vcruz305/vllm-exl3) + a
|
| 41 |
+
[vcruz305/exllamav3](https://github.com/vcruz305/exllamav3) build with fractional-K kernels + the
|
| 42 |
+
recipe's `vllm_naive_n05` model plugin (attention TP2, experts EP2, Ray over RoCE).
|
| 43 |
+
|
| 44 |
+
```bash
|
| 45 |
+
cp config/cluster.env.example config/cluster.env && $EDITOR config/cluster.env
|
| 46 |
+
bash scripts/install.sh # on BOTH nodes
|
| 47 |
+
bash scripts/download.sh # on BOTH nodes — fetches this pack + the DSpark drafter
|
| 48 |
+
bash scripts/start.sh # Ray head + worker, then the OpenAI-compatible API on port 8000
|
| 49 |
+
```
|
| 50 |
+
|
| 51 |
+
`download.sh` pulls this repository and
|
| 52 |
+
[NaiveAI/Naive-N0.5-Flash-FP8-Draft](https://huggingface.co/NaiveAI/Naive-N0.5-Flash-FP8-Draft) for
|
| 53 |
+
speculative decoding. The server is OpenAI-compatible at `http://<head-ip>:8000/v1`, model id
|
| 54 |
+
`naive-n0.5-flash`; loading takes about 7.5 minutes. Reasoning (`<think>`) and tool calls parse
|
| 55 |
+
correctly with the recipe's chat template and parsers — see its README for the details.
|
| 56 |
|
| 57 |
## Specifications
|
| 58 |
|
| 59 |
| | |
|
| 60 |
|---|---|
|
| 61 |
| Base model | [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) (FP8 release as the direct source) |
|
| 62 |
+
| Architecture | MoE, 309B total / 15.5B active, 48 layers, hidden 4096, 64 heads / 4 KV heads, head_dim 192 |
|
| 63 |
+
| Attention | hybrid 39 SWA layers (window 128) + 9 DSA layers (top-2048 selection), GQA4, no full attention |
|
| 64 |
+
| MoE | 256 routed experts, 8 per token, expert intermediate 2048 |
|
| 65 |
+
| Context / vocab | 1,048,576 tokens / 152,576 |
|
| 66 |
+
| Quantization | EXL3 trellis, `mul1` codebook — uniform 3.5 bpw, 6-bit `lm_head`, embeddings unquantized, calibration 250×2048 |
|
| 67 |
+
| Format / size | `exl3` (`--quantization exl3`) — ~137 GB (~128 GiB), 24 shards |
|
|
|
|
|
|
|
|
|
|
| 68 |
| License | MIT (inherited from the original model) |
|
| 69 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
## Files
|
| 71 |
|
| 72 |
| File | Notes |
|
| 73 |
|---|---|
|
| 74 |
+
| `model-00001..00024-of-00024.safetensors` | the quantized weights |
|
| 75 |
+
| `model.safetensors.index.json`, `quantization_config.json` | tensor index and the full EXL3 metadata |
|
| 76 |
+
| `config.json` | carries the EXL3 `quantization_config` incl. the `non_routed_exl3` map (the vLLM-rewritten pack config) |
|
| 77 |
+
| `config.json.native`, `model.safetensors.index.json.native` | the equivalent native **ExLlamaV3** pack metadata |
|
|
|
|
| 78 |
| `modeling_naive_n05_flash.py`, `configuration_naive_n05_flash.py` | model code, loaded via `trust_remote_code` |
|
| 79 |
+
| tokenizer + `chat_template.jinja`, `generation_config.json` | unchanged from the original |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
+
## Notes
|
| 82 |
|
| 83 |
+
- **Beyond 2,048 tokens the DSA layers are approximate.** The model's 9 DSA layers attend to the
|
| 84 |
+
top-2,048 tokens picked by a learned indexer; the recipe runs them as *dense* attention, which is
|
| 85 |
+
exact up to 2,048 tokens but past that sees more than the reference model does. Long-context
|
| 86 |
+
answers may differ from the original; the 1M native context needs a sparse attention backend that
|
| 87 |
+
is not implemented yet.
|
| 88 |
+
- **This is an early recipe, not a tuned one.** Measured decode on 2× DGX Spark: ~34.9 tok/s on pure
|
| 89 |
+
code, ~23.9 mixed, ~19.5 reasoning (~23.5 without speculation). Performance tables, the profile,
|
| 90 |
+
and the limitations are in the recipe's README.
|
| 91 |
+
- **One request at a time**, bf16 KV only, and keep the Sparks otherwise idle (`GPU_MEM_UTIL=0.80`).
|
| 92 |
|
| 93 |
## License and credits
|
| 94 |
|
| 95 |
- **Original model:** [NaiveAI/Naive-N0.5-Flash](https://huggingface.co/NaiveAI/Naive-N0.5-Flash) —
|
| 96 |
+
© the NaiveAI Team, [MIT](https://huggingface.co/NaiveAI/Naive-N0.5-Flash/blob/main/LICENSE); a copy
|
| 97 |
+
ships here as `LICENSE`. Built on Xiaomi's
|
| 98 |
+
[MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Base) and DeepSeek Sparse Attention.
|
| 99 |
+
- **This artifact:** EXL3 3.5 bpw quantization produced with
|
| 100 |
+
[exllamav3](https://github.com/turboderp-org/exllamav3) via the
|
| 101 |
+
[vcruz305](https://github.com/vcruz305/exllamav3) fork. Weights are modified; tokenizer and chat
|
| 102 |
+
template are not.
|
| 103 |
+
- **Recipe:** [sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark](https://github.com/sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark)
|
| 104 |
+
(MIT). Note that [`vllm-exl3`](https://github.com/vcruz305/vllm-exl3) is **AGPL-3.0** — relevant if
|
| 105 |
+
you expose the server to others.
|