Instructions to use Olt1z/DSV4-Flash-Vision-ablit-EXL3-Kalibrated with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Trellis
How to use Olt1z/DSV4-Flash-Vision-ablit-EXL3-Kalibrated with Trellis:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
DSV4-Flash-Vision-ablit-EXL3-Kalibrated
GaelicThunder's Kalibrated pack, remounted in the standard EXL3 layout so ExLlamaV3 can load it. Text and images, 1M context, on a single 140 GB card.
Nothing was requantized here. No Hessians, no calibration pass, no GPU. The expert tensors are his bytes, unchanged; what changed is where they live and what they are called.
What this is
| comes from | |
|---|---|
| routed experts of 22 layers, calibrated 3-bit | GaelicThunder/…-Kalibrated, tp1/ |
| everything else — attention, router, hyper-connections, shared experts, embed, head, the 3 MTP blocks, the vision tower | vcruz305/DSV4-Flash-Vision-ablit-EXL3-MixedK |
119.2 GB on disk (111.0 GiB) · 70 safetensors shards · 138,291 tensors
43 layers, 256 routed experts, 6 active per token, 3 MTP blocks
28 layers at 3-bit 0 1 3 8 9 11 12 13 19 21 22 23 24 25 26 27 28 29 30 31 32 34 35 37 39 40 41 42
15 layers at 2-bit 2 4 5 6 7 10 14 15 16 17 18 20 33 36 38
non-expert tensors BF16
GaelicThunder promoted 22 layers, ranked by the measured quantization error of their expert
tensors. MixedK's own six 3-bit layers (3 13 21 22 28 41) are kept as they were.
Requirements
exllamav3 ≥ v1.4.9. That release added the DeepSeek-V4 vision tower, and it is not a new
architecture class — the tower loads as a component of DeepseekV4ForCausalLM, gated on the
presence of vision.patch_embed.proj. Earlier versions load the text side and silently
ignore the vision tensors.
With TabbyAPI, set vision: true — its vision
path is generic (if "vision" in self.config.model_classes), so no per-model wiring is needed.
Memory
The DSA cache is compressed per layer, which is what makes 1M cheap here. Read off
compress_ratios and exllamav3/cache/dsa.py:
21 CSA layers (m=4) (448 + 64 + 128) × 2 / 4 = 320 B/token/layer
20 HCA layers (m=128) (448 + 64) × 2 / 128 = 8 B/token/layer
2 sliding layers no paged pool at all
total 6,880 B/token
1,048,576 tokens -> 6.72 GiB of cache
weights -> 111.0 GiB
117.7 GiB to serve the full context
Quality, and what has not been measured
GaelicThunder measured his pack at 90 % retention of the original's per-token
probability (87 % prose, 99.8 % math, 92 % code), against MixedK's 81 %, on 64,859 frozen
tokens. His numbers, his method — see
docs/KALIBRATED.md.
Those numbers were not measured on this repository. His pack stores the non-expert tensors in FP8 block format, because that is what the sparkinfer image loads. This remount takes them from MixedK, in BF16 — the same experts on a higher-precision trunk. The expected retention is therefore ≥ 90 %, but nobody has run the probe on these files. Treat 90 % as a floor inherited from the parent, not as a measurement of this artifact.
Why a remount was needed at all
The Kalibrated pack ships as rank-sliced tp1: tensors pre-cut per tensor-parallel rank, for
a loader that picks its own slice. ExLlamaV3 stores whole tensors and splits at load time, so
it neither writes nor expects that. With tp = 1 the "slice" is the whole tensor, so the
bytes are identical and only the name differs by one infix:
published layers.27.ffn.experts.0.w1.rank0.trellis
here layers.27.ffn.experts.0.w1.trellis
The header is rewritten padded with spaces to its original length, so every data_offsets
entry stays put and the weight bytes are never touched. Two more things the rename alone
would not fix: the source shard for a promoted layer also carries that layer's old 2-bit
experts under the same names, so its dense tensors are split out into their own file; and
neither parent publishes a model.safetensors.index.json, which is written here.
How it was built, and what was verified
enxertar_camadas_exl3.py, included in this repo. It refuses
rather than guess: a shard mixing two layers, an expert name that does not match the base
exactly, a trellis shape inconsistent with the declared bit width, or any tensor of the base
missing from the final index.
Checked on the finished files, reading them back from disk:
duplicate tensor names across shards ....... 0
names still carrying .rank ................. 0
index vs disk .............................. 138,291 = 138,291, 0 mispointed
bits per layer, from the trellis SHAPE ..... 28 × 3-bit, 15 × 2-bit
config layer_bits vs measured shapes ....... identical
vision.patch_embed / aligner / image_start / mtp.0.attn.wkv / embed / head ... present
The bit widths are read from the tensors (last dim = 16 × K), not trusted from the config.
Credits
- GaelicThunder — the work that matters: the 22 recalibrated 3-bit layers and the measurements behind the layer choice. Here they only changed name. If he wants these files folded into his own pack, they are his to take.
- vcruz305 — the MixedK pack, which is everything else in this repository.
- drowzeys for the abliteration the weights carry; DeepSeek for DeepSeek-V4-Flash-Vision-Exp; turboderp for ExLlamaV3.
License and terms
MIT, as the original. The weights had their refusals removed upstream: they are published for red-teaming, security research and evaluation, and responsibility for any deployment lies with whoever serves them.
- Downloads last month
- 43
Model tree for Olt1z/DSV4-Flash-Vision-ablit-EXL3-Kalibrated
Base model
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp