DSV4-Flash-Vision-ablit-EXL3-Kalibrated

GaelicThunder's Kalibrated pack, remounted in the standard EXL3 layout so ExLlamaV3 can load it. Text and images, 1M context, on a single 140 GB card.

Nothing was requantized here. No Hessians, no calibration pass, no GPU. The expert tensors are his bytes, unchanged; what changed is where they live and what they are called.

What this is

comes from
routed experts of 22 layers, calibrated 3-bit GaelicThunder/…-Kalibrated, tp1/
everything else — attention, router, hyper-connections, shared experts, embed, head, the 3 MTP blocks, the vision tower vcruz305/DSV4-Flash-Vision-ablit-EXL3-MixedK
119.2 GB on disk (111.0 GiB)   ·   70 safetensors shards   ·   138,291 tensors
43 layers, 256 routed experts, 6 active per token, 3 MTP blocks
28 layers at 3-bit   0 1 3 8 9 11 12 13 19 21 22 23 24 25 26 27 28 29 30 31 32 34 35 37 39 40 41 42
15 layers at 2-bit   2 4 5 6 7 10 14 15 16 17 18 20 33 36 38
non-expert tensors   BF16

GaelicThunder promoted 22 layers, ranked by the measured quantization error of their expert tensors. MixedK's own six 3-bit layers (3 13 21 22 28 41) are kept as they were.

Requirements

exllamav3 ≥ v1.4.9. That release added the DeepSeek-V4 vision tower, and it is not a new architecture class — the tower loads as a component of DeepseekV4ForCausalLM, gated on the presence of vision.patch_embed.proj. Earlier versions load the text side and silently ignore the vision tensors.

With TabbyAPI, set vision: true — its vision path is generic (if "vision" in self.config.model_classes), so no per-model wiring is needed.

Memory

The DSA cache is compressed per layer, which is what makes 1M cheap here. Read off compress_ratios and exllamav3/cache/dsa.py:

21 CSA layers (m=4)     (448 + 64 + 128) × 2 / 4   = 320 B/token/layer
20 HCA layers (m=128)   (448 + 64)       × 2 / 128 =   8 B/token/layer
 2 sliding layers       no paged pool at all
                                          total 6,880 B/token

1,048,576 tokens  ->  6.72 GiB of cache
weights           ->  111.0 GiB
                      117.7 GiB to serve the full context

Quality, and what has not been measured

GaelicThunder measured his pack at 90 % retention of the original's per-token probability (87 % prose, 99.8 % math, 92 % code), against MixedK's 81 %, on 64,859 frozen tokens. His numbers, his method — see docs/KALIBRATED.md.

Those numbers were not measured on this repository. His pack stores the non-expert tensors in FP8 block format, because that is what the sparkinfer image loads. This remount takes them from MixedK, in BF16 — the same experts on a higher-precision trunk. The expected retention is therefore ≥ 90 %, but nobody has run the probe on these files. Treat 90 % as a floor inherited from the parent, not as a measurement of this artifact.

Why a remount was needed at all

The Kalibrated pack ships as rank-sliced tp1: tensors pre-cut per tensor-parallel rank, for a loader that picks its own slice. ExLlamaV3 stores whole tensors and splits at load time, so it neither writes nor expects that. With tp = 1 the "slice" is the whole tensor, so the bytes are identical and only the name differs by one infix:

published    layers.27.ffn.experts.0.w1.rank0.trellis
here         layers.27.ffn.experts.0.w1.trellis

The header is rewritten padded with spaces to its original length, so every data_offsets entry stays put and the weight bytes are never touched. Two more things the rename alone would not fix: the source shard for a promoted layer also carries that layer's old 2-bit experts under the same names, so its dense tensors are split out into their own file; and neither parent publishes a model.safetensors.index.json, which is written here.

How it was built, and what was verified

enxertar_camadas_exl3.py, included in this repo. It refuses rather than guess: a shard mixing two layers, an expert name that does not match the base exactly, a trellis shape inconsistent with the declared bit width, or any tensor of the base missing from the final index.

Checked on the finished files, reading them back from disk:

duplicate tensor names across shards ....... 0
names still carrying .rank ................. 0
index vs disk .............................. 138,291 = 138,291, 0 mispointed
bits per layer, from the trellis SHAPE ..... 28 × 3-bit, 15 × 2-bit
config layer_bits vs measured shapes ....... identical
vision.patch_embed / aligner / image_start / mtp.0.attn.wkv / embed / head ... present

The bit widths are read from the tensors (last dim = 16 × K), not trusted from the config.

Credits

  • GaelicThunder — the work that matters: the 22 recalibrated 3-bit layers and the measurements behind the layer choice. Here they only changed name. If he wants these files folded into his own pack, they are his to take.
  • vcruz305 — the MixedK pack, which is everything else in this repository.
  • drowzeys for the abliteration the weights carry; DeepSeek for DeepSeek-V4-Flash-Vision-Exp; turboderp for ExLlamaV3.

License and terms

MIT, as the original. The weights had their refusals removed upstream: they are published for red-teaming, security research and evaluation, and responsibility for any deployment lies with whoever serves them.

Downloads last month
43
Safetensors
Model size
64B params
Tensor type
BF16
·
F32
·
I64
·
I16
·
F16
·
I32
·
I8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Olt1z/DSV4-Flash-Vision-ablit-EXL3-Kalibrated