Qwen3.8-27B latent attention retrofits
Collection
Training-free MLA / GLA retrofits of Qwen3.8-27B with a 2x smaller KV cache. Weights preview; vLLM plugin, quants and code coming early October 2026. • 8 items • Updated • 6
Qwen3.8-27B-GLA-g2-INT4
INT4 (AWQ + GPTQ) build of Qwen3.8-27B-GLA-g2, the training-free Grouped Latent Attention with 2 groups (GLA-g2) retrofit of Qwen3.8-27B: 22.9 GB instead of 55.7 GB, with a calibrated FP8 KV cache by default. The most faithful 4-bit build of the family.
Serving: this checkpoint uses a custom attention architecture. It runs in vLLM with the qwen-mla-vllm plugin (experimental). It does not load in
transformersor stock vLLM. Includes the base model's vision tower and MTP head.
Serving (vLLM plugin, experimental)
pip install "git+https://github.com/sootaugur/qwen-mla-vllm" # installs vllm==0.27.1
vllm serve TelperionAI/Qwen3.8-27B-GLA-g2-INT4 --tensor-parallel-size 2 --reasoning-parser qwen3
On PCIe-only GPUs (no NVLink, e.g. RTX PRO or GeForce), add --disable-custom-all-reduce at TP=2.
Images work as with the base model;
for text-only serving add --limit-mm-per-prompt '{"image": 0, "video": 0}'. The KV cache is FP8 by default
(calibrated per layer on this model); --kv-cache-dtype bfloat16 switches to bf16.
MTP speculative decoding (optional)*. The MTP head is included; enable it with
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'. *On an RTX PRO 6000 (TP=1, FP8 KV) it did
not speed this build up: 59 / 279 tok/s at 1 / 8 concurrent requests without MTP, 42 / 258 with k=1, 56 / 317 with k=3. The quantized weights are already cheap to read there, and each verify step
has a fixed cost on the FP8-cache path. On GPUs with less memory bandwidth weight reads dominate, so MTP is
likely to help; measure it on your hardware.
The plugin is experimental: verified on RTX PRO 6000 (Blackwell) at TP=1 and TP=2 with vLLM 0.27.1, with little real-world use so far. FP8-cache decode runs on the plugin's Triton kernels.
What this is
The model is Qwen3.8-27B-GLA-g2: see its card for the method, calibration data and results. Only the storage format changes here.
- Format: compressed-tensors. MLPs of layers 0-55 and the linear-attention projections: INT4,
group 32, asymmetric (AWQ smoothing + GPTQ). Attention
q_proj/o_projand the MLPs of layers 56-63: INT8, group 128, symmetric. Weight-only; activations stay bf16. - Kept in bf16: the latent projections (
kv_a_proj,k_up_proj,v_up_proj,k_rope_proj, and thekv_b_projthe plugin builds from them at load time), the linear-attentionin_proj_a/in_proj_band norms, embeddings,lm_head, the vision tower and the MTP head. - KV cache: FP8 by default, with per-layer scales calibrated on this model (absmax over a
3.7M-token corpus); config
mla_kv_cache_scales.
How it was built. The retrofit differs from Qwen3.8-27B only in its 16 full-attention layers, so it was grafted onto TelperionAI/Qwen3.8-27B-INT4-AWQ-GPTQ-gdn4, a quantized build of the base model:
- every quantized tensor of the MLPs, linear attention,
o_proj(byte-identical in the retrofit) and the MTP head is reused unchanged; q_projis reused unpermuted: the retrofit'sq_projis exactly the baseq_projwith a fixed rope permutation of its query rows (verified bit-exact), so the plugin applies that permutation to the query at runtime (configmla_q_proj_base_layout) -- exact, and cheap;- the latent projections and attention norms are the retrofit's bf16 tensors.
The reused tensors were calibrated on the base model's activations rather than the retrofit's.
The family
Two retrofits of Qwen3.8-27B, each in four weight formats. All keep the vision tower and MTP head, and all ship calibrated FP8 KV-cache scales.
| weights | MLA (1 GPU) | GLA-g2 (2 GPUs) | size | KV cache default |
|---|---|---|---|---|
| bf16 | Qwen3.8-27B-MLA | Qwen3.8-27B-GLA-g2 | 55.7 GB | bf16 (FP8 opt-in) |
| FP8 | -MLA-FP8 | -GLA-g2-FP8 | 31.5 GB | FP8 |
| INT4 (AWQ+GPTQ) | -MLA-INT4 | -GLA-g2-INT4 | 22.9 GB | FP8 |
| EXL3 4.0 bpw | -MLA-EXL3-4.0bpw | -GLA-g2-EXL3-4.0bpw | 17.2 GB | FP8 |
Fidelity to the bf16 base model, teacher-forced on 562k held-out positions: top-1 disagreement where the base model is confident (top-1 vs top-2 logprob margin 2-5; lower is closer). Quantized builds measured with their default FP8 KV cache.
| bf16 | FP8 | INT4 | EXL3 4.0 bpw | |
|---|---|---|---|---|
| base model's own quantized builds | - | 0.76% | 0.59% | 0.59% |
| MLA | 1.09% | 1.15% | 1.26% | 1.56% |
| GLA-g2 | 2.47% | 3.16% | 2.86% | 3.62% |
FP8 KV on the bf16 retrofits measured identical (1.09% / 2.47%); its effect is within run-to-run noise.
Capacity, full 262k-token sequences that fit at once (RTX PRO 6000, 96 GB, default settings with vision on, no MTP). Base model, bf16: 1.9 on one GPU.
| bf16 | bf16 + FP8 KV | FP8 | INT4 | EXL3 | |
|---|---|---|---|---|---|
| MLA, 1 GPU | 3.7 | 7.3 | 12.7 | 14.5 | 15.6 |
| GLA-g2, 2 GPUs (TP=2) | 13.7 | 26.9 | 31.7 | 33.9 | 34.9 |
Limitations
- Serving requires the experimental vLLM plugin (vLLM 0.27.1 only, for now).
- Not yet benchmarked on downstream tasks (SWE-bench, HELMET); the fidelity measurement above is the evidence so far. A rebench of the quantized family is planned.
- Results are from one harness. Absolute scores differ across harnesses.
License and attribution
Released under Apache-2.0, the license of the base model. Derived from Qwen/Qwen3.8-27B by the Qwen team. This is an independent project, not affiliated with or endorsed by the Qwen team.
- Downloads last month
- 74