KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively โ€” use -ctk q8_0 -ctv q8_0 (half KV memory, negligible quality loss: perplexity +0.002โ€“0.05) or -ctk q4_0 -ctv q4_0 (quarter memory, โ‰ˆ7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1).

The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

Voxtral-Mini-4B-Realtime-2602-TurboQuant-MLX-8bit

8-bit MLX weight-quantized build of mistralai/Voxtral-Mini-4B-Realtime-2602 with a TurboQuant KV-cache profile. Highest-fidelity MLX variant for low-latency streaming ASR on Apple Silicon.

Hardware compatibility

Device VRAM / RAM Recommendation
Apple M4 Max 128 GB ~5.2 GB recommended โ€” headroom for long context
Apple M3 Max 64 GB ~5.2 GB comfortable
Apple M2 Max 32 GB ~4.8 GB fits

Overview

  • Base: mistralai/Voxtral-Mini-4B-Realtime-2602 โ€” 4B real-time ASR model
  • Weight precision: 8-bit (group-wise)
  • KV-cache profile: TurboQuant
  • Approx. on-disk size: ~4 GB
  • Runtime: MLX on Apple Silicon

Quickstart

pip install mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("majentik/Voxtral-Mini-4B-Realtime-2602-TurboQuant-MLX-8bit")

for chunk in audio_stream():
    prompt = tokenizer.apply_chat_template(
        [{"role": "user", "content": [{"type": "audio", "path": chunk}]}],
        add_generation_prompt=True,
    )
    emit(generate(model, tokenizer, prompt=prompt, max_tokens=32))

Model specs

Field Value
Parameters 4B
Weight bits 8
Group size 64
Cache profile TurboQuant
Size on disk ~4 GB
Target hardware Apple Silicon (M1/M2/M3/M4)
License Apache 2.0

RotorQuant vs TurboQuant

TurboQuant RotorQuant
Strategy Per-head static calibration Rotational online re-basis
Memory reduction ~3.5x on KV-cache ~4x on KV-cache
Best for Predictable domains, lowest p50 latency Noisy/multi-speaker streams

See also

About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's release labels, not distinct quantization algorithms โ€” for any given tier, both brand repos carry byte-identical weights produced with the standard MLX / llama.cpp quantizers. No brand-specific speedup is claimed or measured.

Downloads last month
48
Safetensors
Model size
4B params
Tensor type
U32
ยท
F32
ยท
F16
ยท
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for majentik/Voxtral-Mini-4B-Realtime-2602-TurboQuant-MLX-8bit