Native quant?

#1
by singulariti - opened

Are these new models natively q4?

Que 8

How would it be q8 if the total download size is half of the param count?

It's mixed, and the experts are effectively 4-bit. From quantization_config in config.json:

  • quant_method: fp8, fmt: e4m3, weight_block_size: [128, 128] โ†’ the dense/attention/shared weights are FP8 with 128ร—128 block scales (the o_proj layers are in ignored_layers, so those stay BF16)
  • store_dtype: mxfp4, mxfp4_block_size: 32 โ†’ the routed expert weights are stored as MXFP4: 4-bit E2M1 values plus one 8-bit scale per 32 weights = 4.25 bits/weight

That's why the download is about half the param count. The U8 tensors hold 1.000T expert weights. At 4.25 bits that works out to 1.000T ร— 17/32 โ‰ˆ 531 GB. Add ~10.6B BF16 params (21 GB) and 13.4B FP8 params (13 GB) and you get โ‰ˆ 566 GB, which matches the file listing. Averaged over ~1.024T params, that's ~4.4 bits/weight.

So "q8" is right only for the non-expert part. The bulk of the model is MXFP4, the same scheme gpt-oss uses. For memory planning, that's ~527 GiB of weights before any KV cache.

Sign up or log in to comment