original repo thinkingmachines Inkling-2-Mini has only 19.5gb model

#3
by KottCh - opened

original repo thinkingmachines Inkling-2-Mini has only 19.5gb model file, so from where is coming your 'full' 52.2gb model?
can you directly convert original 19.5gb file to gguf?

original repo thinkingmachines Inkling-2-Mini has only 19.5gb model file, so from where is coming your 'full' 52.2gb model?

It says so in the readme.

The NVFP4 and MXFP8 experts of the checkpoint are stored as BF16

that not makes 19.5gb full precision (52.2gb)

The BF16 is an upcast.

The original release model is quanized, as stated. But it is simpler to expand it to eg bf16 first, before requantizing it to eg iq4_nl.
The iq4_nl is pretty close in size and which tensor has how many bytes to the original release.

I can guess that, But. if it is upcast problem arises about comparing between quantized models and 'full' (which is up-cast?).

e.g. here AtomicChat/Inkling-2-Mini-GGUF lot of KDL mesurements against base.. but if it is up-cast, is not it silly?
same at Unsloth..

Why is measuring how much you diverge silly?

it is about calling up-cast weights 'original'.

It doesn't.

Original precision. The NVFP4 and MXFP8 experts of the checkpoint are stored as BF16

Original precision is not Original weights

Original precision is not Original weights

Exactly 😉

ggml-org org

I just modified the BF16 row to clarify, it now reads:

| BF16 | 52.2 GB | 16-bit precision. The NVFP4 and MXFP8 experts of the original checkpoint are upcast to BF16 |

As others have commented, the original weights are distributed using NVFP4 and MXFP8 for the experts.

and? my statement is: KDL mesurements against up-cast / not-original BF16 weights is silly(not academic word). should be mentioned original-quantized weights.

Inkling 2 Mini was trained with 4-bit precision, which we describe in a recent article on Connectionism. We use 4-bit NVFP4 precision in the expert layers on all forward passes from pretraining through RL, bolstered by a dedicated kernel and hardware-accelerated on NVIDIA’s GB300 platform. The final 4 layers’ forward passes ran in 8-bit (MXFP8).
Inkling 2 Mini’s weights are deployed in the same numeric precision they were trained on—20 layers in NVFP4 and 4 in MXFP8—so there is no conversion loss from quantization.

https://thinkingmachines.ai/news/inkling-2-mini/#nvfp4-and-low-bit-quantization

Sign up or log in to comment