# Parameter Count & Shape Discrepancies in 4-bit vs. Higher bit LLM models

**URL:** <https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512>\
**Category:** 🤗Transformers\
**Created:** [January 17, 2024, 4:26pm UTC](https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512 "2024-01-17T16:26:54Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![kdcyberdude](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kdcyberdude/32/27478_2.png) [@kdcyberdude](https://discuss.huggingface.co/u/kdcyberdude)\
**Post date:** [June 3, 2024, 12:09pm UTC](https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512/3 "2024-06-03T12:09:26Z")

</div>

Hi @winnieyangwannan,

To explain briefly, it’s due to the packaging of 4-bit quantized weights. Essentially, one byte holds two parameters, each consisting of 4 bits.

For a more detailed explanation, feel free to check out this discussion: [Hugging Face Forum - Impact of Quantization on Trainable Parameters](https://discuss.huggingface.co/t/less-trainable-parameters-after-quantization/43192/12).

---

_[View the full topic](https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512)._