MiniCPM5-1B-Quanto-FP8

This repository provides openbmb/MiniCPM5-1B quantized to FP8 (Float8 e4m3fn) using optimum-quanto.

It is optimized for 4th Generation Tensor Cores (NVIDIA Ada Lovelace RTX 40xx series, Hopper H100, Blackwell B200) to deliver ultra-low time-to-first-token (TTFT 76ms), 1.78x higher decode throughput (28.3 tok/s), and a compact 1.4GB VRAM footprint without any loss in reasoning, algebra, or anti-hallucination accuracy compared to native bfloat16.

• Base Model: openbmb/MiniCPM5-1B (1.16B parameters, 128k context, LlamaForCausalLM)
• Quantization Method: optimum-quanto FP8 (weights=qfloat8_e4m3fn, activations=None)
• Model Weight Size: 1.22 GB (model.safetensors)
• Benchmark Evaluation Dataset: aoiandroid/minicpm5-1b-quantization-benchmark

---

Performance Benchmarks (NVIDIA RTX 4070 Ti 12GB)

Measured across 8 authentic conversational, logical, mathematical, and structural scenarios:

Metric	bfloat16 (Native)	BitsAndBytes 4bit (NF4)	Quanto FP8 (This Model)	Quanto INT4
Model Weight VRAM	2071.1 MB (2.02 GB)	1108.3 MB (1.08 GB)	1425.0 MB (1.39 GB)	1128.8 MB (1.10 GB)
Peak VRAM During Inference	2109.4 MB	1148.9 MB	1478.7 MB	1166.3 MB
Time-To-First-Token (TTFT)	135.2 ms	193.3 ms	76.4 ms (43.6% faster)	67.0 ms
Decode Speed (TPS)	15.9 tok/s	11.5 tok/s	28.3 tok/s (1.78x faster)	32.5 tok/s
Overall Speed (TPS)	15.8 tok/s	11.4 tok/s	28.1 tok/s	32.3 tok/s
Reasoning Accuracy (Math/Logic)	100% (Full accuracy)	Degraded	100% (Zero loss vs BF16)	Severely degraded
Anti-Hallucination Rejection	Perfect	Unstable	Perfect	Infinite loop

Why FP8 Outperforms BitsAndBytes 4bit:

1. Hardware Tensor Core Execution: On Ada Lovelace GPUs, FP8 GEMM is executed directly on 4th-gen Tensor Cores.
2. Elimination of Dequantization Overhead: Traditional 4-bit (BitsAndBytes NF4) suffers from dynamic dequantization kernel overhead before every GEMM operation, causing an inverted slowdown on small (1B) models.
3. Zero Accuracy Loss: Float8 e4m3fn preserves dynamic range across attention projections, preventing the logic collapse observed in INT4.

---

Quickstart Inference Code

Installation

pip install torch transformers accelerate optimum-quanto

Direct Loading with optimum.quanto

import torch
from optimum.quanto import QuantizedModelForCausalLM
from transformers import AutoTokenizer

model_id = "aoiandroid/MiniCPM5-1B-Quanto-FP8"

# 1. Load model and move to CUDA
model = QuantizedModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.to("cuda")

# 2. Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# 3. Format prompt using chat template
prompt = "こんにちは！自己紹介をしてください。"
messages = [{"role": "user", "content": prompt}]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
inputs.pop("token_type_ids", None)

# 4. Generate with high-speed FP8 Tensor Cores
with torch.no_grad():
    output_tokens = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.8
    )

response = tokenizer.decode(output_tokens[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

---

Evaluation Data and Reproducibility

For full raw benchmark output logs (JSON) and evaluation scripts across all 8 test cases, visit the evaluation dataset repository:

• aoiandroid/minicpm5-1b-quantization-benchmark