Ammu-1.0-Tiny-32M-ONNX-INT8
Quantized Fact-Check Need Classifier | ModernBERT Architecture | 32M Parameters (INT8 Dynamic PTQ) | 95.44% Macro-F1
Overview
Ammu-1.0-Tiny-32M-ONNX-INT8 is the best-performing post-training quantized (PTQ) ONNX INT8 version of Ammu-1.0-FactCheck-Need-tiny. It is a fast, lightweight distilled encoder guardrail model built to decide if a user prompt needs external factual check - search or retrieval (FACT_CHECK_NEEDED) or can be answered directly without search or retrieval (NO_FACT_CHECK_NEEDED). The model is trained on a high-quality dataset of user prompts covering 120 topics, 19 interaction formats, and 29 user personas.
Created with Dynamic INT8 PTQ + Sensitive Layers FP32, this model reduces the 32M ModernBERT FP32 model size from 122.4 MB down to 90.3 MB (26.2% size reduction / 1.36x compression). It speeds up single-query CPU latency to 9.41 ms (1.07x speedup) and increases throughput to 89.4 queries/second (+21.3% throughput gain).
On the holdout test benchmark of 4,214 samples, the quantized model preserves 99.18% of the full FP32 model performance. It achieves an overall Macro-F1 score of 95.44% and a Hard-tier Macro-F1 score of 93.47%. Despite its compact 32M size, it clearly beats the 307M parameter baseline (llm-semantic-router/Vela-1.0-Encoder-307M-FactCheck @ 69.42%) by +26.02% Macro-F1 points, offering high accuracy, low CPU latency, and zero token costs for real-time RAG query routing.
Key Features
- Near-Lossless Quantized Performance: Reaches 95.44% overall Macro-F1 and 93.47% on hard boundary cases. It keeps 99.18% of the FP32 baseline score (only 0.79% difference) and beats the 307M parameter baseline by +26.02%.
- 26.2% Smaller Model Size: Shrinks disk footprint from 122.4 MB to 90.3 MB (a 1.36x compression ratio), making it easy to run on low-memory servers, edge nodes, and microservices.
- Fast CPU Latency & High Throughput: Cuts median single-query latency from 10.08 ms to 9.41 ms while boosting throughput from 73.7 to 89.4 queries/second (+21.3%).
- Sensitive Layer & Head Preservation: Profiles layer sensitivity using validation data to keep critical transformer layers and the classification head in full FP32, while safely quantizing the remaining layers to INT8 (
QuantType.QInt8).
Usage
# Install required dependencies
!pip install uv
!pip uninstall -y torchvision diffusers > /dev/null 2>&1 || true
!uv pip install --system -U "numpy>=2.1.0" "transformers>=4.48.0" "datasets>=3.0.0" "huggingface_hub>=0.28.0" "accelerate" "tabulate" "psutil" "optimum[onnxruntime]>=1.23.0" "onnx>=1.16.0" "onnxruntime>=1.19.0" "scikit-learn"
import sys
try:
import diffusers
except Exception:
sys.modules["diffusers"] = None
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline
# Load the quantized ONNX INT8 Fact-Check model and tokenizer
model_id = "kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8"
model = ORTModelForSequenceClassification.from_pretrained(
model_id,
file_name="model_quantized.onnx",
provider="CPUExecutionProvider"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Initialize the text classification pipeline
classifier = pipeline(
"text-classification",
model=model,
tokenizer=tokenizer
)
# Example user prompts
prompts = [
# Requires external real-world factual check
"India is the most populous country in the world. Do you agree?",
# Requires hidden external fact (Sachin Tendulkar's birth date)
"Assume that I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. How old I am today?",
# Self-contained NLP task on provided text (no fact-check needed)
"Kalyan KS is from India. Extract named entities from the given sentence.",
# Self-contained instruction (no fact-check needed)
"Extract named entities: Kalyan KS is from India.",
# Self-contained creative writing (no fact-check needed)
"Write a short bedtime story about a curious kitten exploring the moon."
]
results = classifier(prompts, max_length=128, truncation=True, padding=True)
# Display predictions
for prompt, res in zip(prompts, results):
print(f"Prompt : {prompt}")
print(f"Prediction: {res['label']} (Confidence: {res['score']:.4f})\n")
Model Evaluation
The model was tested on a blind holdout test set of 4,214 balanced instances from kalyan-ks/fact-check-need-classification across three difficulty levels (Easy: 826, Medium: 1,688, and Hard: 1,700). Testing was run on Google Colab CPU using ONNX Runtime with max_length=128.
Post-Training Quantization (PTQ) Progression: Ammu 1.0 Tiny (32M)
| Quantization Recipe | Weight Prec. | Activation Prec. | Overall Macro-F1 (%) | Easy Macro-F1 (%) | Medium Macro-F1 (%) | Hard Macro-F1 (%) | P50 Latency | Throughput | Model Size | Speedup |
|---|---|---|---|---|---|---|---|---|---|---|
| FP32 Unquantized Baseline | FP32 | FP32 | 96.23 | 99.27 | 96.50 | 94.47 | 10.08 ms | 73.7 q/s | 122.4 MB | 1.00x |
| Dynamic INT8 PTQ (Standard) | INT8 | Dynamic INT8 | 95.42 | 98.91 | 95.68 | 93.47 | 8.33 ms | 77.5 q/s | 86.3 MB | 1.21x |
| Dynamic INT8 + Excluded Head | INT8 | Dynamic INT8 | 95.44 | 98.79 | 95.73 | 93.53 | 9.77 ms | 67.6 q/s | 86.7 MB | 1.03x |
| Dynamic INT8 + Sensitive Layers in FP32 (Selected) | INT8 | Dynamic INT8 | 95.44 | 98.79 | 95.79 | 93.47 | 9.41 ms | 89.4 q/s | 90.3 MB | 1.07x |
| Dynamic INT8 (Per-Channel) | INT8 (per-ch) | Dynamic INT8 | 78.62 | 89.56 | 78.00 | 73.62 | 9.51 ms | 93.9 q/s | 86.4 MB | 1.06x |
| Static INT8 (MinMax Calibration) | INT8 | Static INT8 | 95.21 | 98.55 | 95.56 | 93.24 | 17.50 ms | 55.5 q/s | 86.3 MB | 0.58x |
| Static INT8 (Percentile Calibration) | INT8 | Static INT8 | 95.35 | 98.67 | 95.73 | 93.35 | 17.68 ms | 58.7 q/s | 86.3 MB | 0.57x |
| Static INT8 (Entropy / KL Calibration) | INT8 | Static INT8 | 95.21 | 98.55 | 95.56 | 93.24 | 20.68 ms | 42.1 q/s | 86.3 MB | 0.49x |
Key Architectural Takeaways:
- Distillation Gives Quantization Stability: The distilled Tiny (32M) student is very stable under INT8 dynamic quantization. It keeps over 95.4% Macro-F1 across dynamic setups, retaining 99.18% of the baseline accuracy.
- Best Throughput and Boundary Protection with Dynamic INT8 + Sensitive Layers in FP32: Preserving sensitive layers and the classification head in FP32 provides the highest throughput (89.4 queries/sec) and preserves fine decision boundaries on difficult prompts (93.47% Hard F1).
- Per-Channel Quantization Causes Drift: Using per-channel weight scaling (E4) causes a large accuracy drop (down to 78.62% Macro-F1). Standard per-tensor dynamic quantization is much more reliable for encoder layers.
- Dynamic INT8 is Better than Static on CPU: Dynamic quantization runs much faster on CPU (9.41 ms vs 17.5–20.7 ms for static) and delivers higher accuracy. Static calibration suffers from quantization-dequantization (QDQ) overhead and clips variable-length text activations.
Ammu 1.0 Fact Check Quantized Family Leaderboard
Comparison of the best quantized models across all four Ammu 1.0 Fact Check Need Classification Model sizes:
| Model | Quantization Approach | Parameters | Disk Footprint | P50 Latency (CPU) | Throughput | Hard Macro-F1 (%) | Overall Macro-F1 (%) |
|---|---|---|---|---|---|---|---|
| Ammu-1.0-FactCheck-Need-base-ONNX-INT8 | Dynamic INT8 + Sensitive Layers FP32 | 150M | 284.7 MB | 35.73 ms | 14.7 q/s | 93.35 | 95.13 |
| Ammu-1.0-FactCheck-Need-small-ONNX-INT8 | Static INT8 (Percentile) | 68M | 139.9 MB | 50.74 ms | 20.5 q/s | 84.20 | 85.69 |
| Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8 | Dynamic INT8 + Sensitive Layers FP32 | 32M | 90.3 MB | 9.41 ms | 89.4 q/s | 93.47 | 95.44 |
| Ammu-1.0-FactCheck-Need-nano-ONNX-INT8 | Dynamic INT8 (Per-Channel) | 17M | 53.3 MB | 4.57 ms | 206.9 q/s | 36.41 | 35.74 |
Comparison with 307M Baseline and FP32 Model
| Model | Architecture / Provider | Precision | Easy Macro-F1 (%) | Medium Macro-F1 (%) | Hard Macro-F1 (%) | Overall Macro-F1 (%) |
|---|---|---|---|---|---|---|
| Vela-1.0-Encoder-307M-FactCheck | 307M Encoder Baseline | FP32 | 77.72 | 70.64 | 64.15 | 69.42 |
| Ammu-1.0-FactCheck-Need-tiny | ModernBERT (32M) Baseline | FP32 | 99.27 | 96.50 | 94.47 | 96.23 |
| Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8 | ModernBERT (32M) | INT8 (Dynamic + Sensitive FP32) | 98.79 | 95.79 | 93.47 | 95.44 |
Key Takeaway: Even after INT8 quantization,
Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8outperforms the 307M parameter baseline by +26.02% overall Macro-F1 points and by +29.32% on Hard boundary queries. It achieves this while running in just 9.41 ms CPU latency with 89.4 queries/second throughput.
Quantization Technical Specifications
- Quantization Framework: Microsoft ONNX Runtime Quantization Tools (
onnxruntime.quantization) via Hugging Face Optimum. - Quantization Approach: Dynamic INT8 Post-Training Quantization with Sensitive Layer & Head FP32 Preservation.
- Quantized Operators:
MatMul,Gemm(attention projections and linear feed-forward layers outside sensitive layers). - Excluded Operators: Sensitive transformer layers and classification head nodes (
classifier,pooler,head,out_proj) preserved in full FP32. - Weight Precision: Signed 8-bit Integer (
QuantType.QInt8), per-tensor scaling. - Activation Precision: Dynamic INT8 (computed per prompt on CPU registers).
- Calibration Requirement: None (Dynamic PTQ needs zero calibration data).
Limitations
- CPU Execution Target: This INT8 ONNX model is optimized specifically for CPU inference via ONNX Runtime. For GPU acceleration, FP16 or native TensorRT engines are recommended.
- Language Scope: The model is trained and tested mostly on English text. Accuracy on non-English or code-mixed text may be lower.
- Classification vs. Fact Verification: This model only checks if a prompt needs fact checking (external search or retrieval). It does not verify whether claims are true or false, and does not generate factual text.
- Mixed Intent Prompts: Prompts that mix creative writing and factual questions in a single sentence may have borderline confidence and benefit from custom threshold tuning.
Citation
@misc{kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8-2026,
title = {Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8: Quantized Fact-Check Need Classification Model},
author = {Kalyan KS},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8}
}
- Downloads last month
- 68
Model tree for kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8
Base model
jhu-clsp/ettin-encoder-32mDataset used to train kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8
Article mentioning kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8
Evaluation results
- Macro-F1 (Overall) on Fact-Check Need Benchmark Test Setself-reported0.954
- Macro-F1 (Easy) on Fact-Check Need Benchmark Test Setself-reported0.988
- Macro-F1 (Medium) on Fact-Check Need Benchmark Test Setself-reported0.958
- Macro-F1 (Hard) on Fact-Check Need Benchmark Test Setself-reported0.935