Ammu-1.0-Tiny-32M-ONNX-INT8

Quantized Fact-Check Need Classifier | ModernBERT Architecture | 32M Parameters (INT8 Dynamic PTQ) | 95.44% Macro-F1

Overview

Ammu-1.0-Tiny-32M-ONNX-INT8 is the best-performing post-training quantized (PTQ) ONNX INT8 version of Ammu-1.0-FactCheck-Need-tiny. It is a fast, lightweight distilled encoder guardrail model built to decide if a user prompt needs external factual check - search or retrieval (FACT_CHECK_NEEDED) or can be answered directly without search or retrieval (NO_FACT_CHECK_NEEDED). The model is trained on a high-quality dataset of user prompts covering 120 topics, 19 interaction formats, and 29 user personas.

Created with Dynamic INT8 PTQ + Sensitive Layers FP32, this model reduces the 32M ModernBERT FP32 model size from 122.4 MB down to 90.3 MB (26.2% size reduction / 1.36x compression). It speeds up single-query CPU latency to 9.41 ms (1.07x speedup) and increases throughput to 89.4 queries/second (+21.3% throughput gain).

On the holdout test benchmark of 4,214 samples, the quantized model preserves 99.18% of the full FP32 model performance. It achieves an overall Macro-F1 score of 95.44% and a Hard-tier Macro-F1 score of 93.47%. Despite its compact 32M size, it clearly beats the 307M parameter baseline (llm-semantic-router/Vela-1.0-Encoder-307M-FactCheck @ 69.42%) by +26.02% Macro-F1 points, offering high accuracy, low CPU latency, and zero token costs for real-time RAG query routing.

Key Features

  • Near-Lossless Quantized Performance: Reaches 95.44% overall Macro-F1 and 93.47% on hard boundary cases. It keeps 99.18% of the FP32 baseline score (only 0.79% difference) and beats the 307M parameter baseline by +26.02%.
  • 26.2% Smaller Model Size: Shrinks disk footprint from 122.4 MB to 90.3 MB (a 1.36x compression ratio), making it easy to run on low-memory servers, edge nodes, and microservices.
  • Fast CPU Latency & High Throughput: Cuts median single-query latency from 10.08 ms to 9.41 ms while boosting throughput from 73.7 to 89.4 queries/second (+21.3%).
  • Sensitive Layer & Head Preservation: Profiles layer sensitivity using validation data to keep critical transformer layers and the classification head in full FP32, while safely quantizing the remaining layers to INT8 (QuantType.QInt8).

Usage

# Install required dependencies
!pip install uv
!pip uninstall -y torchvision diffusers > /dev/null 2>&1 || true
!uv pip install --system -U "numpy>=2.1.0" "transformers>=4.48.0" "datasets>=3.0.0" "huggingface_hub>=0.28.0" "accelerate" "tabulate" "psutil" "optimum[onnxruntime]>=1.23.0" "onnx>=1.16.0" "onnxruntime>=1.19.0" "scikit-learn"

import sys
try:
    import diffusers
except Exception:
    sys.modules["diffusers"] = None

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

# Load the quantized ONNX INT8 Fact-Check model and tokenizer
model_id = "kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8"

model = ORTModelForSequenceClassification.from_pretrained(
    model_id,
    file_name="model_quantized.onnx",
    provider="CPUExecutionProvider"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Initialize the text classification pipeline
classifier = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer
)

# Example user prompts
prompts = [
    # Requires external real-world factual check
    "India is the most populous country in the world. Do you agree?",
    # Requires hidden external fact (Sachin Tendulkar's birth date)
    "Assume that I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. How old I am today?",
    # Self-contained NLP task on provided text (no fact-check needed)
    "Kalyan KS is from India. Extract named entities from the given sentence.",
    # Self-contained instruction (no fact-check needed)
    "Extract named entities: Kalyan KS is from India.",
    # Self-contained creative writing (no fact-check needed)
    "Write a short bedtime story about a curious kitten exploring the moon."
]

results = classifier(prompts, max_length=128, truncation=True, padding=True)

# Display predictions
for prompt, res in zip(prompts, results):
    print(f"Prompt    : {prompt}")
    print(f"Prediction: {res['label']} (Confidence: {res['score']:.4f})\n")

Model Evaluation

The model was tested on a blind holdout test set of 4,214 balanced instances from kalyan-ks/fact-check-need-classification across three difficulty levels (Easy: 826, Medium: 1,688, and Hard: 1,700). Testing was run on Google Colab CPU using ONNX Runtime with max_length=128.

Post-Training Quantization (PTQ) Progression: Ammu 1.0 Tiny (32M)

Quantization Recipe Weight Prec. Activation Prec. Overall Macro-F1 (%) Easy Macro-F1 (%) Medium Macro-F1 (%) Hard Macro-F1 (%) P50 Latency Throughput Model Size Speedup
FP32 Unquantized Baseline FP32 FP32 96.23 99.27 96.50 94.47 10.08 ms 73.7 q/s 122.4 MB 1.00x
Dynamic INT8 PTQ (Standard) INT8 Dynamic INT8 95.42 98.91 95.68 93.47 8.33 ms 77.5 q/s 86.3 MB 1.21x
Dynamic INT8 + Excluded Head INT8 Dynamic INT8 95.44 98.79 95.73 93.53 9.77 ms 67.6 q/s 86.7 MB 1.03x
Dynamic INT8 + Sensitive Layers in FP32 (Selected) INT8 Dynamic INT8 95.44 98.79 95.79 93.47 9.41 ms 89.4 q/s 90.3 MB 1.07x
Dynamic INT8 (Per-Channel) INT8 (per-ch) Dynamic INT8 78.62 89.56 78.00 73.62 9.51 ms 93.9 q/s 86.4 MB 1.06x
Static INT8 (MinMax Calibration) INT8 Static INT8 95.21 98.55 95.56 93.24 17.50 ms 55.5 q/s 86.3 MB 0.58x
Static INT8 (Percentile Calibration) INT8 Static INT8 95.35 98.67 95.73 93.35 17.68 ms 58.7 q/s 86.3 MB 0.57x
Static INT8 (Entropy / KL Calibration) INT8 Static INT8 95.21 98.55 95.56 93.24 20.68 ms 42.1 q/s 86.3 MB 0.49x

Key Architectural Takeaways:

  1. Distillation Gives Quantization Stability: The distilled Tiny (32M) student is very stable under INT8 dynamic quantization. It keeps over 95.4% Macro-F1 across dynamic setups, retaining 99.18% of the baseline accuracy.
  2. Best Throughput and Boundary Protection with Dynamic INT8 + Sensitive Layers in FP32: Preserving sensitive layers and the classification head in FP32 provides the highest throughput (89.4 queries/sec) and preserves fine decision boundaries on difficult prompts (93.47% Hard F1).
  3. Per-Channel Quantization Causes Drift: Using per-channel weight scaling (E4) causes a large accuracy drop (down to 78.62% Macro-F1). Standard per-tensor dynamic quantization is much more reliable for encoder layers.
  4. Dynamic INT8 is Better than Static on CPU: Dynamic quantization runs much faster on CPU (9.41 ms vs 17.5–20.7 ms for static) and delivers higher accuracy. Static calibration suffers from quantization-dequantization (QDQ) overhead and clips variable-length text activations.

Ammu 1.0 Fact Check Quantized Family Leaderboard

Comparison of the best quantized models across all four Ammu 1.0 Fact Check Need Classification Model sizes:

Model Quantization Approach Parameters Disk Footprint P50 Latency (CPU) Throughput Hard Macro-F1 (%) Overall Macro-F1 (%)
Ammu-1.0-FactCheck-Need-base-ONNX-INT8 Dynamic INT8 + Sensitive Layers FP32 150M 284.7 MB 35.73 ms 14.7 q/s 93.35 95.13
Ammu-1.0-FactCheck-Need-small-ONNX-INT8 Static INT8 (Percentile) 68M 139.9 MB 50.74 ms 20.5 q/s 84.20 85.69
Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8 Dynamic INT8 + Sensitive Layers FP32 32M 90.3 MB 9.41 ms 89.4 q/s 93.47 95.44
Ammu-1.0-FactCheck-Need-nano-ONNX-INT8 Dynamic INT8 (Per-Channel) 17M 53.3 MB 4.57 ms 206.9 q/s 36.41 35.74

Comparison with 307M Baseline and FP32 Model

Model Architecture / Provider Precision Easy Macro-F1 (%) Medium Macro-F1 (%) Hard Macro-F1 (%) Overall Macro-F1 (%)
Vela-1.0-Encoder-307M-FactCheck 307M Encoder Baseline FP32 77.72 70.64 64.15 69.42
Ammu-1.0-FactCheck-Need-tiny ModernBERT (32M) Baseline FP32 99.27 96.50 94.47 96.23
Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8 ModernBERT (32M) INT8 (Dynamic + Sensitive FP32) 98.79 95.79 93.47 95.44

Key Takeaway: Even after INT8 quantization, Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8 outperforms the 307M parameter baseline by +26.02% overall Macro-F1 points and by +29.32% on Hard boundary queries. It achieves this while running in just 9.41 ms CPU latency with 89.4 queries/second throughput.

Quantization Technical Specifications

  • Quantization Framework: Microsoft ONNX Runtime Quantization Tools (onnxruntime.quantization) via Hugging Face Optimum.
  • Quantization Approach: Dynamic INT8 Post-Training Quantization with Sensitive Layer & Head FP32 Preservation.
  • Quantized Operators: MatMul, Gemm (attention projections and linear feed-forward layers outside sensitive layers).
  • Excluded Operators: Sensitive transformer layers and classification head nodes (classifier, pooler, head, out_proj) preserved in full FP32.
  • Weight Precision: Signed 8-bit Integer (QuantType.QInt8), per-tensor scaling.
  • Activation Precision: Dynamic INT8 (computed per prompt on CPU registers).
  • Calibration Requirement: None (Dynamic PTQ needs zero calibration data).

Limitations

  • CPU Execution Target: This INT8 ONNX model is optimized specifically for CPU inference via ONNX Runtime. For GPU acceleration, FP16 or native TensorRT engines are recommended.
  • Language Scope: The model is trained and tested mostly on English text. Accuracy on non-English or code-mixed text may be lower.
  • Classification vs. Fact Verification: This model only checks if a prompt needs fact checking (external search or retrieval). It does not verify whether claims are true or false, and does not generate factual text.
  • Mixed Intent Prompts: Prompts that mix creative writing and factual questions in a single sentence may have borderline confidence and benefit from custom threshold tuning.

Citation

@misc{kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8-2026,
  title = {Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8: Quantized Fact-Check Need Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8}
}
Downloads last month
68
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8

Quantized
(1)
this model

Dataset used to train kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8

Article mentioning kalyan-ks/Ammu-1.0-FactCheck-Need-tiny-ONNX-INT8

Evaluation results

  • Macro-F1 (Overall) on Fact-Check Need Benchmark Test Set
    self-reported
    0.954
  • Macro-F1 (Easy) on Fact-Check Need Benchmark Test Set
    self-reported
    0.988
  • Macro-F1 (Medium) on Fact-Check Need Benchmark Test Set
    self-reported
    0.958
  • Macro-F1 (Hard) on Fact-Check Need Benchmark Test Set
    self-reported
    0.935