safecompress-tinyaya-consensus5-fp16prot-nf4

Safety-aware compression of Tiny Aya Global (SafeCompress). Decoder layers [0, 7, 14, 16, 35] are kept in FP16; all other layers are quantized to NF4.

FP16 (reference) uniform 4-bit this model
harmful-prompt refusal (EN+HI, 200 prompts, phrase matcher) 0.655 0.605 0.660 (95% CI 0.595-0.725)
first-token refusal gap on harmful prompts (logits) 1.86 1.69 2.29
Global-MMLU EN+HI accuracy 0.475 0.457 0.462
Belebele EN+HI accuracy 0.770 0.760 0.755
benign-prompt refusal 0.025 0.010 0.020
GPU memory (GB) 6.70 2.53 3.10

Load: AutoModelForCausalLM.from_pretrained("cybertronak/safecompress-tinyaya-consensus5-fp16prot-nf4", device_map="auto") (needs bitsandbytes and a CUDA GPU). You must have accepted the base model's terms. Limitations: 200 harmful and 200 benign prompts, refusal measured with a phrase matcher plus the first-token gap (no judge model), one model, one quantizer family. Differences between protected-layer selections are within overlapping confidence intervals.

Downloads last month
14
Safetensors
Model size
3B params
Tensor type
F16
ยท
U8
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for cybertronak/safecompress-tinyaya-consensus5-fp16prot-nf4

Quantized
(31)
this model

Collection including cybertronak/safecompress-tinyaya-consensus5-fp16prot-nf4