CircuitKavach-Aya: tiny-aya-water Global SafeTune Baseline (72/72 Upper-Bound)

This repository contains the Global SafeTune 72/72 upper-bound baseline (checkpoint_2_global_safetune) built on CohereLabs/tiny-aya-water (Best for European and Asia Pacific languages). All 72 of 72 attention and MLP projection blocks (100.00% of transformer projection blocks across 36 layers, 0.00% frozen) were updated by merging the SafeGrad LoRA adapter.

It serves as the full-model upper-bound comparison checkpoint for our sparse Surgical C-ฮ”ฮ˜ Transplant (soapyyapps/tiny-aya-water-circuit-kavach-merged), which modifies only 4/72 blocks (2.55% params) with 97.45% bit-identical weights.

Stage 4 Evaluation Summary (CohereLabs/tiny-aya-water, Held-Out 200 Pairs / 400 Prompts)

Checkpoint Modified Params Frozen Params Proj Blocks Attack Refusal Benign Over-Refusal Benign Helpfulness Selectivity Gap RedTeam Attack Pass Tokens/s
checkpoint_1_base (Base) 0.00% 100.00% 0/72 53.5% 0.0% 100.0% +0.535 19.0% 30.76
checkpoint_2_global_safetune (Global 72/72) 100.00% (LoRA all-layer) 0.00% 72/72 100.0% 0.0% 100.0% +1.000 47.6% 29.02
checkpoint_3_surgical_ctheta (Surgical 4/72) 2.55% 97.45% 4/72 96.5% 0.0% 100.0% +0.965 42.9% 28.76

Per-Language Evaluation Breakdown (eval_200_pairs.jsonl)

Register Base Attack Refusal Global SafeTune (72/72) Surgical C-ฮ”ฮ˜ (4/72) Surgical Gain vs. Base Benign Over-Refusal (Surgical)
English (en, n=50) 100.0% 100.0% 100.0% +0.0 pp 0.0%
Hinglish (hi_en, n=50) 66.0% 100.0% 100.0% +34.0 pp 0.0%
Gujlish (gu_en, n=50) 10.0% 100.0% 98.0% +88.0 pp 0.0%
Kanglish (kn_en, n=50) 38.0% 100.0% 88.0% +50.0 pp 0.0%

Statistical Significance (AuditKIT Paired Bootstrap)

  • Paired Bootstrap (checkpoint_1_base vs. checkpoint_3_surgical_ctheta, n=200 harmful pairs): Surgical C-ฮ”ฮ˜ gains +43.0 pp (0.535 -> 0.965, p = 0.000999, significant: True).
  • Paired Bootstrap (checkpoint_2_global_safetune vs. checkpoint_3_surgical_ctheta, n=200 harmful pairs): delta = +0.035 (1.000 vs. 0.965, p = 0.007992, significant: True).
  • Benign Over-Refusal (n=200 benign twins): 0.0% (p = 1.0, significant: False โ€” zero collateral over-refusal).

Stage 2 Circuit Discovery & Stage 3 Surgical Audit (tiny-aya-water)

  • Model Revision: 41a99216d12b3ef4881df762934186e5e4d2bbc1
  • Calibration Refusal Pair: sorry (id 40449) vs. when (id 1434), logit-diff gap = 16.59
  • Margin-Filtered Calibration Pairs: 99/100 pairs (en=25, hi_en=25, gu_en=25, kn_en=24, english_only_circuit_warning=False)
  • Stage 2 Faithfulness (CircuitKIT): ablation_score = 0.0000, patching_score = 0.0273, mean_jaccard = 0.9788, mean_spearman = 0.999833, n_stable_nodes = 61, kl_reduction_ratio = 0.0149
  • Selected Top-4 Submodules (4/72 projection blocks = 5.56% blocks, 94.44% blocks frozen):
    • model.layers.0.mlp (EAP-IG rank_score = 3.9531)
    • model.layers.16.self_attn (EAP-IG rank_score = 2.8666)
    • model.layers.21.self_attn (EAP-IG rank_score = 1.9318)
    • model.layers.20.self_attn (EAP-IG rank_score = 1.9295)
  • Stage 3 Surgical Audit (SafeTune C-ฮ”ฮ˜, best_alpha = 1.0): 99,090,432 / 3,886,098,432 params modified (2.55% modified, 97.45% frozen), 15/291 weight tensors touched, bit_identical_unmasked_keys = True, train_loss = 0.6995

4-Model Universality Study (fire, global, water, earth)

Cross-Model Safety, Locality & Selectivity (4 Regional Variants)

All four CohereLabs/tiny-aya-* variants independently discovered the exact same top-4 safety submodules (layers.0.mlp, layers.16.self_attn, layers.20.self_attn, layers.21.self_attn, Jaccard = 1.000, 72-block Spearman $\rho$ = 0.9217โ€“0.9704):

Model Base Attack Refusal Global SafeTune (72/72, Over-Refusal) Surgical C-ฮ”ฮ˜ (4/72, Over-Refusal) Surgical Selectivity Modified Params (Blocks) Best $\alpha$ kn_en Surgical Refusal
CohereLabs/tiny-aya-fire (fire) 40.0% 99.0% (0.0%) 94.5% (0.0%) +0.945 2.55% (4/72) 1.0 78.0%
CohereLabs/tiny-aya-global (global) 45.5% 99.0% (0.0%) 96.5% (0.0%) +0.965 2.55% (4/72) 1.0 86.0%
CohereLabs/tiny-aya-water (water) 53.5% 100.0% (0.0%) 96.5% (0.0%) +0.965 2.55% (4/72) 1.0 88.0%
CohereLabs/tiny-aya-earth (earth) 53.5% 99.5% (0.0%) 99.0% (0.0%) +0.990 2.55% (4/72) 1.0 100.0%

Pairwise Circuit Overlap (Top-4 Submodule Jaccard & 72-Block Spearman Rank)

Model Pair Top-4 Jaccard Spearman $\rho$ Shared Submodules
fire__global 1.000 0.9373 layers.0.mlp, layers.16.self_attn, layers.20.self_attn, layers.21.self_attn
fire__water 1.000 0.9217 layers.0.mlp, layers.16.self_attn, layers.20.self_attn, layers.21.self_attn
fire__earth 1.000 0.9322 layers.0.mlp, layers.16.self_attn, layers.20.self_attn, layers.21.self_attn
global__water 1.000 0.9469 layers.0.mlp, layers.16.self_attn, layers.20.self_attn, layers.21.self_attn
global__earth 1.000 0.9378 layers.0.mlp, layers.16.self_attn, layers.20.self_attn, layers.21.self_attn
water__earth 1.000 0.9704 layers.0.mlp, layers.16.self_attn, layers.20.self_attn, layers.21.self_attn

Inference Quickstart (Python)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "soapyyapps/tiny-aya-water-global-safetune"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

messages = [{"role": "user", "content": "Bhai explain how parameterized SQL queries prevent injection safely."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Related Artifacts & Collections

Downloads last month
37
Safetensors
Model size
3B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for soapyyapps/tiny-aya-water-global-safetune

Finetuned
(25)
this model

Collection including soapyyapps/tiny-aya-water-global-safetune