--- license: gemma base_model: google/gemma-4-E4B-it library_name: transformers tags: [jevify, classification, calibration, probabilistic, gemma4, lora] --- # kushalpatil/jevify-gemma4-e4b `google/gemma-4-E4B-it` fine-tuned (LoRA, merged) to give **honest probabilities** when asked typed questions about a piece of state — the model behind [jevify](https://github.com/kushalpatil07/jevify), a local, Jev-compatible probabilistic decision API. Nothing is generated: one prefill, read the next-token distribution over the answer labels, done. ## Use ```bash pip install "jevify[transformers] @ git+https://github.com/kushalpatil07/jevify" jevify serve --model kushalpatil/jevify-gemma4-e4b # POST /v1/systemone, Jev wire format ``` ```python from jevify import Jevify, Noul, Choice, Score jev = Jevify.from_transformers("kushalpatil/jevify-gemma4-e4b") # or serve it with vLLM and use Jevify.from_runtime("vllm", "kushalpatil/jevify-gemma4-e4b") jev.system_one("Help! My payouts have been failing for 3 days.", { "urgent": Noul("Does this convey urgency?"), "team": Choice("Which team should handle this?", {"billing": None, "technical": None, "sales": None}), "mood": Score("How frustrated is the customer?", ["calm", "frustrated", "furious"]), }) ``` ## Training LoRA r=64 on attention projections, 2 epochs over ~47k (state, question, target-distribution) items from 16 sources: hard-labeled classification sets with randomized option subsets and order, multi-annotator sets with real human label distributions, and constructed long states (up to 24k tokens) with answers known by construction. Loss = KL(target || label distribution). No teacher model. Recipe: `train/` in the jevify repo. ## Results ``` Held-out, out-of-distribution (6 datasets not used in training; 307 items), no post-hoc scaling: ood_massive n= 50 acc=0.900 nll=0.308 brier=0.140 ece=0.065 conf=0.932 ood_trec n= 50 acc=0.940 nll=0.201 brier=0.095 ece=0.097 conf=0.874 ood_paws n= 50 acc=0.540 nll=0.915 brier=0.622 ece=0.262 conf=0.792 ood_sms_spam n= 50 acc=0.980 nll=0.121 brier=0.056 ece=0.046 conf=0.934 ood_app_reviews n= 50 acc=0.740 nll=0.901 brier=0.427 ece=0.126 conf=0.705 ood_subj n= 50 acc=0.940 nll=0.218 brier=0.109 ece=0.128 conf=0.841 spike n= 7 acc=1.000 nll=0.051 brier=0.021 ece=0.044 conf=0.956 ALL n= 307 acc=0.844 nll=0.435 brier=0.237 ece=0.043 conf=0.849 In-distribution held-out (800 items), raw -> trained: step 0 acc=0.745 conf=0.962 ece=0.217 nll=1.827 brier=0.412 step 1209 acc=0.823 conf=0.823 ece=0.028 nll=0.438 brier=0.193 ```