3ntr0py-t4m3r/GLiNER2.5-Decide-GGUF

GLiNER2.5-Decide — a decision model for llama.cpp

GLiNER2.5-Decide is a zero-shot classifier: you give it a label set and some text, it scores every label in a single forward pass. This package ports it to llama.cpp as a decision model, served behind upstream's /v1/systemone endpoint.

The decision head is a 2-layer MLP (2.1M parameters) that reads the final hidden state at each [L] label marker and emits one logit per label. Here it lives inside the GGUF and llama.cpp applies it for you — there is no per-token head to run in your own process, and no /embedding round trip.

$ curl -s localhost:8080/v1/systemone -H 'Content-Type: application/json' -d '{
    "state": "Patient reports crushing chest pain radiating to the left arm.",
    "questions": {
      "urgency": {"type": "choice", "instructions": "How urgent is this?",
                  "criteria": {"routine": null, "soon": null, "emergency": null}}
    }
  }' | jq '.answers.urgency'
{
  "type": "choice",
  "choice": "emergency",
  "probabilities": {"routine": 0.06, "soon": 0.34, "emergency": 0.60},
  "confidence": 0.40
}

---

Quick start

Requires llama.cpp built with the decision port — plain upstream master carries it as of PR #29818 (2026-10-02). There are no prebuilt binaries here; build it yourself.

llama-server --model gliner-decision-q8_0.gguf --ctx-size 512 -ngl 99

--ctx-size 512 is not arbitrary. DeBERTa's disentangled attention materialises an [n_embd, n_row, n_col] tensor per term, so activations are O(n_token^2 * n_embd) — quadratic in sequence length. At n=2048 the two attention terms alone are ~17 GB each and the context reserve asks for ~35 GB, which will OOM a 24 GB card. At n=512 they are ~1.1 GB each. Real decision prompts measured 19–85 tokens, so 512 is comfortable.

If the model fails to load with wrong number of tensors; expected 394, got 390, the binary is from a pre-decision tree that cannot see decision.* tensors. Check grep -c COMMON_DECISION_TYPE_GLINER common/common.h in your checkout — it must be non-zero.

---

Contents

models/    gliner-decision-{f16,q8_0,q4_k_m}.gguf
patches/   0001-deberta-v3-upstream-port.mbox — the port as a reviewable patch
bench/     results.json — the measurement behind every number below
scripts/   bench_decision.py, gliner2_gate0.py, verify_phase{1,2}_*.py, ...
gliner.sh  annotated llama-server launcher with the reasoning for each flag

---

Models

file	size	argmax flips vs f16	max |dp|
gliner-decision-f16.gguf	836 MB	— (reference)	—
gliner-decision-q8_0.gguf	446 MB	0 / 40	0.0109
gliner-decision-q4_k_m.gguf	286 MB	2 / 40	0.0632

Measured over a pinned 10-state × 4-question probe set (40 comparisons), CPU backend, scripts/bench_decision.py. Per-question probabilities, sha256s and raw output are in bench/results.json.

Use Q8_0. It is 1.9x smaller than f16 and flips nothing on this probe. Q4_K_M is smaller again and its two flips are knife-edge rather than failures — one crossed the noul 0.5 threshold by 0.021, the other sat on a case where f16's own top-two margin was 0.0035 — but on a genuinely near-tied question it is the one that will move. Validate it against your own data before relying on it.

---

Calibration: use short label keys

This is the one thing worth reading before you write a single request.

The probe set ran the same 10 states twice against the same server, once with short label keys and once with sentence descriptions. Median top-minus-runner-up margin:

criteria style	example	median margin
short keys	{"routine": null, "soon": null, "emergency": null}	0.2375
sentences	{"routine": "can wait several days, no deadline pressure", ...}	0.0512

Short keys separate 4.6x better.

This is the opposite of the upstream blog post, which measured the same effect in the other direction for a different model — describing the options helped Julia-1 route "I was charged twice" from shipping to billing. Here it hurts. GLiNER scores one marker token per label, so a long criteria string dilutes the marker instead of sharpening it.

Write labels, not sentences:

"criteria": { "routine": null, "soon": null, "emergency": null }   // good
"criteria": { "routine": "can wait several days", "soon": "today" } // diluted

This also decides your quantization risk. A 0.05 median margin is only about 2x the Q4_K_M drift of 0.0632, so a poorly-separated question can flip under Q4_K_M; the same question under Q8_0 (drift 0.0109) will not. If your label set cannot be expressed as short keys, stay on Q8_0.

---

Question types

Three, all answered in one forward pass over the state.

type	you send	you get
choice	an option set, each with an optional description	top option + probability per option + confidence
score	2–10 ordered levels, lowest first	expected level (may fall between two) + probabilities
noul	a yes/no question	probability of yes

Questions in one request are answered independently, and the state is processed once. state may be a string or a chat-message list. This model is text-only.

---

What changed on 2026-10-04 (and why older copies are wrong)

Earlier revisions of this repository published gliner2-decide-encoder-{f32,q4_k,q8_0}.gguf. Those were never decision models. They were the Sep 26 encoder-only conversion, quantized before the decision head was merged, and they carried no decision.* metadata at all. The check is the deberta-v3.decision.type key, which only exists when a head is present:

$ grep -ac 'deberta-v3\.decision\.type' gliner-decision-q8_0.gguf      # -> 1
$ grep -ac 'decision\.head_hidden\.weight' gliner-decision-q8_0.gguf   # -> 1
$ grep -ac 'deberta-v3\.decision\.type' gliner2-decide-encoder-q4_k.gguf   # -> 0

(Count lines rather than use grep -q: on binary data several keys can share a line, and what you want is the count of distinct markers present.)

Served against them, llama.cpp correctly answers 501 This model is not a decision model. The file even announced itself as general.name = "Gliner Decide Staged", so it identified as a decision model while holding none of one. Nothing caught it because it could not answer a single decision question, so it was never load-bearing for anything.

The rule this illustrates: merge the decision head before quantizing, then quantize the whole thing. Merging a head into an already-quantized base is a known footgun — on llama.cpp PR #29818 it flipped argmax on 10/17 questions with max |dp| 0.875. llama-quantize itself handles decision heads correctly; the ordering is the entire discipline.

If you pinned an older copy of this repo, the filenames you have do not match what is published now.

---

Quantization and the decision head

You may see older advice saying "do not quantize the decision head". That is half right, and the current measurement says so more precisely. After llama-quantize:

tensor	f16	q8_0	q4_k_m
decision.head_hidden.weight [1024, 2048]	F16	Q8_0	Q4_K
decision.head_hidden.bias [2048]	F32	F32	F32
decision.scorer_out.weight [2048, 1]	F16	F16	F16
decision.scorer_out.bias [1]	F32	F32	F32

The 2M-parameter hidden matrix is quantized, and Q8_0 still produces zero flips. The final output projection is protected automatically: llama-quantize will not quantize a 1-wide tensor, so biases and the output layer stay at high precision on their own. Quantization decisions can be left to the tool.

(The quantized files record scorer_out.weight with shape [2048] rather than [2048, 1]; ggml drops trailing unit dimensions on quantize. Both are 2048 elements and it is read by name. Not a defect.)

---

Patches

patches/0001-deberta-v3-upstream-port.mbox is the port as a reviewable mail box, against upstream. Plain upstream master already contains the decision support (COMMON_DECISION_TYPE_GLINER in common/common.h), so you only need this patch for older checkouts. patches/deberta-v3-upstream-port.diff is the same change as a plain diff, kept for tools that do not read mbox.

---

Status

milestone	state
serves through upstream /v1/systemone	pass — HTTP 200, verified end to end
decision head merged into GGUF, pre-quantization	pass
q8_0 quantization drift vs f16	pass — 0/40 flips, max|dp| 0.0109
q4_k_m quantization drift vs f16	2/40 flips, both knife-edge
block structure vs PyTorch (embedding, attention, FFN, both post-LN residuals)	pass — max|d| 0.00e+00 across all 24 layers
end-to-end parity vs PyTorch	not closed
vision input	not supported (text-only)

Read the third row from the bottom carefully. Every individual block matches PyTorch's output_hidden_states=True exactly, yet the end-to-end residual against a true float64 reference is not closed. It is deterministic — bit- identical at -t 1 and -t 8 — and the graph is not transposed or mis-scaled, which rules out the obvious structural explanations. A per-layer error profile has not been run, and that is the measurement which would distinguish accumulated numerical error from a localized fault. Until it is, treat this as a working decision model, not a parity-cleared one, and check it against PyTorch on your own data.

---

Scripts

script	purpose
bench_decision.py	quantization drift and separation, as in bench/results.json
gliner2_gate0.py	validates the head against the torch reference in numpy (max_abs_delta 5.9e-07); regenerates the prompt serialisation
verify_phase1_gguf.py	tensor inventory and bucket-function check (512×512 vs make_log_bucket_position)
verify_phase2_parity.py	full-prompt parity against the PyTorch reference
dump_torch_hidden.py	per-token torch hidden states, for the parity gate
stage_encoder.py	builds the decision GGUF from the HF checkpoint

Run the bench with:

python3 scripts/bench_decision.py --server $(which llama-server) \
    --models models/gliner-decision-f16.gguf models/gliner-decision-q8_0.gguf \
    --out bench/results.json

The first --models entry is the reference. It runs CPU-only (-ngl 0) so the numbers reproduce anywhere.