PollardWeights/Qwen2.5-7B-Instruct-Pollard

Qwen2.5-7B-Instruct — Pollard

Pollard shrank this model: 15.20 GB (f16) → 2.04 GB — 87% smaller, 7.5× down.

The smallest rung here; larger, higher-fidelity rungs are listed below.

format	this model's size
f16	15.20 GB
Q8_0	~8.06 GB
Q6_K	~6.23 GB
Q4_K_M	~4.41 GB
PollardMix (this repo's IQ1_KT)	2.04 GB

Pollard builds of Qwen/Qwen2.5-7B-Instruct made with Pollard Weights — a ladder of measured-allocation quants (bits placed by per-layer sensitivity, not a uniform crush).

Standard GGUF — runs in stock llama.cpp / ik_llama.cpp, Ollama, LM Studio, except where noted. IQ1_KT needs ik_llama.cpp: its allocation puts ik_llama-only atoms on the tensors it protects. The rest run anywhere.

Model details

Parameter count	~7.6B
Architecture	qwen2
Input support	text
imatrix	yes — see calibration
Perplexity measured	yes — table below

Which file should I choose?

Every rung is the same weights, sized to a different RAM budget by the measured allocation. Pick the largest one that fits your machine with room for context:

• ~8 GB RAM / VRAM → Q6_K (6.25 GB). stock atoms — near-lossless
• ~6 GB RAM / VRAM → IQ4_XS (4.46 GB). stock atoms — recommended default
• ~6 GB RAM / VRAM → IQ3_S (3.81 GB). stock atoms
• ~4 GB RAM / VRAM → IQ1_KT (2.04 GB). (ik_llama.cpp) trellis atoms — ik_llama.cpp only; the flagship mixed 1-bit build

Available files (WikiText-2 raw test, ctx 2048, 145 chunks)

f16 reference PPL 6.5250 ±0.0419.

file	PPL	size	tok/s	Mean KLD	runs in	notes
Qwen2.5-7B-Instruct-Pollard-IQ1_KT.gguf	10.2251	2.04 GB	101.8	0.537	ik_llama	trellis atoms — ik_llama.cpp only; the flagship mixed 1-bit build
Qwen2.5-7B-Instruct-Pollard-IQ3_S.gguf	6.9843	3.81 GB	132.2	—	any llama.cpp	stock atoms
Qwen2.5-7B-Instruct-Pollard-IQ4_XS.gguf	6.6534	4.46 GB	102.7	—	any llama.cpp	stock atoms — recommended default
Qwen2.5-7B-Instruct-Pollard-Q6_K.gguf	6.5463	6.25 GB	97.7	0.0035	any llama.cpp	stock atoms — near-lossless

tok/s measured on an RTX 5070 Ti (16 GB), full GPU offload, Windows/CUDA.

The numbers (WikiText-2 raw, ctx 2048, 145 chunks)

Perplexity with error bars, against the f16 reference. Sizes are the published byte counts:

build	PPL	vs f16	size	bpw
f16 reference	6.5250 ±0.0419	—	15.23 GB	16.00
Q6_K	6.5463 ±0.0421	+0.3%	6.25 GB	6.57
IQ4_XS	6.6534 ±0.0431	+2.0%	4.46 GB	4.68
IQ3_S	6.9843	+7.0%	3.81 GB	4.00
IQ1_KT	10.2251 ±0.0699	+56.7%	2.04 GB	2.14

Q6_K costs 0.3% perplexity at 41% of f16's size. IQ4_XS costs 2.0% at 29%.

What the stock rebuild cost

IQ4_XS and IQ3_S were rebuilt on stock ggml atoms. The files they replaced carried ik_llama-only atoms (IQ4_K, IQ5_K) on every attn_v while wearing stock type names, so they could not open in stock llama.cpp, Ollama or LM Studio — which is what the old card promised they would do. Both versions were measured on the same corpus:

rung	ik_llama atoms	stock atoms	PPL change	size change
IQ4_XS	6.66 @ 4.22 GB	6.6534 @ 4.46 GB	-0.01	+5.6%
IQ3_S	6.96 @ 3.50 GB	6.9843 @ 3.81 GB	+0.02	+8.8%

The trade is legible: perplexity is unchanged and the files are a few per cent larger. ik_llama's IQ4_K/IQ5_K are genuinely more efficient atoms, and giving them up costs size rather than quality. In exchange these two rungs now load in every runtime their names imply. IQ1_KT was left on trellis atoms because trellis has no stock equivalent at all — that is where ik_llama's advantage is structural rather than incidental.

Checked with pollard-ggufcheck, which reads ggml type IDs out of the header rather than trusting the filename.

Measured notes

The IQ1_KT flagship beats a uniform 1-bit IQ1_KT baseline (PPL 11.86, Mean KLD 0.689, top-1 65.1%) on every metric at the same size class — PPL −14%, Mean KLD −22%, top-1 +4.2 pts. The K-quant ladder is imatrix-guided; on a dense model that is where the bits-per-byte win lives (the measured-KL knapsack is reserved for MoE, and is not claimed here).

Mean KLD is carried for Q6_K and IQ1_KT, whose files are unchanged. It is not restated for the two rebuilt rungs: those numbers were measured against the files that have been replaced, and re-measuring them is outstanding rather than assumed.

Prompt format

ChatML, the Qwen2.5-Instruct template:

<|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant

The template is embedded in the GGUF metadata, so llama.cpp, Ollama and LM Studio apply it for you.

Download a specific file

pip install -U "huggingface_hub[cli]"
hf download PollardWeights/Qwen2.5-7B-Instruct-Pollard \
  --include "Qwen2.5-7B-Instruct-Pollard-IQ4_XS.gguf" --local-dir ./

How to run

These are standard GGUF and run with llama.cpp:

llama-server -hf PollardWeights/Qwen2.5-7B-Instruct-Pollard:IQ4_XS

or from a local file:

llama-cli    -m Qwen2.5-7B-Instruct-Pollard-IQ4_XS.gguf -ngl 99 -p "Explain why the sky is blue."
llama-server -m Qwen2.5-7B-Instruct-Pollard-IQ4_XS.gguf -ngl 99      # OpenAI-compatible API + web UI at :8080

They also work in anything built on llama.cpp — LM Studio, koboldcpp, Jan, ramalama, Ollama (ollama run hf.co/PollardWeights/Qwen2.5-7B-Instruct-Pollard).

imatrix (calibration)

The importance matrix (qwen7b_calib3.imatrix, included) was computed on Calib-3.0: 13.9 MB of mixed prose, code, reasoning and dialogue, 60 chunks at ctx 512.

ARM / AVX

llama.cpp repacks weights into an interleaved layout at load time for faster inference on ARM and AVX machines — no special file needed, online repacking covers these quants. The old Q4_0_4_4/4_8/8_8 variants are not required.

Errata

• IQ1_KT carries ik_llama-only atoms and needs ik_llama.cpp to run; stock llama.cpp rejects any ggml type above 42 outright. Checked with pollard-ggufcheck, from the files' tensor types rather than their names.
• Measured allocation places bits by per-layer sensitivity under a size budget.
• Single machine; replication invited.

Credits & license

• Base model: Qwen/Qwen2.5-7B-Instruct (Qwen)
• Quantization tooling: llama.cpp (ggml-org)
• Method + tooling: Pollard Weights — measure first, no claim before a number.
• License: apache-2.0, inherited from the base model.

Built with Pollard Weights — frontier models, small hardware, no compromise.