For now, I also tried a few small quality and CPU experiments on my side:
Short answer
I think this is a worthwhile direction to investigate, particularly because the question is CPU autoregressive inference, not compression in isolation. The interesting possibility is to design the representation and the matrix kernel together: a compact index stream, a small reusable table, and an access pattern that the CPU can execute efficiently. Your original post already recognizes the lookup-overhead problem, so I would make that a measurable part of the experiment rather than treat it as an objection to the idea.
My provisional answer to “could this make CPU inference significantly more efficient?” is possibly, but the speedup has not yet been demonstrated. Three different questions need separate evidence:
- Storage: how many bytes does the actual format use, including indices, table entries, scales, alignment, and any exceptions?
- Quality: how much does the representation change the model’s output on held-out text, rather than just its weight reconstruction error?
- CPU runtime: can a packed-weight kernel turn fewer weight bytes into fewer DRAM transactions and more generated tokens per second than an optimized quantized baseline?
The first question is partly arithmetic; the other two need measurements. One subtlety turned out to matter a lot in my pilot: a single model-global table of 256 reconstructed real values is different from a single model-global table of 256 normalized codepoints followed by block-specific scaling. Both reuse a small table, but only the first limits the actual decoded weight values to one set of 256 numbers.
The practical route I would start with
I would keep the initial procedure small and separate its gates.
- Quality gate: on one fixed checkpoint and held-out text, compare the strict global-real256 proposal, shared normalized256 plus local FP16 scales, and a conventional Q8-like control. Use model NLL or perplexity. The limited comparison below is already a first data point.
- Size gate: compute actual encoded size, including all side information. The strict 8-bit global-index format approaches 8.0 bits per eligible weight; the shared normalized format with one FP16 scale per 32 weights approaches 8.5 bpw. That difference matters when the alternative is already Q8_0 rather than FP16.
- Runtime gate: if quality is acceptable, compare a fused packed-index CPU kernel against optimized Q8/Q4/IQ implementations on the same model, device, thread count, and decoding workload. Separate single-stream token generation from prompt processing; collect tokens/s and, if available, cache/DRAM counters.
This is a suggested order of discriminating tests, not a request that you perform an entire benchmarking project before discussing the idea. It is also important not to combine a GPU model-quality result with an unrelated CPU microkernel time and present the pair as a measured end-to-end compressed inference result.
A small quality experiment
I used a pinned SmolLM2-135M checkpoint, transformed its eligible transformer linear matrices, reconstructed the approximated weights to FP32, and ran an ordinary forward pass on a short WikiText-2 RAW test prefix. On a T4, the 3,071 scored next-token predictions gave:
| Representation, same model and text |
Short-prefix PPL |
| FP32 reference |
21.174 |
| Exactly one global table of 256 decoded real values |
394.824 |
| One shared normalized256 table + FP16 scale per 32 weights |
21.208 |
| Q8-like B32 numerical control |
21.223 |
This is one small checkpoint, one fitting setup, and a short prefix, not a universal result about global codebooks. A later L4 check on three separated text windows gave a pooled PPL ratio to FP32 of 25.264x for the strict fixed-global-real fit but 1.001724x for the shared normalized B32 variant. Another L4 two-window check gave FP32 PPLs of 22.845 / 28.924, versus 22.876 / 28.951 for that normalized shared control. Those checks used the same checkpoint family and do not replace an official full-test evaluation.
The result suggests a useful branch: if the shared table matters more than globally identical reconstructed real weights, local scaling may preserve quality at a modest storage cost. If the exact global set of 256 real values is the central hypothesis, it should remain a separately labeled experiment; the poor result for my fit is not a proof that every possible fitting method must fail.
A small CPU experiment
Separately, I ran synthetic one-thread C++ GEMV comparisons on a Colab Xeon. For a 2,048 by 8,192 matrix under a representative optimized build:
| Native kernel |
Median time |
| Explicit FP32 AVX2 |
7.039 ms |
| G256 lookup via AVX2 gather |
29.467 ms |
| The same G256 encoded values, block-32 predecode + AVX2 |
9.760 ms |
| Q8-style explicit AVX2 |
2.980 ms |
The two G256 kernels used the same index/codebook values and FP32 block scales and produced matching outputs. Predecoding was about 3.02x faster than this gather path but still 1.39x slower than this FP32 AVX2 reference. The Q8 row is not quality-matched to G256. The timed G256 also used FP32 scales, about 9 bpw, unlike the FP16 scales, about 8.5 bpw, in the GPU model-quality experiment.
So I would not claim a compressed-LLM tokens/s result, a cache-residency benefit, or even a single optimized G256 format that combines both experiments. What the CPU test does show is that the lookup/decode strategy can radically change the cost of the same encoded weights. That makes the kernel design question worth pursuing.
The main design branches
If exactly 256 decoded real values must be shared model-wide: keep that constraint explicit, measure quality across calibration choices and codebook sizes, and only introduce outliers or additional scales as clearly labeled extensions. An exception path can be useful, but it changes what “exactly 256 values” means.
If the key idea is reusing a globally shared normalized table: one can test block scales, smaller subtables, or table selectors while accounting for their bits. My small quality probe favors this route, but its CPU advantage over standard scaled quantization remains untested.
If the key objective is CPU decoding speed: compare direct gathers, block predecoding, SIMD lookup, and integer dot-product paths under the actual target ISA. A 16-symbol register shuffle and a 256-symbol lookup are not interchangeable performance problems.
There are close implementation precedents. Gope et al.'s Arm CPU codebook work combines fine-grained tables with CPU kernels, and the merged llama.cpp IQ4_NL implementation uses 16 nonlinear codepoints with FP16 block-32 scales. They provide design and code references, but neither is a measurement of this strict model-global fixed-real256 representation. The detailed comparisons and limitations are below.
1. The original seven research questions, answered separately
1. How much redundancy or similarity exists between LLM weights?
Approximate weight sharing is feasible in the broad sense, but the narrower question is how much quality-preserving redundancy remains under the specific constraint of one shared set of reconstructed real numbers. In a CPU numerical screening on nine actual SmolLM2-135M linear matrices, the strict global-real256 approximation had aggregate held-out weight NMSE 0.00597034. A global normalized256 table with FP16 block-32 scales gave 0.00003648, close to a Q8-like numerical control at 0.00003393. The difference is real for that fitting setup; it is not an estimate of entropy or redundancy across modern LLM architectures.
For a broader study, per-layer distributions, tail behavior, code usage frequencies, and fitting sensitivity would help distinguish a good stored-file compressor from a good CPU GEMV format.
2. How does the number of clusters affect model quality?
This is still an important gap: my experiments concentrated on 256 entries, not a complete controlled K sweep. A useful next check would compare several codebook sizes using the same calibration source and held-out text, plotting actual bits per weight against model NLL. For example, K values around 64, 128, and 256 can be tested cheaply before adding larger formats. Once K exceeds 256, an 8-bit index is insufficient; 512 symbols require at least nine packed bits or a wider storage type.
The global/per-layer distinction, scaling, fit seed, and exception budget should be controlled separately from K.
3. Can perplexity and downstream quality be preserved?
The answer is conditional on the representation. The short-prefix T4 result was 21.174 PPL for FP32, 394.824 for my strict global fixed-real256 fit, and 21.208 for the shared normalized256+B32 version. The separated-window L4 comparison supported the same qualitative distinction. These were model-forward quality tests after FP32 reconstruction, not packed CPU generation tests.
Weight NMSE cannot replace PPL: later layerwise calibration runs improved aggregate NMSE in both seeds while PPL moved in opposite directions. More detail follows; the present results do not establish cross-model robustness.
4. Can matrix operations work directly on compact indices?
Yes, as an implementable operation: the synthetic C++ kernels consumed packed indexed weights without materializing an entire FP32 matrix before timing. Existing codebook-based kernels also demonstrate the general idea. The unanswered question is whether an optimized 256-entry lookup-and-dot kernel for the chosen numerical format can compete with tuned quantized CPU kernels.
The G256 gather versus block-predecode contrast isolates part of the access strategy, not tokens/s; CPU ISA and compiler output can change the ranking.
5. Does the representation reduce actual DRAM bandwidth?
For 100 million eligible FP16 weights, two bytes each correspond to roughly 200 MB. A strict global256 table of 512 bytes plus one byte per index is about 100 MB plus negligible table overhead. That is a legitimate nominal storage comparison. The actual DRAM traffic during generation is not measured by that arithmetic: index blocks, activations, scales, temporary buffers, cache misses, and prefetch behavior all matter.
Against Q8_0 at roughly 8.5 bpw, the strict 8-bpw index payload saves only about 5.9%; Q4/IQ may be smaller still. The baseline therefore matters.
6. Will a cache-resident codebook increase CPU throughput?
A 256-entry FP16 table is only 512 bytes, so keeping the table near the CPU is plausible. But the much larger index stream still has to be consumed, and each index may introduce dependent lookup or decoding work. As your post already notes, the data representation and computation must be optimized together. The right test measures cache/DRAM activity and tokens/s, not merely that the codebook would fit inside a cache.
Different ways of consuming the same indices can have very different instruction costs, without establishing a model-level speedup.
7. How should it compare with INT8, INT4, GPTQ, and AWQ?
I would distinguish format comparisons from algorithm or runtime comparisons. Q8_0, IQ4_NL, and similar GGML types have concrete storage layouts; GPTQ and AWQ are quantization approaches that can feed different runtime formats and kernels. The llama.cpp encoding reference, the GPTQ paper, and AWQ are useful starting points.
The central comparison remains actual encoded bytes, model quality, and CPU decode throughput, with quality differences reported rather than hidden.
2. What is globally shared? Reconstruction equations and effective bits
The most important representation distinction can be written in two short definitions:
Strict global fixed-real256:
decoded_weight[i] = C[index[i]]
Global normalized256 with local scale:
decoded_weight[i] = scale[block(i)] * C[index[i]]
In both cases the table C may be shared across many layers. In the first, however, all eligible reconstructed weights belong to one fixed set of at most 256 real values. In the second, the shared normalized codepoints are the same, while the block scale creates different actual real values. Having 256 possible INT8 codes alone does not imply having only 256 decoded real values throughout a scaled model. That extra freedom can improve quality, and it has a measurable storage and arithmetic cost. Calling both forms “a global codebook” without specifying the reconstruction equation would blur the central experimental result.
The following figures describe approximate eligible-linear payload, not full-checkpoint size, runtime traffic, or end-to-end memory use:
| Weight representation |
Approx. bits/weight |
Main overhead |
| FP16 |
16.0 |
None beyond the values themselves |
| Strict fixed-global256, uint8 indices |
8.0 |
One amortized global table |
| Shared normalized256, FP16 scale / 64 weights |
8.25 |
16/64 scale bits |
| Shared normalized256, FP16 scale / 32 weights |
8.5 |
16/32 scale bits |
| Synthetic CPU G256 with FP32 scale / 32 weights |
9.0 |
32/32 scale bits |
| GGML Q8_0 |
8.5 |
FP16 scale per 32 quantized values |
| GGML Q4_0 / IQ4_NL |
4.5 |
FP16 scale per 32 four-bit codes |
The GGML figures can be checked against pinned quantized type definitions and encoding documentation. Actual model files may use mixed types, leave some matrices unquantized, or include padding and metadata. Therefore “8.5 bpw” here is a format-level estimate for transformed eligible linears, not a claim that a complete SmolLM2 checkpoint would occupy exactly 8.5 bits for every parameter.
A third option is several small shared subtables with a table ID per group. A fourth is an exception path for sensitive weights. Both preserve some of the table-reuse intuition, but each changes either the number of decoded values or the encoded payload. If the strict global-real256 constraint remains the scientific target, these extensions should be presented alongside it, not silently treated as the same method.
The storage calculation also helps distinguish the 256-entry proposal from a larger table example. A 65,536-entry FP16 table would occupy 128 KiB, which may fit in some cache levels, but selecting that many symbols requires a 16-bit index. The index stream alone would then match FP16’s two bytes per weight. Table residency and compact indices are separate requirements.
Each format should be labeled by reconstruction rule, true bytes, output quality, and CPU execution path.
3. Quality experiments: exact scope and the calibration result
Dataset, model, and what was actually transformed
The model experiments used the pinned SmolLM2-135M revision and selected parts of the pinned WikiText-2 RAW test files. The eligible transformed weights were all 210 transformer linear matrices, comprising 106,168,320 weights. Other model parameters remained outside the format comparison. That distinction matters when comparing hypothetical compressed bpw with the size of an entire stored model.
For model-quality tests, the weights were reconstructed to FP32 for normal T4/L4 forward. Checkpoint and restoration controls were checked in the user’s Colab runs, and returned ZIP/notebook pairs were audited offline. This is not an independent rerun of the raw checkpoint.
Weight-only screening
The first CPU numerical experiment examined nine real model matrices. Fixed-global real256 gave aggregate held-out NMSE 0.00597034; shared normalized256 with FP16 B32 scales gave 0.00003648; the Q8-like proxy gave 0.00003393. A Gaussian-vector projection test supplied a further arithmetic check, but it was synthetic and cannot substitute for model NLL or PPL.
First short model-forward sweep
For the T4 run, 3,071 causal targets were scored within the first 3,072 tokens of the WikiText-2 RAW test split. The complete six-condition comparison was:
| Same T4 quality evaluation |
PPL |
| FP32 |
21.174 |
| Strict global fixed-real256 |
394.824 |
| Sample-fitted layerwise fixed-real256 |
228,824.216 |
| Shared normalized256 + FP16 B32 scales |
21.208 |
| Shared normalized256 + FP16 B64 scales |
21.191 |
| Q8-like B32 proxy |
21.223 |
The sampled layerwise fit was particularly poor, not proof that every per-layer codebook is inferior. Differences among the near-FP32 B32, B64, and Q8 controls are too small on this prefix to rank reliably.
Two robustness checks with different questions
One L4 check used three separated 1,024-token windows, at offsets 0, 32,768, and 180,000, with 3,069 scored targets. Relative to the FP32 reference within that evaluation, pooled PPL ratios were 25.264x for fixed-global real256, 3382.812x for the sampled layerwise fixed-real fit, 1.001724x for normalized shared+B32, 0.999687x for normalized shared+B64, and 1.001347x for the Q8-like proxy. A ratio just below one is not evidence of a reliable accuracy improvement; what matters here is the contrast between formats.
A later L4 test refitted all 30 layers under nested 10k and 100k sampled weights per layer, with two fitting seeds and separate held-out weight samples. It evaluated two 1,024-token text windows, offsets 0 and 32,768, each providing 1,023 causal targets. FP32 PPL was 22.845 / 28.924, and the shared normalized256+B32 positive control was 22.876 / 28.951. The control’s near-FP32 result survived this expanded full-linear setup.
The separately fitted fixed-real layerwise tables were more complicated:
| Fit seed |
Samples per layer |
PPL, offset 0 |
PPL, offset 32,768 |
| 0 |
10k |
1,442.865 |
991.837 |
| 0 |
100k |
173,597.588 |
25,373.610 |
| 1 |
10k |
102,485.194 |
50,380.821 |
| 1 |
100k |
4,664.414 |
969.239 |
For both fitting seeds, increasing the sample count improved aggregate held-out weight NMSE. Nevertheless, the model-output result moved in opposite directions: seed 0 became much worse, while seed 1 became substantially less bad but remained far from FP32. The exact high PPLs are not competitive quality results; they are evidence of sensitivity to the fitted tables. The experiment does not justify saying that “more calibration hurts.” It just shows that lower pooled weight error did not guarantee lower PPL for these severely degraded configurations.
Importantly, none of the reported PPL measurements is the complete official WikiText-2 test-set PPL, and none came from the synthetic C++ packed CPU kernel. They are short, pinned forward evaluations that inform which representation is worth implementing.
4. A more discriminating layer-swap experiment
The non-monotonic calibration result prompted a narrower causal check. Instead of fitting a wholly new codebook or quantizing one layer in an otherwise perfect model, I used the already fitted 10k and 100k layerwise fixed-real tables from the preceding experiment.
For each of the two seeds, a single transformer’s seven eligible linear matrices were switched from its 10k fit to its 100k fit while the other 29 layers remained in the all-10k configuration. The reverse substitution was tested in the all-100k background. The two earlier text windows were reused, with baseline and restoration controls. This produced 240 one-layer intervention PPL rows, plus reference and interaction checks.
Across both seeds and both swap directions, zero-based layers 2 and 11 had especially large absolute changes in mean token-level NLL. In the all-10k background, a single-layer 10k-to-100k substitution yielded the following mean changes, in nats per scored token:
| Zero-based layer |
Seed 0 |
Seed 1 |
| Layer 2 |
+1.883 |
-0.712 |
| Layer 11 |
+1.890 |
-1.757 |
Positive means worse NLL; negative means better NLL. The direction reversed with the fitting seed. That is a measured within-configuration intervention, not a claim that these are universally sensitive layers in every LLM. The reverse-context substitutions had opposite signs but different magnitudes, showing that independent one-layer effects cannot simply be added to predict the whole-model transition.
There was a useful weight-error clue. When moving from 10k to 100k, those two layers’ held-out NMSE worsened locally by about 3.17% and 8.71% in seed 0, but improved locally by 18.39% and 15.90% in seed 1. A pooled model-wide NMSE improvement had hidden those influential local regressions in seed 0. However, other layers contain mismatches, so it would still be wrong to claim that local NMSE fully predicts NLL.
The next cheap discriminator, if deeper quality analysis is desired, is to swap the seven individual projections within layers 2 and 11, then try the two-layer combination and a genuinely new text position. That could distinguish a single projection effect from broader interactions. It has not been done. Exploratory activation differences were observed, but they were selected after seeing the layer effect; they do not identify a specific tail, activation shift, or outlier as the cause.
Most importantly, this intervention used per-layer fixed-real256 tables, not your strict one-table-for-the-entire-model global-real256 format. It is useful evidence about calibration and model-output sensitivity, not a direct demonstration that the global format must fail. The normalized global256+B32 result remains a separate positive control.
5. Why the native CPU implementation matters
An early Xeon C++ GEMV experiment compared four representations on three synthetic shapes. For 1,024 by 4,096, median times were FP32 8.602 ms, fixed-global lookup 8.593 ms, shared normalized lookup with scales 12.492 ms, and a Q8-style proxy 12.580 ms. On the largest shape, the methods were all around 34 ms. No consistent gain over the tested FP32 reference appeared.
Inspecting the returned executable, however, showed that the FP32 hot loop benefited from YMM-width SIMD operations while several lookup/Q8 paths performed substantially scalar per-weight decode and accumulation. The scaled lookup path also repeatedly retrieved scales that could be hoisted per block. So the first experiment was a useful implementation smoke test, not a fair verdict on an optimized indexed architecture. The GCC optimization reference explains the compilation controls; the specific instruction differences came from inspecting the actual binary, not from documentation alone.
A following 42-row native study varied autovectorization and explicit kernels. One auto-generated G256 gather-heavy path was markedly slower than a scalar alternative under a particular GCC build, while moving scale work out of the innermost loop helped. That result narrows the kernel question: one cannot assume auto-vectorization of an indexed table will generate the most efficient memory accesses on every target.
The later experiment broadened this to nine kernel alternatives, four GCC build configurations, and three shapes, for 108 measured kernel/shape/build rows, with repeated samples and output checks. In the largest 2,048-by-8,192 shape under a representative optimized configuration:
| Implementation |
Median latency |
Important interpretation |
| FP32 explicit AVX2 |
7.039 ms |
A tuned dense-float comparison within this prototype |
| G256 AVX2 gather |
29.467 ms |
Tested direct indexed lookup path |
| G256 predecode32 then AVX2 |
9.760 ms |
Same G256 weights and scales as gather |
| Naive 256-symbol nibble-shuffle candidate |
267.368 ms |
This implementation failed; not a general shuffle verdict |
| Q8-style explicit AVX2 |
2.980 ms |
Different numerical approximation; not quality-matched |
The same-value comparison between G256 gather and predecode is one of the cleaner CPU observations. But the predecode path was still slower than the explicit FP32 SIMD reference. It is equally important that the CPU tests were synthetic, one-thread GEMV, used FP32 B32 scales, and never generated LLM tokens or measured memory-controller activity. There is no basis yet for combining the near-FP32 GPU quality result with a CPU speed number to claim a working 8.5-bpw compressed LLM.
The IQ4_NL merged PR illustrates SIMD-friendly lookup/dot design, but its 16-entry 4-bit table is not a cheap analogue of arbitrary 256-entry FP16 gathers. A 256-entry kernel may need different layout, block predecode, or multiple subtables; correctness and actual instructions remain part of the test.
The most informative runtime improvement would be a fused packed-weight kernel that consumes the intended index/table/scale format directly, rather than expanding the model to FP32 before decoding. Only then does the proposed cache benefit become testable against an optimized production baseline.
6. A minimal fair CPU decoding benchmark
For native inference, freeze the checkpoint, tensor coverage, test data, CPU, backend revision, threads, context, and generation length. Compare FP16, optimized Q8_0, a relevant Q4/IQ format, and the candidate codebook. Report encoded bytes and quality alongside speed, including unmatched-quality tradeoffs.
Measure text generation (tg) separately from prompt processing (pp). The official llama-bench guide documents the distinction, repetitions, thread settings, and prefilled context depth. For an integrated CPU-only format, an illustrative command is:
./llama-bench -m path/to/model.gguf -ngl 0 -p 0 -n 128 \
-d 0,512 -t 1,2,4 -r 5 -o json
This is a proposed test command, not one already run for G256. The model and format must first be supported by the chosen runtime; the options should be checked against its specific version. The tool excludes tokenization and sampling time, so its reported tokens/s has a defined scope.
Where permissions and CPU support allow, measure cycles, instructions, LLC misses, and actual DRAM/controller traffic. Intel PCM and Linux perf stat are possible tools, subject to event availability and perf access controls. Shared Colab environments may not expose the necessary counters. Estimated bytes from array sizes are not a substitute for observed DRAM transactions.
If a format lowers transferred bytes and raises matched-quality decode tokens/s, that would support the practical bandwidth hypothesis. If bytes fall but speed does not rise, decode instructions or other work may dominate. If speed rises without lower traffic, the gain may be better arithmetic/layout rather than codebook-cache residency. Each outcome helps isolate the mechanism.
7. Adjacent work and low-cost branches
The Arm shared-codebook study and llama.cpp IQ4_NL PR are my first choices for implementation context. Additional references fit different branches rather than the exact same format:
- Q4X on RISC-V vector CPUs illustrates small codebooks and hardware-aware lookup on a different ISA; its reported performance is not an x86 G256 result.
- Deep Compression and AQLM provide weight-sharing and multi-codebook context, but not the proposed single scalar table.
- SqueezeLLM, SCVQ, and AWQ suggest ways to think about sensitivity, weighted fitting, or limited exceptions if quality is the bottleneck. They do not prove the mechanism behind my v6 layer effects.
- T-MAC and Vec-LUT are CPU lookup-oriented examples with different activation or partial-product semantics. QIGen is relevant when fast quantized kernel generation and layout become the main development cost.
The remaining cheap checks are a controlled codebook-size sweep for strict global256 and individual-projection swaps in layers 2 and 11. The decisive, more expensive CPU test remains packed fused model inference. None is required before sharing these observations.
Where I would leave it
I would keep the original strict 256-real-value proposal as a clearly defined research target, with the shared normalized256 plus local-scale representation as a separate, empirically more promising quality branch. The present pilots suggest that fitting and layer sensitivity can dominate perplexity, while CPU access strategy can dominate synthetic GEMV time. They do not yet establish a real-model CPU decode speedup, DRAM reduction, or a universal verdict on the strict scheme.
That is still a useful result: the experiment can now distinguish what is globally shared, what quality it preserves, and whether the CPU benefits beyond ordinary quantization. I think a careful comparison along those lines could teach us something valuable even if the first implementation is not faster.