Those three cases can produce a similar “90% at layer 20” plot, but they imply very different amounts of saved computation.
The smallest addition that would make the result much easier to interpret would be one line defining the metric, for example:
A compact table like this would resolve most of the remaining branches without requiring a large benchmark:
That peak-and-decline shape may contain more information than the absolute layer number.
Why the metric and computation path matter
1. My current best guess about the 90%
From the surrounding thread and the public code, my leading interpretation is that the intermediate head receives a target-model hidden state and predicts vocabulary logits, approximately:
intermediate_logits = head(hidden_state_at_layer_L)
correct = (
intermediate_logits.argmax(-1)
== target_final_logits.argmax(-1)
)
That would make the reported percentage a form of draft–target top-1 token agreement.
This interpretation is consistent with the public TISAwareDrafter wrapper, whose documented interface is:
input: hidden states from the target model
output: vocabulary-sized predicted-token logits
The same public repository also describes acceptance length as the number of speculative tokens produced before a mismatch with the target model in eval_drafter.py.
However, I would not treat that interpretation as confirmed yet. The evaluator and checkpoint names visible in the image — eval_p9_multi_depth.py and p9_adapter_500steps — are not present in the current public repository, and several related public drafter paths are still marked as placeholders or TODOs. Therefore, the public code establishes the intended architecture and vocabulary, but not the exact implementation of this particular 90% metric.
2. Three different meanings of “retrieving information from layer 20”
A. Full-forward representation probe
input
→ layers 1–20
→ save h20
→ layers 21–32
→ final target prediction
h20 → probe/head → predicted label or token
In this case, the result shows that the evaluated target is decodable from the layer-20 representation.
That is useful evidence about the representation, but it does not by itself save inference computation, because the full model may already have been executed to obtain or evaluate that state.
The safest wording would be something like:
The head can decode the evaluated target from the layer-20 representation under this setup.
That is slightly narrower than saying that the information has been causally “retrieved” or that the later layers are unnecessary. A probe can sometimes read information that the original model does not use in the same way.
B. LayerSkip-style self-speculative early exit
draft:
run layers 1–20
predict candidate tokens
verify:
continue through layers 21–32
accept or correct candidates
This is close to the setup in LayerSkip. LayerSkip trains intermediate layers to support early-exit logits, then uses the remaining layers to verify and correct draft tokens.
The current Transformers assisted-decoding documentation describes this as self-speculative decoding: intermediate layers act as the assistant, while weights and caches can be shared with the target model.
If P9 is doing this, layer 20 has a direct compute interpretation. It is using 20 of 32 decoder layers for each draft step, followed by verification with the remaining layers.
Even then, a 37.5% reduction in draft depth would not automatically mean a 37.5% wall-clock reduction. The result also depends on:
- how many draft tokens are generated per round,
- how many are accepted,
- whether layers 1–20 are reused during verification,
- KV-cache layout and transfer costs,
- LM-head and adapter costs,
- batch size and sequence length.
C. EAGLE-style feature-conditioned drafter
target model produces an internal feature
↓
lightweight drafter
↓
candidate token sequence
↓
target verification
The public TIS wrapper is also compatible with this interpretation: it explicitly accepts target-model hidden states and labels its underlying path as an EAGLE-3 forward pass.
EAGLE-3 uses internal target-model features and multi-layer feature fusion for direct token prediction. In such a design, “layer 20” may identify the feature tap rather than the full computational cost of the drafter.
Therefore, before translating 20 / 32 layers into a speed claim, it would help to know whether P9 is an actual early exit or a lightweight feature-conditioned adapter.
3. A decision tree for the reported accuracy
What is the 90%?
├── Strict greedy top-1 agreement
│ │
│ ├── Shared target prefix at every position
│ │ └── One-step agreement / first-token acceptance proxy
│ │
│ └── Drafter-generated prefix
│ └── Autonomous rollout agreement
│
├── Target-verified speculative acceptance
│ └── Directly relevant to lossless speculative decoding
│
├── Stochastic sampling acceptance
│ └── Depends on both target and draft probability distributions
│
├── Ground-truth next-token accuracy
│ └── Language-model or task accuracy, not target agreement
│
├── Top-k inclusion
│ └── Candidate coverage, weaker than strict top-1 agreement
│
└── Retrieval or classification accuracy
└── Representation/task result, separate from decoding acceptance
Strict top-1 agreement
For greedy decoding, one possible metric is:
draft_token = draft_logits.argmax(-1)
target_token = target_logits.argmax(-1)
accuracy = (draft_token == target_token).float().mean()
This is easy to understand and useful, especially for the first speculative position.
Verified acceptance
If the drafter proposes several tokens and the target verifies them, a more decoding-specific result would be:
accept@1
accept@2
accept@3
...
mean accepted length
For later positions, acceptance is conditional on the earlier speculative tokens having survived. A single average percentage can hide a large difference between:
90% at the first draft token
and:
90% across an autonomous multi-token speculation chain
The second would be substantially stronger.
Sampling acceptance
If generation uses sampling rather than greedy decoding, speculative decoding generally uses an acceptance/rejection rule based on the draft and target distributions. That acceptance probability is not equivalent to top-1 accuracy.
It would therefore help to state whether the plot was produced with:
do_sample = False
or with a particular temperature and sampling acceptance rule.
Ground-truth accuracy
If 90% is accuracy against corpus ground truth, then it should be compared with the full 32-layer target model’s accuracy on the same positions.
Without that baseline, it is difficult to tell whether the head is:
- approximating the target,
- outperforming it on a restricted task,
- or being evaluated on a simpler subset such as answer positions.
4. Why the layer-20 location is plausible but not remarkable by itself
Using an intermediate layer as a draft model is already an established direction.
LayerSkip combines:
- layer dropout,
- an early-exit loss,
- intermediate-layer token prediction,
- and self-speculative verification.
The model is deliberately trained so that earlier layers can produce useful logits. The remaining layers verify or correct the candidates.
The official Hugging Face assisted-decoding guide now exposes compatible models through assistant_early_exit.
Therefore, the fact that an intermediate layer in a 32-layer model can predict target-like tokens is not unprecedented.
What may be more interesting here is one or more of the following:
- the head was trained for only 500 steps,
- the original training depth appears to have been layer 16,
- the same head may generalize to layer 20,
- layer 20 performs better than its training depth,
- and performance then declines sharply at layers 24 and 28.
If the same layer-16-trained head was applied unchanged at all four depths, the result may be demonstrating cross-layer transfer of the readout, not just early-exit quality.
5. Shared head versus one head per depth
The interpretation changes considerably depending on whether the head was shared.
Same layer-16-trained head at all depths
observed accuracy
=
information available at that depth
+
compatibility with the layer-16-trained head
+
normalization/calibration differences
A drop at layers 24 and 28 would not necessarily mean that information disappeared. It might mean that the hidden-state distribution moved away from the basis or scale expected by the head.
Separately trained head for each depth
layer 16 → head trained on layer 16
layer 20 → head trained on layer 20
layer 24 → head trained on layer 24
layer 28 → head trained on layer 28
This is a stronger comparison of how linearly or cheaply decodable the target is at each depth, assuming training data, steps, parameters, and regularization are matched.
The Tuned Lens is relevant here. It trains a separate affine probe for each Transformer block because directly applying a shared unembedding or naive logit lens to every layer can be brittle.
A particularly informative small control would therefore be:
| Depth |
Shared layer-16 head |
Per-depth calibrated head |
| 16 |
— |
— |
| 20 |
— |
— |
| 24 |
— |
— |
| 28 |
— |
— |
Possible interpretations:
- Both columns peak at 20: stronger evidence for a depth-specific phenomenon.
- Only the shared head peaks at 20: calibration or representation-basis mismatch becomes more likely.
- Per-depth heads rise again at 24/28: the information may still be present, but not aligned with the shared head.
- Both decline after 20: the P9 training objective or evaluated task may genuinely favor an intermediate representation.
This does not need to become a large training project. Even a short per-depth calibration run may distinguish several hypotheses.
6. A small independent sanity check
I ran a small control on the same 32-layer model family, mistralai/Mistral-7B-v0.3, using held-out WikiText-2 token positions.
This was not a reproduction of P9. It used a simple low-rank readout adapter and a different training objective, so the absolute values should not be compared directly with 88/90/84/72.
The purpose was only to ask whether a peak at layer 20 followed by a sharp decline is an automatic or generic consequence of reading intermediate Mistral states.
The observed agreement with the full model’s top-1 token was:
| Depth |
Raw LM-head readout |
Layer-16-trained shared adapter |
Per-depth adapter |
| 16 |
4.70% |
25.07% |
25.39% |
| 20 |
10.18% |
25.36% |
26.50% |
| 24 |
20.04% |
34.46% |
36.42% |
| 28 |
42.04% |
52.38% |
53.56% |
In this control, all three paths generally improved toward the later layers. The per-depth adapters improved slightly over the shared layer-16 adapter, but the difference was much smaller than the 18-point decline from 90% at layer 20 to 72% at layer 28 in the posted plot.
I would interpret this narrowly:
The P9 peak-and-decline curve does not appear to be automatically implied by model depth alone.
It does not show that the P9 result is wrong. Instead, it makes the exact P9 setup more interesting, because the curve may depend on something specific such as:
- the P9 training objective,
- synthetic versus natural-text data,
- selected token positions,
- the adapter architecture,
- normalization,
- the reference label,
- acceptance criteria,
- or the use of the TIS importance signal.
Limitations of this sanity check include:
- different adapter architecture,
- different loss,
- 4-bit model loading,
- a small evaluation set,
- no autonomous multi-token rollout,
- no P9 checkpoint,
- and no wall-clock speculative-decoding implementation.
I would therefore keep it as a control, not as a competing benchmark.
7. One-step agreement versus autonomous rollout
Suppose the 90% was measured like this:
target prefix → intermediate head → one predicted token
target prefix → full model → reference token
Every prediction begins from a correct target-generated prefix. This is a useful measurement of one-step compatibility.
Actual speculative rollout is harder:
target prefix
↓
draft token 1
↓
draft token 2 conditioned on draft token 1
↓
draft token 3 conditioned on draft tokens 1–2
Once the drafter makes an error, later states can move outside the distribution seen under teacher forcing.
EAGLE-3 is useful context here because it treats drafter design, direct token prediction, feature selection, and multi-token speculation as separate engineering problems.
If rollout evaluation already exists, even one of these would clarify the result considerably:
mean accepted length
or:
accept@1, accept@2, accept@3, accept@4
For example, two systems could both report 90% one-step accuracy but behave differently:
System A: 90%, 82%, 70%, 55%
System B: 90%, 89%, 87%, 84%
Their first-token plots look identical, but their useful speculative chains are very different.
8. “Sweet spot” may depend on task and context
Even if layer 20 is the best point in this run, the best exit depth need not be fixed across:
- tasks,
- domains,
- prompts,
- generation positions,
- easy versus difficult tokens,
- context lengths,
- or speculation depths.
DEL reports that the best early-exit layer and speculation length can be task-specific and can even vary with the current sequence context.
That suggests a useful progression of claims:
best tested depth in this run
↓
candidate sweet spot for this checkpoint and dataset
↓
stable sweet spot across seeds and held-out prompts
↓
general sweet spot across tasks or contexts
The current plot appears sufficient for the first statement. The later statements would need broader evidence.
This is not necessarily a limitation of P9. A context-dependent optimum could itself become useful: the model might choose an earlier or later exit depending on confidence.
9. The exact layer-state contract is worth recording
For reproduction, layer 20 can be ambiguous unless the state contract is specified.
Useful details include:
physical decoder block number
hidden_states tuple index
pre-block or post-block residual
before or after final RMSNorm
shared LM head, adapter, or independent classifier
head training depth
model/checkpoint revision
Some model APIs include the embedding output as the first entry in the returned hidden-state tuple. In that case:
hidden_states[20]
may not be identical to:
output of decoder block 20
Likewise, applying the final RMSNorm before the LM head can materially change an intermediate readout.
A compact statement would be enough, for example:
Depth 20 means the post-block output of decoder block 20,
then final RMSNorm, then the shared P9 adapter and LM head.
10. Accuracy and wall-clock speed are separate results
A high agreement or acceptance rate is necessary for useful speculative decoding, but it is not sufficient for a speedup.
End-to-end performance depends on:
draft cost
+ verification cost
+ cache/state reuse
+ memory movement
+ LM-head cost
+ accepted tokens per round
+ runtime implementation
A useful practical example is vLLM issue #39940. In the reported comparison, EAGLE-3 acceptance rate and acceptance length were approximately unchanged between two versions, while the measured speculative-decoding speed benefit fell substantially because latency elsewhere in the implementation changed.
That issue is not evidence of the same cause here. It is simply a concrete example of why:
high acceptance ≠ guaranteed wall-clock speedup
For P9, the smallest eventual runtime report could be:
| Configuration |
Tokens/s or TPOT |
Mean accepted length |
| Normal 32-layer decoding |
— |
— |
| P9 speculative decoding |
— |
— |
with:
hardware
batch size
input length
output length
speculation length
greedy or sampling
There is no need to turn the current intermediate experiment into a full serving benchmark immediately. The accuracy curve can stand as a representation/drafter result, and runtime can be reported separately when that path is ready.
11. A compact artifact would resolve most interpretations
The lightest option is simply a metric definition:
prediction = ...
reference = ...
accuracy = ...
The next-lightest option is a small JSON or CSV:
{
"model": "...",
"checkpoint": "...",
"training_depth": 16,
"shared_head_across_depths": true,
"metric": "target_greedy_top1_agreement",
"evaluation_mode": "teacher_forced",
"eval_examples": 100,
"eval_tokens": 12345,
"seed": 42,
"results": {
"16": {"correct": 0, "total": 0, "accuracy": 0.88},
"20": {"correct": 0, "total": 0, "accuracy": 0.90},
"24": {"correct": 0, "total": 0, "accuracy": 0.84},
"28": {"correct": 0, "total": 0, "accuracy": 0.72}
}
}
If the evaluator and checkpoint are intended for later publication, recording these would also help future readers:
source commit
checkpoint SHA256
model revision
evaluation-script revision
That is mainly about preserving the meaning of the result as the project evolves, rather than requiring a large reproduction package now.
12. My final interpretation by branch
If 90% is held-out verified self-rollout acceptance
I would call it a strong candidate result.
The next result that would matter most is mean accepted length or actual decoding throughput.
If 90% is one-step target top-1 agreement
I would call it a promising intermediate-head result.
The unusual depth curve makes it worth investigating, but autonomous rollout remains a separate test.
If 90% is ground-truth accuracy
It should be compared with full-model ground-truth accuracy on the same positions.
It may indicate a strong task-specific head, but it is not automatically a speculative-decoding acceptance result.
If 90% is top-k, retrieval, or classification accuracy
It can still be a useful representation result.
I would label the scope explicitly so readers do not confuse it with strict draft-token acceptance.
If the same layer-16 head was used at every depth
The cross-depth transfer itself is interesting.
A small per-depth calibration control would help distinguish representation quality from readout compatibility.
If a separate head was trained at each depth
The evidence for a true layer-20 optimum is stronger, assuming matched training conditions.
If layer 20 was a real early exit
The result connects naturally to LayerSkip-style self-speculation, and cache/activation reuse becomes central to the speed story.
If layer 20 was only a feature tap
The result is closer to an EAGLE-style feature-conditioned drafter, and 20 / 32 should not be interpreted directly as the fraction of inference computation used.
)
