Hmm… I think most of the pieces fit now, but they seem to describe two slightly different things:
I tried a small independent replication using only the public 60-query evaluation set, the public checkpoints, and a fixed set of natural scoring wrappers.
The public Colab notebook is here.
The short version is more positive than where I started:
The broad numerical result appears reproducible. What remains unclear is not whether the public TIS checkpoints can produce useful context layouts, but whether their scalar scores are already calibrated as query-relevance scores.
I did not try to recover the exact private trainer or evaluator that produced the published numbers. Instead, I fixed a few ordinary public-artifact-based routes in advance and ran them over all 60 queries.
On a set this size, one answer changes EM by about 1.67 percentage points, so I would treat 20% and 21.7% as the same broad performance regime rather than attach much meaning to a one-answer difference.
| Route |
Reported in TIS 2.0 |
Independent notebook |
| Baseline average |
16.7% |
16.7% |
| Stage 3 |
21.7% |
20.0% |
| v8b |
— |
20.0% |
| Dedicated reranker |
21.7% |
20.0% |
| TF-IDF |
17.8% average |
18.3% |
| BM25 |
— |
18.3% |
| Passage length |
— |
20.0% |
| Fixed random |
— |
16.7% |
So the broad 16.7% → approximately 20–22% result looks real enough to me.
The more interesting question is what kind of signal produced it.
Story A — context-layout policy
───────────────────────────────
same five passages
↓
discard the original retrieval slot
↓
produce one deterministic final order
↓
same prompt for early / middle / end
↓
measured position gap = 0
Story B — relevance scoring
───────────────────────────────
question + passages
↓
learned importance score
↓
relevant passage receives a higher score
↓
better passage ranking
↓
better answer generation
My results support Story A fairly strongly.
Story B still looks plausible, but only partially established.
The broad result appears reproducible
The notebook uses:
- the public MS-MARCO query IDs from the published result artifact;
- five passages per query;
- all 60 queries;
- the public Mistral backbone;
- the public Stage 3, v8b, and dedicated reranker checkpoints;
- fixed evaluator routes rather than searching until a particular EM number appeared;
- one generated result per canonicalized route and query;
- separate early, middle, and end prompts only for the unreordered baseline.
The independent baseline results were:
| Baseline position |
Relaxed EM |
| Early |
18.3% |
| Middle |
15.0% |
| End |
16.7% |
| Average |
16.7% |
Several public-checkpoint routes then reached 20%:
| Canonical route |
Relaxed EM |
| Stage 3, high score first |
20.0% |
| v8b, high score first |
20.0% |
Dedicated best.pt, correct query |
20.0% |
This is enough for me to say that the direction is not obviously a dead end. Public TIS checkpoints can construct context layouts that operate in roughly the reported accuracy range.
It is not yet enough to say that they consistently outperform all simpler ordering rules.
Zero position gap is real, but it is a property of the final layout
The early, middle, and end examples contain the same five passages. Only the original gold-passage slot changes.
Once a deterministic reranker discards that original slot and produces one final order, the three variants collapse into the same prompt:
early input ┐
middle input ├─ same passages ─ deterministic order ─ same prompt
end input ┘
↓
same prediction
↓
zero gap
That means zero gap is a genuine result, but it does not by itself measure scorer quality.
TF-IDF, passage length, a learned scorer, or even a fixed random canonical order can all structurally remove the original early/middle/end distinction.
I think the cleanest interpretation is therefore to report two properties separately:
Position invariance
Does the method remove dependence on the original retrieval slot?
Ordering quality
Does the selected canonical order improve ranking or answer quality?
This does not weaken the context-layout result. It makes it easier to see exactly what was achieved.
Stage 3 looks usable as a layout signal, but not yet calibrated as passage relevance
For Stage 3, I used separate passage encoding and aggregated the top 10% of the public sigmoid token scores.
This was not the degenerate all-100/max route: the passage scores were non-flat and normally produced five distinct passage values.
Generation results were:
| Stage 3 condition |
Relaxed EM |
| High score first |
20.0% |
| Low score first |
16.7% |
| Fixed random order |
16.7% |
| Raw-logit mean, high first |
15.0% |
That provides a small directional hint in favor of the descending Stage 3 order.
However, the passage-ranking result points in the opposite direction:
| Stage 3 direction |
Selected-passage MRR |
| High score first |
0.408 |
| Low score first |
0.537 |
So the Stage 3 checkpoint clearly produces a non-trivial signal that changes passage layout, and one natural wrapper reaches 20% without passage-ranking retraining.
But the scalar direction does not currently behave like a calibrated passage-relevance score.
My current reading would be:
Stage 3 may transfer as a general context-utility or layout signal, but the stronger interpretation—high token importance directly means high query relevance at passage granularity—still needs calibration.
This also fits the original purpose of the Stage 3 checkpoint: it was trained for KV-cache compression and retrieval preservation, not originally as a query-specific passage reranker.
The smallest useful controls here seem to be:
- always report descending and ascending;
- expose both raw logits and sigmoid scores;
- state the token-to-passage aggregation rule;
- check length-normalized aggregation;
- calibrate score direction on held-out passage-ranking data;
- report answer-bearing-passage MRR separately from generation EM.
v8b reaches the same performance band, but its direction is ambiguous
For v8b, I used mean pre-sigmoid token logits.
| v8b direction |
Relaxed EM |
Selected-passage MRR |
| High score first |
20.0% |
0.451 |
| Low score first |
20.0% |
0.454 |
The two directions frequently produced different prompts and predictions, but they ended with the same accuracy and almost identical passage-ranking MRR.
So v8b also contains a layout-sensitive signal that can reach the broad reported range, but I could not identify a stable relevance orientation from this test.
The v8b checkpoint was trained with ranking, retrieval, stability, and hard-anchor objectives, so it is not especially surprising if its scalar captures several notions of utility rather than one clean passage-relevance axis.
The dedicated checkpoint is query-conditioned, but correct-query relevance is not yet isolated
The dedicated checkpoint is definitely responsive to the query.
Replacing each correct question with a lexical-near question from another example changed:
- the top-ranked passage in 36.7% of cases;
- the complete passage order in 81.7% of cases.
So the query branch is not being ignored.
However, the correct query did not improve the relevance metrics:
True-query minus wrong-query selected-passage MRR:
-0.007
Bootstrap interval:
approximately -0.073 to +0.058
Generation was also:
| Dedicated condition |
Relaxed EM |
| Correct query |
20.0% |
| Lexical-near wrong query |
25.0% |
I would not interpret the 25% wrong-query result as evidence of a reliable inverse effect—the paired uncertainty is much too large for that.
But I also would not interpret the current result as evidence that the checkpoint has learned correct-query relevance.
The narrower conclusion is:
Does query input affect the scores?
YES
Does the correct query reliably improve the ranking?
NOT DEMONSTRATED
The TIS 2.0 post describes the dedicated route in terms of cross-attention, per-passage relevance, RMSNorm, and InfoNCE passage-pair training.
The public checkpoint is usable, and it is genuinely query-conditioned. The part that seems to need another step is alignment between the query-conditioned movement and the desired relevant-passage direction.
The most direct repair path would be:
- optimize positive-vs-negative passage margins within the same query;
- use lexical-near hard negatives;
- include correct-query versus wrong-query validation;
- subtract or normalize query-common score shifts;
- select checkpoints using passage MRR or pairwise accuracy, not only an absolute token score loss.
Simple baselines remain highly competitive
Ranking on all 60 examples looked like this:
| Route |
Selected-passage MRR |
| TF-IDF |
0.606 |
| BM25 |
0.606 |
| Passage length |
0.545 |
| Stage 3, low score first |
0.537 |
| Dedicated, wrong query |
0.478 |
| Dedicated, correct query |
0.470 |
| v8b, high score first |
0.451 |
| Stage 3, high score first |
0.408 |
Generation was much flatter:
| Route |
Relaxed EM |
| TIS checkpoint routes |
20.0% |
| Passage length |
20.0% |
| TF-IDF |
18.3% |
| BM25 |
18.3% |
| Fixed random |
16.7% |
So I would avoid reading the current experiment as evidence that learned TIS scores already dominate lexical or structural baselines.
What it does show is that several public TIS checkpoints can independently construct competitive context layouts.
The length result is also worth retaining as a control. In this particular five-passage MS-MARCO construction, selected passages appear to carry a useful length bias.
Ranking and answer extraction appear to be separate bottlenecks
The evaluation contains several stages:
token score
↓
passage score
↓
passage ranking
↓
context layout
↓
generator attention
↓
answer extraction
↓
exact match
A route can improve one stage without improving the later stages proportionally.
For example:
- TF-IDF and BM25 had the strongest passage-ranking MRR;
- their generation EM was only 18.3%;
baseline_early already places the selected/gold passage first, but reached only 18.3%;- multiple weaker ranking routes reached 20%.
That supports one of the useful observations already made in the TIS 2.0 post: reordering does not solve the remaining answer-extraction problem.
I would therefore report ranking and generation as separate endpoints:
| Ranking endpoint |
Generation endpoint |
| Selected-passage R@1 |
Relaxed EM |
| Selected-passage MRR |
Strict EM |
| Answer-bearing-passage MRR |
Token F1 |
| Descending vs. ascending |
Prediction-change rate |
| Correct vs. wrong query |
Generator or prompt sensitivity |
A two-component interpretation may fit the evidence better
One possible way to organize the current results is:
final passage score
=
intrinsic context utility
+
λ × query-specific relevance
+
μ × lexical relevance
The first term could be what Stage 3 and v8b are already partly providing:
- information density;
- structural salience;
- anchor-like content;
- generator-friendly placement;
- query-independent passage utility.
The second term would be the job of the dedicated query-aware model:
- positive passage above hard negatives;
- correct question above wrong-question controls;
- relative scoring within one candidate set;
- semantic relevance rather than a common score shift.
The third term is optional, but the TF-IDF/BM25 results show that lexical relevance is still a strong prior in this benchmark.
This decomposition would make the transfer result easier to interpret:
KV/token training
↓
intrinsic context-utility prior
↓
usable without passage retraining
query-aware passage training
↓
query-specific correction
↓
relevance calibration
At the moment, those two ideas seem to be compressed into one scalar called “importance.”
Separating them may allow the current Stage 3 result to remain useful without requiring its high-score direction to already mean query relevance.
The shortest repair paths I can see
If Stage 3 / v8b is intended as context utility:
keep descending and ascending controls
publish the intended token aggregation
expose logits and post-sigmoid scores
calibrate direction on held-out examples
report layout stability separately from relevance
If the dedicated head is intended as query relevance:
train on within-query passage pairs
use lexical-near hard negatives
normalize query-common score shifts
validate correct query against wrong queries
report passage ranking before generation
If the main result is position robustness:
describe deterministic canonicalization as the intervention
compare TIS, TF-IDF, length, and random canonical orders
report zero gap separately from accuracy gain
The missing internal route would still be useful, but it is no longer a prerequisite
The exact internal trainer and evaluator are still not uniquely identifiable from the current clean release.
That seems understandable given the explanation in post #6 that the clean releases are assembled from a messier internal working version.
But I no longer think the missing internal route prevents the broad result from being tested.
The independent public implementation already reaches the same general range.
The internal route would now mainly help answer narrower questions:
- Was Stage 3 intended to be sorted ascending or descending at passage level?
- Which token-to-passage aggregation was used?
- Were passages encoded independently or with the question/context?
- Which exact wrapper was intended for the dedicated
best.pt checkpoint?
In other words:
The missing wrapper now looks less like a prerequisite for reproducing the broad result, and more like the shortest way to identify the intended score orientation and aggregation contract.
Independent notebook and evaluation protocol
The independent public notebook pins the public source and checkpoint revisions used in the run.
The evaluator uses:
- 60 public query IDs;
- five passages per query;
- Mistral-based hidden states;
- greedy answer generation;
- relaxed EM, strict EM, and token F1;
- per-route passage orders and passage-ranking metrics;
- correct-query and lexical-near wrong-query controls;
- exact ascending controls for Stage 3 and v8b;
- TF-IDF, BM25, passage length, and deterministic random controls.
The canonicalized routes are evaluated once per query.
They are not copied into three nominal early/middle/end rows, because those copies would contain exactly the same prompt and prediction.
Only the unreordered baseline retains three position variants.
baseline_early already places the selected/gold passage in the first slot, so a second gold_first condition would be an exact duplicate and is intentionally omitted.
Stage 3 and v8b score-direction controls
Stage 3
The public direct scorer is applied to final hidden states. The main route uses post-sigmoid token scores and averages the top 10%.
The route was checked for:
- flat passage scores;
- all-100 saturation;
- unique passage values;
- descending-versus-ascending equality;
- random-order equivalence.
It was non-degenerate.
Results:
| Metric |
Descending |
Ascending |
| Generation EM |
20.0% |
16.7% |
| Selected-passage MRR |
0.408 |
0.537 |
The raw-logit mean descending route reached 15.0% EM.
v8b
The fixed route averages pre-sigmoid token logits.
| Metric |
Descending |
Ascending |
| Generation EM |
20.0% |
20.0% |
| Selected-passage MRR |
0.451 |
0.454 |
These controls make it difficult to interpret either scalar direction as universally equivalent to passage relevance.
Correct-query versus wrong-query controls
Each query was paired with the most lexically similar different query among the 60 examples.
The dedicated checkpoint was evaluated with:
- the correct query;
- that lexical-near wrong query;
- the same five passages;
- the same token aggregation;
- no checkpoint updates.
Results:
| Diagnostic |
Result |
| Full order changed |
81.7% |
| Top passage changed |
36.7% |
| True − wrong selected-MRR |
−0.007 |
| Bootstrap interval |
approximately −0.073 to +0.058 |
| Correct-query generation EM |
20.0% |
| Wrong-query generation EM |
25.0% |
This establishes functional query dependence, but not a reliable correct-query relevance benefit.
Public architecture and checkpoint observations
The current public source revision includes several neighboring routes:
- direct scoring from
ImportanceUpdateHead;
- a concat-style
QueryAwareImportanceHead;
CrossAttentionImportanceScorer;- multiple training scripts and objectives.
The public dedicated checkpoint strictly matches the concat-style query-aware head:
mean-pooled query projection
+
per-token context projection
↓
concatenation
↓
MLP
↓
sigmoid token score
This is enough to evaluate the checkpoint independently.
It does not establish which exact trainer or evaluator from the internal working version produced the published checkpoint and number.
That provenance distinction is why I treat this as an independent public implementation rather than an exact historical reproduction.
What I am not concluding
This test does not establish that:
- the TIS idea is invalid;
- the published result was fabricated;
- the public checkpoints are random or broken;
- query input is ignored;
- Stage 3 contains no transferable signal;
- TF-IDF or passage length will dominate on other datasets;
- 20.0% is meaningfully different from 21.7%;
- the independent wrapper matches the private historical evaluator;
- a good passage rank should automatically produce a correct answer;
- the scalar can never be calibrated into a useful relevance score.
The narrower conclusions are:
- the broad numerical range is independently reproducible;
- deterministic reordering removes the original slot dependence;
- Stage 3, v8b, and the dedicated checkpoint all affect context layout;
- score direction and query-semantic calibration remain unresolved;
- simple lexical and structural controls should remain in the main result table.
My current reading is therefore more positive than where I started.
The broad 16.7% → approximately 20–22% result is independently reproducible from the public artifacts, and deterministic context reordering really does remove dependence on the original retrieval slot.
The unresolved part is narrower:
Is the scalar called “importance” already a calibrated query-relevance score, or is it currently better understood as a more general context-utility signal?
The context-layout interpretation already looks useful.
The query-relevance interpretation seems to need score-direction calibration and stronger query-relative supervision.
The public notebook is linked above. The most useful comparison with the internal working route would probably be the intended passage-level score direction and token aggregation, rather than another exact-decimal reproduction run.
