Hmm… I tried it a little using only Claude Free:
It worked, including the full deterministic pass, Claude adjudication, and the three generated artifacts.
I first used a tiny synthetic document as a smoke test, then ran a more meaningful A/B comparison on Chain-of-Verification Reduces Hallucination in Large Language Models using the same Claude model, the same PDF, and separate fresh conversations.
My provisional result from this single paper was:
|
Ordinary review |
DESi Review |
| Breadth of methodological criticism |
Better in this run |
Narrower |
| Concrete questions for the authors |
Better in this run |
Fewer, but structured |
| Traceability to an extracted claim |
Informal |
Strong |
| Fixed verdict vocabulary |
No |
Yes |
| Machine-readable intermediate state |
No |
Yes |
| Visible extraction noise |
Mostly hidden |
Explicit |
| Replayable deterministic layer |
No |
Yes |
| Semantic verdict reproducibility |
Not tested |
Not guaranteed |
So, in this example, I would not say that DESi produced a better scientific review overall. The ordinary review found more of the methodological concerns I would want surfaced.
However, DESi made the review process much easier to inspect. I could see the extracted claim inventory, the items that were later rejected as NOT_A_CLAIM, the verdict assigned to every required strong claim, and the exact hashed artifacts produced by the run.
That seems like a meaningful difference between:
- generating a useful review, and
- generating a review process whose intermediate state can be inspected and compared.
This was only one real paper, one model setting, and one run per condition, so I would treat it as a worked example rather than an evaluation result.
The comparison I ran
I used Claude Sonnet 5 with the medium setting and two fresh conversations.
For the ordinary run:
Critically review the attached paper.
Identify its main claims, unsupported or overstated statements, evidence gaps,
methodological weaknesses, reproducibility concerns, limitations, and useful
questions for the authors.
For the DESi run:
Use the DESi Review skill to review the attached paper.
Follow the complete skill workflow and return all generated artifacts.
I kept the raw outputs without editing them. DESi returned:
report.mdartifact.jsonadjudication.json
That basic A/B route seems reasonable as a default:
- Same source document.
- Same model and model setting.
- Fresh conversation for each condition.
- Save the exact prompts.
- Preserve the unedited outputs and all DESi artifacts.
- Treat DESi-only and ordinary-only findings as differences requiring inspection, not automatically as true positives or false negatives.
What happened in the real-paper DESi run
The deterministic layer extracted:
- 83 claim candidates
- 58 strong-claim candidates
- 46 evidence gaps
- 7 overclaim-risk hits
- 4 reproducibility-risk categories
The Claude adjudication over the 58 strong candidates produced:
| Verdict |
Count |
SUPPORTED |
30 |
NOT_A_CLAIM |
25 |
OVERSTATED |
2 |
UNSUPPORTED |
1 |
The large NOT_A_CLAIM count was informative.
It does not establish a general extraction precision of 33/58: this is one paper, the adjudicator can itself be wrong, and the denominator is the selected strong-claim set rather than all extracted items.
But it does show that, for this document, the semantic layer had substantial cleanup work to do after deterministic extraction. That cleanup burden is normally difficult to see in an ordinary fluent review.
The seven deterministic overclaim-risk hits were all triggered by the word first, including procedural uses such as “first step” and “first compare.” This appears consistent with the measured defect list already documented for the skill: lexical extraction is intentionally simple and can promote definitions, setup descriptions, or rhetorical and procedural sentences into claim candidates.
That did not prevent the workflow from completing, because Claude could classify those candidates as NOT_A_CLAIM. Still, I found it useful that the upstream noise remained visible rather than disappearing inside the final prose.
Where the ordinary review was stronger in this run
The ordinary review surfaced several methodological issues more directly and in more actionable form.
1. Inference-budget comparability
The paper compares a multi-stage, potentially many-call CoVe process against several single-call or differently prompted baselines.
The ordinary review asked whether part of the gain should be interpreted as a method improvement, and how much should instead be attributed to a substantially larger inference budget.
The paper does discuss prompt-count overhead, but a measured wall-clock, token, or cost comparison would make this trade-off easier to interpret.
2. Precision versus saying less
For long-form biographies, the reported FACTSCORE rises while the average number of generated facts falls from 16.6 to 12.3.
Likewise, on the Wikidata task, hallucinated entities fall substantially, but correct entities also fall.
The paper reports these values, but the ordinary review foregrounded the distinction between:
- correcting incorrect content while preserving useful content, and
- improving precision partly by suppressing content.
That seems important when interpreting the central result.
3. Correlated self-verification errors
The same base model creates the response and answers the verification questions without external grounding.
The ordinary review noted that the model may therefore preserve systematic errors shared across both stages, especially for rare facts. The paper’s own rarity breakdown is compatible with this concern: the improvement is weakest for very rare facts, where external knowledge or retrieval is most necessary.
4. Model and prompt generality
Most central experiments use Llama 65B.
The ordinary review asked about:
- other model families and sizes,
- sensitivity to the few-shot demonstrations,
- prompt ablations,
- and whether the joint/factored differences survive across current instruction-tuned models.
5. Statistical uncertainty
Several test sets are relatively small, and the paper reports point estimates without confidence intervals, repeated-run variance, or significance tests.
The ordinary review treated small differences such as the relative ordering of factored and two-step variants cautiously.
6. Dependence on an automated factuality judge
The long-form result depends on FACTSCORE, which itself uses a retrieval-and-model-based factuality pipeline.
The ordinary review asked whether errors in that evaluation layer had been manually sampled or independently checked.
7. Baseline symmetry
The ordinary review questioned the comparison between a few-shot base Llama condition and zero-shot or zero-shot-CoT instruction-tuned conditions.
The paper explains some prompt-format decisions, but a matched few-shot instruction-tuned baseline would separate the effects more cleanly.
These are candidate review findings rather than a gold-standard judgment. Still, most could be connected directly to the paper’s tables, experimental setup, or stated limitations, and they were more explicit in the ordinary review than in the DESi output.
Where DESi added something useful
DESi’s most useful contribution in this run was not a larger list of criticisms. It was the explicit decomposition of the review state.
For each required candidate, I could inspect:
- claim ID,
- extracted text,
- assigned category,
- deterministic support signals,
- Claude verdict,
- short reason,
- verdict tally,
- source-ledger hash,
- and final run hash.
This made it possible to distinguish several different events that an ordinary review tends to merge:
- A sentence was never extracted.
- A sentence was extracted under the wrong category.
- A candidate was correctly rejected as
NOT_A_CLAIM.
- A real claim was found but its evidence was judged insufficient.
- A claim was judged supported but may still have been missed by the ordinary review.
- The final review prose omitted something present in the intermediate analysis.
One DESi structural question was especially useful.
The CoVe paper reports that the model answers isolated short factual questions much more accurately than it produces those facts inside a longer response. DESi asked, in effect:
Is this specifically a benefit of the verification framing, or would any isolated short re-questioning of the same fact produce the same gain?
That suggests a relatively cheap control:
- generate the original long-form answer,
- ask the same fact as an isolated short question without calling it verification,
- compare that with the full CoVe verification procedure.
Such a control could help separate:
- a genuine verification-workflow effect,
- from a general short-question versus long-generation effect.
The ordinary review did not formulate that control as clearly.
A useful interpretation of “complete”
One boundary became clearer after running the implementation.
The emitter enforces completeness over the set of extracted strong claims required by the protocol. In the current skill these are the selected:
main_claimresult_claimnovelty_claimgeneralization_claim
items.
That is a useful guarantee:
Every required item in the current ledger received an allowed verdict.
It is not the stronger guarantee:
Every important claim or review issue in the paper was discovered.
I found it helpful to think of this as closed-world completeness: completeness inside the extracted ledger.
This is not a defect in the idea of a gate. It is the natural boundary of any gated pipeline whose upstream inventory may contain both false positives and false negatives. The benefit is that the boundary becomes inspectable and can be measured separately.
A practical evaluation could therefore report at least three quantities independently:
-
Extraction coverage
Which important source claims entered the ledger?
-
Adjudication quality
Were the extracted candidates classified appropriately?
-
Review usefulness
Did the resulting output identify important, correct, actionable issues?
Otherwise, a successful completion gate can be mistaken for successful document coverage.
Replay also has two meanings
The deterministic paper-review artifact reproduced the same replay hash when rerun on the same normalized input.
The Claude adjudication artifact, however, correctly records that semantic adjudication is not replay-stable. Its hash identifies the artifact generated in that run; it does not claim that another Claude run must return the same verdicts.
I think that distinction is valuable:
- artifact identity: these are the exact bytes and verdicts produced in this run;
- procedural replay: the deterministic transformation can be reproduced;
- semantic stability: repeated model adjudication produces equivalent judgments.
The first two are partly addressed by the current artifacts. The third would need repeated runs.
For accumulating A/B reports over time, it may also help to bind the adjudication artifact to as many of these as practical:
- source-document hash,
- PDF/text conversion method,
- DESi commit,
- DESi-Paper-Review commit,
- Skill package or protocol version,
- exact model identifier and setting,
- raw model response,
- and execution timestamp.
The current artifact already records important ledger and verdict hashes, but the additional provenance would make comparisons across future model and Skill versions easier.
Small smoke test that helped me understand the two layers
Before using a real paper, I used a deliberately thin document containing three statements:
# Abstract
We introduce a novel review method that improves claim coverage.
# Results
Our method achieved 92% accuracy on a dataset of 1,000 examples.
# Conclusion
These results demonstrate that the method generalizes to all scientific domains.
The deterministic layer recognized the numeric result sentence as having an inline support signal because it contained numbers.
Claude still classified it as UNSUPPORTED, because the document supplied no task definition, dataset description, measurement protocol, evaluation setup, or baseline.
That was a useful demonstration of the intended separation:
- the deterministic layer can record observable textual signals;
- the semantic layer can decide that those signals do not constitute adequate evidence.
The universal generalization was classified as OVERSTATED, while the novelty/improvement claim was UNSUPPORTED.
The ordinary review was much more verbose and added missing-method, baseline, statistical, dataset, and reproducibility questions. DESi was less expansive, but it provided a compact, governed claim-by-claim result.
Installation and first-run observations
The first run was not completely frictionless, although it ultimately succeeded on Claude Free.
Free-plan availability
The post currently lists a paid Claude plan as a requirement. Anthropic’s current Custom Skills documentation says Skills are available on Free, Pro, Max, Team, and Enterprise plans when code execution is enabled.
My successful run was on Claude Free, so the paid-plan requirement may simply need a freshness update.
The upload route matched the current Skill installation instructions:
Customize → Skills → + → Create skill → Upload a skill
Network access
On the first attempt, the sandbox could not reach GitHub.
Claude stopped rather than inventing a deterministic artifact by hand. After GitHub access became available, the workflow could proceed.
So the requirement for one-time repository access per session was real in this test. A future reader who sees an initial clone failure should probably treat network egress as the first branch to check, rather than interpreting it as a clean DESi result.
Inspection before installation
After GitHub became reachable, Claude initially declined to run editable installs from repositories it had not inspected.
It proceeded after I selected a clone-and-inspect-first route. It checked the build configuration and searched the execution path for obvious network, subprocess, and dynamic-execution behavior before installing.
This added several turns to the first run, but it also seemed like reasonable behavior for a third-party Skill containing executable scripts. Anthropic’s documentation describes Skills as potentially containing executable code, so first-run inspection is a foreseeable part of the experience rather than necessarily a DESi-specific malfunction.
A practical description might therefore be:
- close to a five-minute path when code execution, network access, and repository trust are already resolved;
- potentially longer on a first run where Claude performs a safety inspection.
Once installation completed, the doctor check returned:
DESI_PAPER_REVIEW_MVP_READY
core_identity = 1.0
and the full review completed.
Additional deterministic-layer checks
Separately from the Claude A/B run, I ran the public deterministic layer on a Colab Free CPU environment.
At the commit I tested, the repository’s:
- 25 tests,
doctor,- and replay smoke test
all passed.
The current public implementation is available in DESi-Paper-Review, and it consistently labels its output REVIEW_ASSISTANCE_ONLY, which seems like the right boundary for the present system.
A few small boundary cases also illustrated what the current rule layer is sensitive to:
- moving a citation from the claim sentence to the next sentence changed the inline-support result;
- adding a year could create an inline-support signal;
- changing an
Introduction heading to Background could remove a candidate;
Fig. could interact with sentence splitting;- a novelty phrase inside References could become a candidate;
- a non-English text could produce zero claims;
- a position-style text could trigger empirical reproducibility warnings.
These are broadly consistent with the limitations already documented on the DESi applications page. I would treat them as useful regression-corpus seeds rather than surprising failures.
The emitter tests also suggested that the current gate is best described as a minimum required strong-claim coverage gate.
It rejected:
- a missing required strong-claim verdict;
- and a verdict outside the four permitted labels.
Some other written-protocol constraints did not appear to be enforced as a complete machine schema in the tested version. For example, an unknown claim ID or some additional malformed metadata could pass the minimum gate.
This may be entirely intentional for a first package. If fuller protocol enforcement is a goal, explicit schema validation for claim IDs, interest enums, required structural fields, and reason constraints could make the contract even clearer.
Where I think this leaves the experiment
For me, this A/B supports a narrower and more interesting claim than “DESi writes a better review.”
It suggests that explicit architecture can change what is observable about the review process, even when it does not produce the broadest final critique.
In this run:
- ordinary prompting produced the richer scientific review;
- DESi produced the more inspectable review state;
- the deterministic extractor introduced visible noise;
- Claude removed much of that noise;
- the completion gate guaranteed coverage of the selected ledger;
- and the artifacts made the boundary between extraction, adjudication, and final interpretation easier to discuss.
That seems useful because ordinary review quality and auditability are not the same objective.
Possible next evaluation routes
A simple decision tree may help keep future tests comparable.
If the main goal is auditability
The current A/B is already informative.
Measure:
- whether every required ledger item is adjudicated;
- whether the result can be traced to source text;
- whether artifacts preserve versions and hashes;
- whether two people can compare runs without reconstructing the model’s reasoning from prose.
If the main goal is better scientific-review recall
Use papers with a small human-checked issue set, or introduce controlled defects into otherwise realistic documents.
Measure separately:
- important issues found,
- unsupported warnings,
- missed issues,
- and reviewer effort required to verify the output.
Neither the ordinary review nor DESi-only findings should be treated as the gold set.
If the main goal is identifying which architectural component helps
A four-condition ablation would be more diagnostic than a two-condition A/B:
- ordinary review;
- ordinary review plus the same four-verdict checklist;
- claim ledger plus per-claim review, but no hard completion gate;
- full DESi Review.
That could help separate the effects of:
- a more detailed instruction,
- claim inventory,
- item-by-item adjudication,
- and enforced completion.
If the main goal is improving extraction
A small checked regression corpus may produce more information than another large-paper run.
Useful cases include:
- definitions,
- section headers,
- procedural uses of
first,
- adjacent-sentence evidence,
- figure/table evidence,
- References,
- rhetorical questions,
- negative claims,
- multi-sentence claims,
- non-empirical genres,
- and non-English text.
Each case could specify:
- expected extracted span,
- expected category,
- whether it is a claim,
- and whether evidence is local, elsewhere in the document, or unavailable.
Bottom line
This was a successful first real run on Claude Free.
I found a mixed result rather than a simple win:
- the ordinary review was broader and more methodologically useful;
- DESi was more explicit, auditable, and machine-comparable;
- the extractor produced substantial but visible candidate noise;
- the semantic layer removed much of it;
- and the generated artifacts made it possible to locate where disagreements entered the pipeline.
That seems aligned with the question in the original post: not whether one model suddenly becomes an infallible reviewer, but what changes when the same model is placed inside an explicit epistemic workflow.
For this one paper, the largest change I observed was not “better judgment.” It was better inspectability of how the judgment was assembled.
: