Add VibeThinker-trained (SLERP d=.5 t=.5) to tables and plot; document how it differs from base and the multi-model pipeline
Browse files- README.md +144 -53
- benchmarks.png +2 -2
README.md
CHANGED
|
@@ -35,7 +35,9 @@ gate, strict-7 and six-lane average of any arm in the tables below for which eac
|
|
| 35 |
computed, including Qwen3-8B, Qwen3.5-4B, VibeThinker-3B and gpt-oss-120b (the 120B has no gate or strict-7
|
| 36 |
value).
|
| 37 |
|
| 38 |
-
|
|
|
|
|
|
|
| 39 |
|
| 40 |
## Highlights
|
| 41 |
|
|
@@ -62,6 +64,10 @@ value).
|
|
| 62 |
on Track B, where VibeThinker-3B is ahead on the 10-dataset macro (0.8097 against 0.7901) and
|
| 63 |
TwIL-LM3-Pro is ahead on the 14-dataset macro (0.7425 against 0.7262). That 14-dataset lead
|
| 64 |
comes entirely from BBH-logic (0.9540 against 0.6107); without that row TwIL-LM3-Pro trails.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
* **Structured formal output.** Tuned for the objects rather than the prose: FOL translation,
|
| 66 |
entailment labels, semantic parses, Lean statements and Lean proof critique.
|
| 67 |
* **Runs anywhere.** 3.66B parameters in bf16 (6.82 GiB), with a Q4\_K\_M GGUF at 2.09 GiB for
|
|
@@ -109,23 +115,23 @@ dedicated decode-throughput protocol: `ans/s` is defined throughout as
|
|
| 109 |
`tok/s ÷ mean generation length`, so it measures completed answers rather than raw decode rate.
|
| 110 |
Cells marked † need the engine note below.
|
| 111 |
|
| 112 |
-
| lane / metric | TwIL-LM3-Pro | Granite-4.2-3B base | VibeThinker-3B | Qwen3.5-4B | TwIL-LM3 | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ |
|
| 113 |
-
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 114 |
-
| lean_formalize token_f1 | 0.5092 | 0.2943 | 0.2087 | 0.4996 | 0.5869 | 0.3690 | 0.1321 | 0.4655 | 0.4022 | **0.6306** |
|
| 115 |
-
| rule_induction derivation | 0.4195 | 0.2267 | 0.2038 | 0.5078 | 0.3192 | 0.0825 | 0.0615 | 0.1936 | 0.3680 | **0.6518** |
|
| 116 |
-
| entailment_label accuracy | 0.6700 | 0.3000 | 0.5500 | 0.2400 | 0.5750 | 0.3300 | 0.4700 | 0.5400 | 0.5800 | **0.7750** |
|
| 117 |
-
| mcq_answer accuracy | **0.4100** | 0.1200 | 0.1700 | 0.0000 | 0.1100 | 0.0000 | 0.0150 | 0.0750 | 0.0000 | 0.0700 |
|
| 118 |
-
| semantic_parse token_f1 | 0.4295 | 0.3910 | 0.2422 | 0.3826 | **0.4416** | 0.3102 | 0.3665 | 0.3778 | 0.4257 | 0.4331 |
|
| 119 |
-
| lean_critic accuracy | **0.7950** | **0.7950** | 0.6000 | 0.5400 | 0.6600 | 0.5300 | 0.5900 | 0.5500 | **0.7950** | 0.5550 |
|
| 120 |
-
| lm_corpus perplexity ↓ | **2.3130** | 2.6334 | 16.9253 | 2.3916 | 2.8972 | 2.8478 | 4.3815 | 4.9472 | 2.5440 | 912.23 § |
|
| 121 |
-
| math_corpus perplexity ↓ | 3.6983 | 4.4864 | 27.0318 | **3.5926** | 3.8229 | 4.7531 | 6.7472 | 8.3323 | 4.0083 | 1045.63 § |
|
| 122 |
-
| average, 6 lanes | **0.5389** | 0.3545 | 0.3291 | 0.3617 | 0.4488 | 0.2703 | 0.2725 | 0.3670 | 0.4285 | 0.5192 |
|
| 123 |
-
| **macro gate** | **0.5539** | 0.4313 | 0.4118 | 0.4466 | 0.4218 | 0.2925 | 0.3473 | 0.3757 | 0.5336 | — |
|
| 124 |
-
| **strict-7** | **0.2879** | 0.1821 | 0.2021 | 0.1121 | 0.1971 | 0.1229 | 0.1579 | 0.1714 | 0.2093 | — |
|
| 125 |
-
| macro_primary | **0.5875** | 0.4825 | 0.4637 | 0.4313 | 0.4475 | 0.3450 | 0.4188 | 0.4213 | 0.5750 | — |
|
| 126 |
-
| tok/s | 21169 † | 21916 † | 28010 † | 11097 † | 15880 | 16160 | 25230 | 22480 | 9420 | 3374 |
|
| 127 |
-
| mean gen length | 1902 | 2951 | 2688 | 2486 | **564** | 696 | 2296 | 1830 | 2094 | 1005 |
|
| 128 |
-
| **ans/s** | 11.1 † | 7.4 † | 10.4 † | 4.5 † | **28.1** | 23.2 | 10.9 | 12.0 | 4.5 | 3.4 |
|
| 129 |
|
| 130 |
‡ **gpt-oss-120b** runs MXFP4 weights at tensor-parallel 2 — quantized and multi-GPU, so its
|
| 131 |
throughput rows are not directly comparable to the single-GPU BF16 arms. Its `procedural` lane
|
|
@@ -136,6 +142,19 @@ average cannot be computed for it; that is what the — cells mean, not a zero.
|
|
| 136 |
format and tokenizer make the corpus lanes score a different quantity. The number is reported
|
| 137 |
for completeness but is not a comparable measurement, and is excluded from the bolding.
|
| 138 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
† **Throughput for TwIL-LM3-Pro, its base and VibeThinker-3B** was measured with the same
|
| 140 |
dedicated protocol and prompt file as the peer columns (128 prompts × 512 generated tokens, EOS
|
| 141 |
ignored, greedy, `gpu_memory_utilization` 0.45, idle GPU), run on a single H200 with vLLM
|
|
@@ -204,27 +223,27 @@ spots in absolute terms are `procedural` (strict 0.1200, loose 0.2350) and FOL t
|
|
| 204 |
|
| 205 |
### Track B — held-out benchmarks
|
| 206 |
|
| 207 |
-
| dataset | TwIL-LM3-Pro | Granite-4.2-3B base | VibeThinker-3B | Qwen3.5-4B | TwIL-LM3 | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ |
|
| 208 |
-
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 209 |
-
| gsm8k | 0.9433 | 0.9533 | 0.9600 | 0.8633 | 0.8733 | 0.8300 | 0.8767 | 0.9133 | 0.9567 | **0.9767** |
|
| 210 |
-
| svamp |
|
| 211 |
-
| gsm_symbolic | 0.8267 | 0.8900 | 0.8167 | 0.7767 | 0.7567 | 0.8067 | **0.9767** | 0.9267 | 0.8133 | 0.8467 |
|
| 212 |
-
| arc_cot | 0.8600 | 0.8933 | 0.9033 | 0.9267 | 0.8467 | 0.7967 | 0.8667 | 0.9033 | 0.9633 | **0.9667** |
|
| 213 |
-
| logicbench | 0.7933 | 0.7733 | **0.8600** | 0.7200 | 0.7167 | 0.5733 | 0.6267 | 0.7200 | 0.8567 | 0.8533 |
|
| 214 |
-
| strategyqa | 0.5967 | 0.6267 | 0.6467 | 0.6967 | 0.6500 | 0.6533 | 0.6433 | 0.6667 | 0.7400 | **0.7867** |
|
| 215 |
-
| drop | 0.7367 | 0.7467 | 0.8467 | 0.6233 | 0.7467 | 0.6733 | 0.6900 | 0.6633 | **0.8833** | 0.8500 |
|
| 216 |
-
| csqa | 0.7667 | 0.7733 | 0.7367 | 0.7767 | 0.7367 | 0.7500 | 0.7433 | 0.7700 | **0.8633** | 0.8367 |
|
| 217 |
-
| musr | 0.6409 | 0.5922 | 0.5799 | 0.5696 | 0.4957 | 0.4932 | 0.4867 | 0.5703 | 0.6301 | **0.6852** |
|
| 218 |
-
| mmlu_redux | 0.7867 | 0.7733 | 0.8100 | 0.8433 | 0.6667 | 0.6000 | 0.7133 | 0.8367 | 0.8500 | **0.9467** |
|
| 219 |
-
| ifeval | 0.7500 | 0.7633 | 0.6633 | 0.2400 ¶¶¶ | 0.6433 | 0.7167 | 0.7300 | **0.8900** | 0.8400 | 0.7900 |
|
| 220 |
-
| rudas_ood | 0.0437 ¶¶ | 0.0012 | 0.0056 | 0.0084 ¶¶¶ | 0.0365 | **0.0733** | 0.0017 | 0.0061 | 0.0468 | 0.0000 ¶ |
|
| 221 |
-
| bbh_logic | 0.9540 | 0.9013 | 0.6107 | 0.9647 | 0.6633 | 0.5333 | 0.5713 | 0.7700 | 0.6367 | **0.9980** |
|
| 222 |
-
| math500 | 0.7467 | 0.6567 | 0.7900 | 0.3600 ¶¶¶ | 0.6900 | 0.4233 | 0.7133 | 0.7800 | 0.6100 | **0.8433** |
|
| 223 |
-
| **macro (10 CoT datasets)** | 0.7901 | 0.7942 | 0.8097 | 0.7683 | 0.7339 | 0.6997 | 0.7523 | 0.7884 | 0.8493 | **0.8689** |
|
| 224 |
-
| **macro (all 14)** | 0.7425 | 0.7332 | 0.7262 | 0.6611 | 0.6694 | 0.6245 | 0.6814 | 0.7378 | 0.7591 | **0.8086** |
|
| 225 |
-
| tok/s | 21169 † | 21916 † | 28010 † | 11097 † | 15880 | 16160 | 25230 | 22480 | 9420 | 3374 |
|
| 226 |
-
| mean gen length | ≈792 | ≈1282 | ≈1789 | ≈2787 | **482** | 510 | ≈796 | ≈1327 | ≈1931 | 801 |
|
| 227 |
-
| **ans/s** | 26.7 † | 17.1 † | ≈15.7 † | ≈4.0 † | **32.9** | 31.7 | ≈31.7 | ≈16.9 | 4.9 | 4.2 |
|
| 228 |
|
| 229 |
‡ MXFP4 weights, tensor-parallel 2 — quantized and multi-GPU, so not directly comparable to the
|
| 230 |
single-GPU BF16 rows. ¶ 74% of its `rudas_ood` generations hit the length cap, so that cell is a
|
|
@@ -250,7 +269,8 @@ extracting anyway. First, it holds its own base on the held-out suite (10-datase
|
|
| 250 |
against 0.7942, a difference well inside the sampling noise at n = 300 per dataset) while gaining
|
| 251 |
in-domain, which is the point of the WiSE-FT stage. Second, the 14-dataset macro rises 0.0093
|
| 252 |
over the base, driven by BBH-logic, MATH-500 and MuSR. Third, it edges LFM2.5-8B-A1B on both
|
| 253 |
-
macros at less than half the parameters and leads
|
|
|
|
| 254 |
|
| 255 |
VibeThinker-3B is the stronger held-out model on the 10-dataset macro (0.8097 against 0.7901). It
|
| 256 |
is ahead of TwIL-LM3-Pro on seven datasets — GSM8K, ARC, LogicBench, StrategyQA, DROP,
|
|
@@ -270,25 +290,67 @@ MuSR-team sit slightly above the 2% cap-hit threshold (3.0%, 3.0% and 2.8%).
|
|
| 270 |
### Against the tuned VibeThinker-3B
|
| 271 |
|
| 272 |
The tables above use the public VibeThinker-3B checkpoint. The same post-training pipeline was also
|
| 273 |
-
applied to it, and
|
| 274 |
-
|
| 275 |
-
|
|
|
|
| 276 |
|
| 277 |
| model | macro gate | macro_primary | B10 | B14 | Track A truncation |
|
| 278 |
|---|---:|---:|---:|---:|---:|
|
| 279 |
| TwIL-LM3-Pro | **0.554** | **0.588** | 0.790 | **0.743** | 24.2% |
|
|
|
|
| 280 |
| VibeThinker-3B, WiSE-FT λ = 0.50 | 0.541 | **0.588** | **0.802** | 0.728 ◊ | 14.3% |
|
| 281 |
|
| 282 |
-
◊ There is no B14 row for the λ = 0.50 configuration; the figure is the
|
| 283 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 284 |
|
| 285 |
-
|
| 286 |
-
|
| 287 |
-
|
| 288 |
-
|
| 289 |
-
|
| 290 |
-
|
| 291 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 292 |
|
| 293 |
### Checkpoint selection
|
| 294 |
|
|
@@ -399,6 +461,34 @@ Five stages on top of the base model:
|
|
| 399 |
γ = 3.0. The run was resumed at step 800 with β = 0.02 and trained through step 2580, and the
|
| 400 |
published checkpoint is **step 2580**. This release is the merged policy, not a LoRA adapter.
|
| 401 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 402 |
## Limitations and caveats
|
| 403 |
|
| 404 |
|
|
@@ -419,7 +509,8 @@ card, which describes the same harness. For Track B, the arms checked (including
|
|
| 419 |
and Qwen3.5-4B, on all 18 tasks) share the same sampled rows and decoding, but the serving engine differs between
|
| 420 |
arms (vLLM 0.19.1 for TwIL-LM3-Pro, its base, Qwen3.5-4B, Qwen3-8B and LFM2.5-8B-A1B; vLLM 0.11.2 for
|
| 421 |
TwIL-LM3, Llama-3.2-3B and VibeThinker-3B), and the engine version is part of the protocol hash.
|
| 422 |
-
|
|
|
|
| 423 |
n = 200 per lane on Track A and n = 300 per dataset on Track B, differences of two to three points
|
| 424 |
are within sampling noise.
|
| 425 |
|
|
|
|
| 35 |
computed, including Qwen3-8B, Qwen3.5-4B, VibeThinker-3B and gpt-oss-120b (the 120B has no gate or strict-7
|
| 36 |
value).
|
| 37 |
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
*VibeThinker-trained is the public VibeThinker-3B after the same post-training pipeline (SLERP, d = 0.5, t = 0.5); see [Against the tuned VibeThinker-3B](#against-the-tuned-vibethinker-3b). It has no strict-7 or six-lane-average value, so those bars show n/a.*
|
| 41 |
|
| 42 |
## Highlights
|
| 43 |
|
|
|
|
| 64 |
on Track B, where VibeThinker-3B is ahead on the 10-dataset macro (0.8097 against 0.7901) and
|
| 65 |
TwIL-LM3-Pro is ahead on the 14-dataset macro (0.7425 against 0.7262). That 14-dataset lead
|
| 66 |
comes entirely from BBH-logic (0.9540 against 0.6107); without that row TwIL-LM3-Pro trails.
|
| 67 |
+
* **The pipeline is not tied to one model.** The same recipe was run on five base models. On
|
| 68 |
+
VibeThinker-3B it lifts the Track A macro gate from 0.374 to 0.508 (SLERP, the
|
| 69 |
+
*VibeThinker-trained* column) while the Track B 10-dataset macro moves from 0.815 to 0.802 — see
|
| 70 |
+
[One pipeline, several models](#one-pipeline-several-models).
|
| 71 |
* **Structured formal output.** Tuned for the objects rather than the prose: FOL translation,
|
| 72 |
entailment labels, semantic parses, Lean statements and Lean proof critique.
|
| 73 |
* **Runs anywhere.** 3.66B parameters in bf16 (6.82 GiB), with a Q4\_K\_M GGUF at 2.09 GiB for
|
|
|
|
| 115 |
`tok/s ÷ mean generation length`, so it measures completed answers rather than raw decode rate.
|
| 116 |
Cells marked † need the engine note below.
|
| 117 |
|
| 118 |
+
| lane / metric | TwIL-LM3-Pro | Granite-4.2-3B base | VibeThinker-3B | VibeThinker-trained ★ | Qwen3.5-4B | TwIL-LM3 | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ |
|
| 119 |
+
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 120 |
+
| lean_formalize token_f1 | 0.5092 | 0.2943 | 0.2087 | 0.527 | 0.4996 | 0.5869 | 0.3690 | 0.1321 | 0.4655 | 0.4022 | **0.6306** |
|
| 121 |
+
| rule_induction derivation | 0.4195 | 0.2267 | 0.2038 | 0.227 | 0.5078 | 0.3192 | 0.0825 | 0.0615 | 0.1936 | 0.3680 | **0.6518** |
|
| 122 |
+
| entailment_label accuracy | 0.6700 | 0.3000 | 0.5500 | 0.685 | 0.2400 | 0.5750 | 0.3300 | 0.4700 | 0.5400 | 0.5800 | **0.7750** |
|
| 123 |
+
| mcq_answer accuracy | **0.4100** | 0.1200 | 0.1700 | — | 0.0000 | 0.1100 | 0.0000 | 0.0150 | 0.0750 | 0.0000 | 0.0700 |
|
| 124 |
+
| semantic_parse token_f1 | 0.4295 | 0.3910 | 0.2422 | 0.392 | 0.3826 | **0.4416** | 0.3102 | 0.3665 | 0.3778 | 0.4257 | 0.4331 |
|
| 125 |
+
| lean_critic accuracy | **0.7950** | **0.7950** | 0.6000 | 0.685 | 0.5400 | 0.6600 | 0.5300 | 0.5900 | 0.5500 | **0.7950** | 0.5550 |
|
| 126 |
+
| lm_corpus perplexity ↓ | **2.3130** | 2.6334 | 16.9253 | 7.350 | 2.3916 | 2.8972 | 2.8478 | 4.3815 | 4.9472 | 2.5440 | 912.23 § |
|
| 127 |
+
| math_corpus perplexity ↓ | 3.6983 | 4.4864 | 27.0318 | 7.000 | **3.5926** | 3.8229 | 4.7531 | 6.7472 | 8.3323 | 4.0083 | 1045.63 § |
|
| 128 |
+
| average, 6 lanes | **0.5389** | 0.3545 | 0.3291 | — | 0.3617 | 0.4488 | 0.2703 | 0.2725 | 0.3670 | 0.4285 | 0.5192 |
|
| 129 |
+
| **macro gate** | **0.5539** | 0.4313 | 0.4118 | 0.508 | 0.4466 | 0.4218 | 0.2925 | 0.3473 | 0.3757 | 0.5336 | — |
|
| 130 |
+
| **strict-7** | **0.2879** | 0.1821 | 0.2021 | — | 0.1121 | 0.1971 | 0.1229 | 0.1579 | 0.1714 | 0.2093 | — |
|
| 131 |
+
| macro_primary | **0.5875** | 0.4825 | 0.4637 | 0.579 | 0.4313 | 0.4475 | 0.3450 | 0.4188 | 0.4213 | 0.5750 | — |
|
| 132 |
+
| tok/s | 21169 † | 21916 † | 28010 † | — | 11097 † | 15880 | 16160 | 25230 | 22480 | 9420 | 3374 |
|
| 133 |
+
| mean gen length | 1902 | 2951 | 2688 | — | 2486 | **564** | 696 | 2296 | 1830 | 2094 | 1005 |
|
| 134 |
+
| **ans/s** | 11.1 † | 7.4 † | 10.4 † | — | 4.5 † | **28.1** | 23.2 | 10.9 | 12.0 | 4.5 | 3.4 |
|
| 135 |
|
| 136 |
‡ **gpt-oss-120b** runs MXFP4 weights at tensor-parallel 2 — quantized and multi-GPU, so its
|
| 137 |
throughput rows are not directly comparable to the single-GPU BF16 arms. Its `procedural` lane
|
|
|
|
| 142 |
format and tokenizer make the corpus lanes score a different quantity. The number is reported
|
| 143 |
for completeness but is not a comparable measurement, and is excluded from the bolding.
|
| 144 |
|
| 145 |
+
★ **VibeThinker-trained** is the public VibeThinker-3B after the same post-training pipeline as
|
| 146 |
+
TwIL-LM3-Pro, in its SLERP (d = 0.5, t = 0.5) configuration; it is not a checkpoint in this
|
| 147 |
+
repository, and [the section below](#against-the-tuned-vibethinker-3b) explains how it differs
|
| 148 |
+
from the base model. Its figures are taken from the internal family comparison tables, to three
|
| 149 |
+
decimals, not from the per-lane reports behind the other columns, so its column is indicative
|
| 150 |
+
rather than strictly paired with them. `macro gate` and `macro_primary` are the family tables'
|
| 151 |
+
"macro primary (rule)" and "primary" columns, which have the same definitions as above. The
|
| 152 |
+
family tables score `mcq_answer` with loose-match credit (0.695), so the strict MCQ row, the
|
| 153 |
+
six-lane average and strict-7 — which all need strict scoring — are left as —. Its perplexities
|
| 154 |
+
come from that separate run; the family table records the untuned VibeThinker-3B at 18.70 and
|
| 155 |
+
21.80 there, against 16.93 and 27.03 in the columns above, so read them within their own source.
|
| 156 |
+
Throughput and generation length were not measured for it.
|
| 157 |
+
|
| 158 |
† **Throughput for TwIL-LM3-Pro, its base and VibeThinker-3B** was measured with the same
|
| 159 |
dedicated protocol and prompt file as the peer columns (128 prompts × 512 generated tokens, EOS
|
| 160 |
ignored, greedy, `gpu_memory_utilization` 0.45, idle GPU), run on a single H200 with vLLM
|
|
|
|
| 223 |
|
| 224 |
### Track B — held-out benchmarks
|
| 225 |
|
| 226 |
+
| dataset | TwIL-LM3-Pro | Granite-4.2-3B base | VibeThinker-3B | VibeThinker-trained ★ | Qwen3.5-4B | TwIL-LM3 | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ |
|
| 227 |
+
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 228 |
+
| gsm8k | 0.9433 | 0.9533 | 0.9600 | 0.930 | 0.8633 | 0.8733 | 0.8300 | 0.8767 | 0.9133 | 0.9567 | **0.9767** |
|
| 229 |
+
| svamp | 0.9500 | 0.9200 | 0.9367 | **0.953** | 0.8867 | 0.8500 | 0.8200 | 0.9000 | 0.9133 | 0.9400 | 0.9400 |
|
| 230 |
+
| gsm_symbolic | 0.8267 | 0.8900 | 0.8167 | 0.883 | 0.7767 | 0.7567 | 0.8067 | **0.9767** | 0.9267 | 0.8133 | 0.8467 |
|
| 231 |
+
| arc_cot | 0.8600 | 0.8933 | 0.9033 | 0.877 | 0.9267 | 0.8467 | 0.7967 | 0.8667 | 0.9033 | 0.9633 | **0.9667** |
|
| 232 |
+
| logicbench | 0.7933 | 0.7733 | **0.8600** | 0.850 | 0.7200 | 0.7167 | 0.5733 | 0.6267 | 0.7200 | 0.8567 | 0.8533 |
|
| 233 |
+
| strategyqa | 0.5967 | 0.6267 | 0.6467 | 0.647 | 0.6967 | 0.6500 | 0.6533 | 0.6433 | 0.6667 | 0.7400 | **0.7867** |
|
| 234 |
+
| drop | 0.7367 | 0.7467 | 0.8467 | 0.790 | 0.6233 | 0.7467 | 0.6733 | 0.6900 | 0.6633 | **0.8833** | 0.8500 |
|
| 235 |
+
| csqa | 0.7667 | 0.7733 | 0.7367 | 0.687 | 0.7767 | 0.7367 | 0.7500 | 0.7433 | 0.7700 | **0.8633** | 0.8367 |
|
| 236 |
+
| musr | 0.6409 | 0.5922 | 0.5799 | 0.588 | 0.5696 | 0.4957 | 0.4932 | 0.4867 | 0.5703 | 0.6301 | **0.6852** |
|
| 237 |
+
| mmlu_redux | 0.7867 | 0.7733 | 0.8100 | 0.813 | 0.8433 | 0.6667 | 0.6000 | 0.7133 | 0.8367 | 0.8500 | **0.9467** |
|
| 238 |
+
| ifeval | 0.7500 | 0.7633 | 0.6633 | 0.557 | 0.2400 ¶¶¶ | 0.6433 | 0.7167 | 0.7300 | **0.8900** | 0.8400 | 0.7900 |
|
| 239 |
+
| rudas_ood | 0.0437 ¶¶ | 0.0012 | 0.0056 | 0.000 | 0.0084 ¶¶¶ | 0.0365 | **0.0733** | 0.0017 | 0.0061 | 0.0468 | 0.0000 ¶ |
|
| 240 |
+
| bbh_logic | 0.9540 | 0.9013 | 0.6107 | 0.829 | 0.9647 | 0.6633 | 0.5333 | 0.5713 | 0.7700 | 0.6367 | **0.9980** |
|
| 241 |
+
| math500 | 0.7467 | 0.6567 | 0.7900 | 0.790 | 0.3600 ¶¶¶ | 0.6900 | 0.4233 | 0.7133 | 0.7800 | 0.6100 | **0.8433** |
|
| 242 |
+
| **macro (10 CoT datasets)** | 0.7901 | 0.7942 | 0.8097 | 0.802 | 0.7683 | 0.7339 | 0.6997 | 0.7523 | 0.7884 | 0.8493 | **0.8689** |
|
| 243 |
+
| **macro (all 14)** | 0.7425 | 0.7332 | 0.7262 | 0.728 | 0.6611 | 0.6694 | 0.6245 | 0.6814 | 0.7378 | 0.7591 | **0.8086** |
|
| 244 |
+
| tok/s | 21169 † | 21916 † | 28010 † | — | 11097 † | 15880 | 16160 | 25230 | 22480 | 9420 | 3374 |
|
| 245 |
+
| mean gen length | ≈792 | ≈1282 | ≈1789 | — | ≈2787 | **482** | 510 | ≈796 | ≈1327 | ≈1931 | 801 |
|
| 246 |
+
| **ans/s** | 26.7 † | 17.1 † | ≈15.7 † | — | ≈4.0 † | **32.9** | 31.7 | ≈31.7 | ≈16.9 | 4.9 | 4.2 |
|
| 247 |
|
| 248 |
‡ MXFP4 weights, tensor-parallel 2 — quantized and multi-GPU, so not directly comparable to the
|
| 249 |
single-GPU BF16 rows. ¶ 74% of its `rudas_ood` generations hit the length cap, so that cell is a
|
|
|
|
| 269 |
against 0.7942, a difference well inside the sampling noise at n = 300 per dataset) while gaining
|
| 270 |
in-domain, which is the point of the WiSE-FT stage. Second, the 14-dataset macro rises 0.0093
|
| 271 |
over the base, driven by BBH-logic, MATH-500 and MuSR. Third, it edges LFM2.5-8B-A1B on both
|
| 272 |
+
macros at less than half the parameters and leads every untuned arm on SVAMP (0.9500; the
|
| 273 |
+
tuned VibeThinker-3B is at 0.953, one example out of 300 higher).
|
| 274 |
|
| 275 |
VibeThinker-3B is the stronger held-out model on the 10-dataset macro (0.8097 against 0.7901). It
|
| 276 |
is ahead of TwIL-LM3-Pro on seven datasets — GSM8K, ARC, LogicBench, StrategyQA, DROP,
|
|
|
|
| 290 |
### Against the tuned VibeThinker-3B
|
| 291 |
|
| 292 |
The tables above use the public VibeThinker-3B checkpoint. The same post-training pipeline was also
|
| 293 |
+
applied to it, and two of its tuned configurations are the closest same-scale comparisons to
|
| 294 |
+
TwIL-LM3-Pro: WiSE-FT (λ = 0.50), and the SLERP merge (d = 0.5, t = 0.5) that appears in the
|
| 295 |
+
tables and plot as **VibeThinker-trained**. These values come from the family comparison tables
|
| 296 |
+
rather than from a per-lane raw report, so they are shown as a summary only:
|
| 297 |
|
| 298 |
| model | macro gate | macro_primary | B10 | B14 | Track A truncation |
|
| 299 |
|---|---:|---:|---:|---:|---:|
|
| 300 |
| TwIL-LM3-Pro | **0.554** | **0.588** | 0.790 | **0.743** | 24.2% |
|
| 301 |
+
| VibeThinker-trained (SLERP, d = 0.5, t = 0.5) | 0.508 | 0.579 | **0.802** | 0.728 | — |
|
| 302 |
| VibeThinker-3B, WiSE-FT λ = 0.50 | 0.541 | **0.588** | **0.802** | 0.728 ◊ | 14.3% |
|
| 303 |
|
| 304 |
+
◊ There is no B14 row for the λ = 0.50 configuration; the figure is the one recorded for the SLERP
|
| 305 |
+
configuration in the row above. Truncation was not recorded for the SLERP configuration.
|
| 306 |
+
|
| 307 |
+
Against VibeThinker-trained, TwIL-LM3-Pro is ahead on Track A (macro gate 0.554 against 0.508,
|
| 308 |
+
`macro_primary` 0.588 against 0.579) and on the 14-dataset macro (0.743 against 0.728), and behind
|
| 309 |
+
on the 10-dataset macro (0.790 against 0.802). Against the WiSE-FT λ = 0.50 configuration the two
|
| 310 |
+
are effectively tied on Track A (gate 0.554 against 0.541, `macro_primary` equal at 0.588, both
|
| 311 |
+
within sampling noise at n = 200), and the tuned VibeThinker-3B is ahead on the 10-dataset macro
|
| 312 |
+
with a lower truncation rate. TwIL-LM3-Pro's edge is the 14-dataset macro, a gap that cannot be
|
| 313 |
+
broken down per dataset from the summary values.
|
| 314 |
+
|
| 315 |
+
#### How VibeThinker-trained differs from the base VibeThinker-3B
|
| 316 |
+
|
| 317 |
+
**Base VibeThinker-3B** is WeiboAI's public checkpoint, unmodified, and is what the untuned columns
|
| 318 |
+
in the tables above measure. **VibeThinker-trained** starts from those same weights and changes
|
| 319 |
+
them in two ways:
|
| 320 |
+
|
| 321 |
+
* **Formal-logic post-training.** A rank-64 LoRA is trained on the same synthetic formal-logic
|
| 322 |
+
corpus and Track A objectives used for TwIL-LM3-Pro (FOL translation, entailment, semantic
|
| 323 |
+
parsing, Lean formalisation and critique, procedural reasoning, rule induction), with multipath
|
| 324 |
+
distillation supplying the reasoning traces.
|
| 325 |
+
* **Merging back toward the base.** The tuned weights are not used raw. They are merged into the
|
| 326 |
+
pretrained VibeThinker-3B weights with a DARE-SLERP merge (d = 0.5, t = 0.5, the configuration
|
| 327 |
+
labels used in the family tables), the same conservative interpolation step that protects
|
| 328 |
+
held-out capability in TwIL-LM3-Pro.
|
| 329 |
+
|
| 330 |
+
The family tables record no reinforcement-learning (MGPO) run for VibeThinker-3B, so
|
| 331 |
+
VibeThinker-trained reflects the supervised and merging stages only, whereas TwIL-LM3-Pro also has
|
| 332 |
+
the MGPO stage. It is a reference point for the pipeline, not a checkpoint shipped in this
|
| 333 |
+
repository.
|
| 334 |
|
| 335 |
+
What that changes, on the family tables (one source, so the comparison is like for like):
|
| 336 |
+
|
| 337 |
+
| metric | VibeThinker-3B (base) | VibeThinker-trained | change |
|
| 338 |
+
|---|---:|---:|---:|
|
| 339 |
+
| macro gate | 0.374 | 0.508 | +0.134 |
|
| 340 |
+
| macro_primary | 0.444 | 0.579 | +0.135 |
|
| 341 |
+
| entailment | 0.480 | 0.685 | +0.205 |
|
| 342 |
+
| rule induction | 0.094 | 0.227 | +0.133 |
|
| 343 |
+
| lean_critic | 0.580 | 0.685 | +0.105 |
|
| 344 |
+
| `lm_corpus` perplexity ↓ | 18.70 | 7.35 | 2.5x lower |
|
| 345 |
+
| `math_corpus` perplexity ↓ | 21.80 | 7.00 | 3.1x lower |
|
| 346 |
+
| Track B, 10-dataset macro | 0.815 | 0.802 | −0.013 |
|
| 347 |
+
| Track B, 14-dataset macro | 0.729 | 0.728 | −0.001 |
|
| 348 |
+
|
| 349 |
+
The in-domain gain is large and the held-out change is inside sampling noise at n = 300 per
|
| 350 |
+
dataset, which is the same trade the pipeline makes on Granite 4.2. Two per-dataset shifts are
|
| 351 |
+
worth knowing: BBH-logic rises from 0.609 to 0.829, and IFEval falls from 0.660 to 0.557. Read the
|
| 352 |
+
base column of this table against the family source only; the lane tables above record the same
|
| 353 |
+
weights at a gate of 0.4118 from a separate run.
|
| 354 |
|
| 355 |
### Checkpoint selection
|
| 356 |
|
|
|
|
| 461 |
γ = 3.0. The run was resumed at step 800 with β = 0.02 and trained through step 2580, and the
|
| 462 |
published checkpoint is **step 2580**. This release is the merged policy, not a LoRA adapter.
|
| 463 |
|
| 464 |
+
## One pipeline, several models
|
| 465 |
+
|
| 466 |
+
The recipe above is not specific to Granite 4.2. Each stage takes a base checkpoint, the shared
|
| 467 |
+
formal-logic corpus and the programmatic verifier as inputs and hands the next stage a checkpoint
|
| 468 |
+
of the same shape, so pointing the pipeline at a different base model does not change its
|
| 469 |
+
structure. In the internal family comparison, the same rank-64 LoRA stage and the same merge
|
| 470 |
+
families (WiSE-FT, DARE, SLERP) were run on five bases that differ in vocabulary, pretraining and
|
| 471 |
+
size: an earlier Granite, SmolLM3, the TwIL base, VibeThinker-3B (151,936-token vocabulary) and
|
| 472 |
+
Granite 4.2 (100,352). The MGPO stage was run on the two Granite bases.
|
| 473 |
+
|
| 474 |
+
The one setting that is chosen per model is the merge coefficient, picked by a sweep over held-out
|
| 475 |
+
performance: the best WiSE-FT weight is 0.15 for SmolLM3, 0.20 for Granite and the TwIL base, and
|
| 476 |
+
0.50 for VibeThinker-3B, which needs to keep far more of the tuned weights than the others do.
|
| 477 |
+
|
| 478 |
+
| base model | best tuned configuration | Track A macro gate (base → tuned) | Track B 10-dataset macro (base → tuned) |
|
| 479 |
+
|---|---|---:|---:|
|
| 480 |
+
| Granite | WiSE-FT λ = 0.20 | 0.309 → 0.454 | 0.777 → 0.786 |
|
| 481 |
+
| SmolLM3 | WiSE-FT λ = 0.15 | 0.344 → 0.446 | 0.723 → 0.736 |
|
| 482 |
+
| TwIL base | WiSE-FT λ = 0.20 | 0.423 → 0.435 | 0.729 → 0.716 |
|
| 483 |
+
| VibeThinker-3B | WiSE-FT λ = 0.50 | 0.374 → 0.541 | 0.815 → 0.802 |
|
| 484 |
+
| Granite 4.2 (this model) | MGPO β = 0.02, step 2580 | 0.4313 → 0.5539 | 0.7942 → 0.7901 |
|
| 485 |
+
|
| 486 |
+
Track A improves on every base, from +0.012 to +0.167, and the Track B change stays within ±0.013,
|
| 487 |
+
inside sampling noise at n = 300 per dataset. The first four rows are from the family comparison
|
| 488 |
+
tables and the last from the paired run described under [Results](#results), so the base gates
|
| 489 |
+
are only comparable within a row. Reinforcement learning was applied only to the Granite bases
|
| 490 |
+
here, so the other rows show what the supervised and merging stages achieve without it.
|
| 491 |
+
|
| 492 |
## Limitations and caveats
|
| 493 |
|
| 494 |
|
|
|
|
| 509 |
and Qwen3.5-4B, on all 18 tasks) share the same sampled rows and decoding, but the serving engine differs between
|
| 510 |
arms (vLLM 0.19.1 for TwIL-LM3-Pro, its base, Qwen3.5-4B, Qwen3-8B and LFM2.5-8B-A1B; vLLM 0.11.2 for
|
| 511 |
TwIL-LM3, Llama-3.2-3B and VibeThinker-3B), and the engine version is part of the protocol hash.
|
| 512 |
+
The VibeThinker-trained column (★) comes from the internal family comparison tables and is not
|
| 513 |
+
covered by the manifest checks described here. Throughput has its own, separate engine caveat (see the † note under the Track A table). With
|
| 514 |
n = 200 per lane on Track A and n = 300 per dataset on Track B, differences of two to three points
|
| 515 |
are within sampling noise.
|
| 516 |
|
benchmarks.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|