[UPDATE] Restructure release: safetensors weights, cbt package, chat demo, clean model card
Browse files- .gitattributes +1 -0
- README.md +118 -50
- cbt/__init__.py +97 -0
- model.py → cbt/model.py +1 -7
- tokenizer.py → cbt/tokenizer.py +0 -0
- cbt/tools.py +92 -0
- cbt_1d_twin_arithmetic_20_70d.json +0 -224
- champion_v3_arithmetic.json +0 -224
- champion_v3_arithmetic_4_16d.json +0 -108
- chat.py +62 -0
- config.json +7 -21
- dataset.py +0 -1468
- equalized_15d_1024_and_ruoss_sweep.json +0 -890
- cbt_2d_arithmetic_20_70d.json → evaluation/cbt_2d/addition_20-70_digits.json +0 -0
- cbt_2d_arithmetic_4_16d.json → evaluation/cbt_2d/addition_4-16_digits.json +0 -0
- cbt_2d_ood.json → evaluation/cbt_2d/tools_out_of_distribution.json +0 -0
- cbt_2d_phrasing.json → evaluation/cbt_2d/tools_rephrased.json +0 -0
- cbt_2d_unseen_copy.json → evaluation/cbt_2d/tools_unseen_entities.json +0 -0
- cbt1d_v5_arithmetic.json → evaluation/twin_1d/addition_20-70_digits.json +0 -0
- cbt_1d_twin_arithmetic_4_16d.json → evaluation/twin_1d/addition_4-16_digits.json +0 -0
- champion_v3_ood.json → evaluation/twin_1d/tools_out_of_distribution.json +1 -1
- champion_v3_phrasing.json → evaluation/twin_1d/tools_rephrased.json +7 -7
- champion_v3_unseen_ood.json → evaluation/twin_1d/tools_unseen_entities.json +10 -10
- interactive.py +0 -329
- champion_final.pt → model.safetensors +2 -2
- RESEARCH_PAPER.md → paper/CBT-18.95M.md +81 -39
- RESEARCH_PAPER.pdf → paper/CBT-18.95M.pdf +2 -2
- random_extrapolation_16_to_100d.json +0 -655
- random_extrapolation_champion_16_to_100d.json +0 -610
- requirements.txt +3 -0
.gitattributes
CHANGED
|
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
RESEARCH_PAPER.pdf filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
RESEARCH_PAPER.pdf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
paper/CBT-18.95M.pdf filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -2,93 +2,161 @@
|
|
| 2 |
language:
|
| 3 |
- en
|
| 4 |
license: mit
|
|
|
|
|
|
|
| 5 |
tags:
|
| 6 |
- arithmetic
|
| 7 |
- length-generalization
|
| 8 |
-
-
|
|
|
|
| 9 |
- 2d-rope
|
| 10 |
- tool-use
|
| 11 |
-
- function-calling
|
| 12 |
-
- reasoning
|
| 13 |
datasets:
|
| 14 |
- synthetic
|
| 15 |
metrics:
|
| 16 |
- accuracy
|
| 17 |
---
|
| 18 |
|
| 19 |
-
#
|
| 20 |
|
| 21 |
-
|
| 22 |
-
- **Research Paper**: [RESEARCH_PAPER.md](https://github.com/NeithanT/CBT/blob/main/RESEARCH_PAPER.md)
|
| 23 |
|
| 24 |
-
|
| 25 |
|
| 26 |
-
|
| 27 |
|
| 28 |
-
|
| 29 |
-
- **Addition**, 30 random problems per length (70% uniform, 15% dense carry, 15% cascades), scored as teacher-forced greedy agreement with the gold scratchpad and answer: **150/150** at 4-16 digits, **60/60** at 20 and 25 digits (seen), **210/210** at 26-55 digits (unseen), **25/30** at 60 and **19/30** at 70 digits. Pooled over the unseen 26-70 digits: **254/270**.
|
| 30 |
-
- **Tool use**: **26/26** exact calls on a 26-prompt phrasing suite, **18/18** on 18 out-of-distribution prompts, and **30/30** exact copies of entities absent from the training corpus (unseen-30). The final fine-tuning data was broadened after the intermediate model's failures on the first two suites were seen, so the unseen-30 set is the cleaner evidence.
|
| 31 |
-
- **1D-RoPE control**: a twin with the same architecture, data and step count but standard 1D-RoPE scores **0/270** at 26-70 digits and **73/74** exact tool calls. The 2D coordinates account for the arithmetic extrapolation, not for the tool-use results.
|
| 32 |
-
- **Baselines**: standard 1D-RoPE models trained on 1-12 digit sums score 0/30 at every length from 14 to 35 digits (character and BPE vocabularies; looped, hierarchical and hybrid variants too).
|
| 33 |
-
|
| 34 |
-
All results are single training runs. Details, limitations and the failure analysis are in the paper.
|
| 35 |
|
| 36 |
---
|
| 37 |
|
| 38 |
-
##
|
| 39 |
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
| **Recurrent Passes (R)** | 3 (effective depth 18) |
|
| 46 |
-
| **Hidden Dimension** | 512 |
|
| 47 |
-
| **Attention Heads** | 8 (head dimension 64) |
|
| 48 |
-
| **Context** | 1024 tokens |
|
| 49 |
-
| **Positional Embedding** | Decoupled 2D-RoPE (half of each head rotated by place $p$, half by step $\tau$; rule-assigned) |
|
| 50 |
-
| **Coordinate Shift Masking** | $k = 45$ |
|
| 51 |
-
| **Vocabulary** | 111 (14 control tokens + 97 characters) |
|
| 52 |
-
| **Weight Tying** | input embedding = output projection |
|
| 53 |
|
| 54 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
|
| 56 |
-
|
| 57 |
|
| 58 |
-
|
| 59 |
|
| 60 |
```python
|
| 61 |
-
import
|
| 62 |
-
|
| 63 |
-
from
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
model.eval()
|
| 70 |
```
|
| 71 |
|
| 72 |
-
|
| 73 |
|
| 74 |
-
|
| 75 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
```
|
| 77 |
|
|
|
|
|
|
|
| 78 |
---
|
| 79 |
|
| 80 |
-
##
|
| 81 |
|
| 82 |
-
|
|
|
|
|
|
|
| 83 |
|
| 84 |
---
|
| 85 |
|
| 86 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
|
| 88 |
```bibtex
|
| 89 |
@article{vargas2026cbt,
|
| 90 |
-
title={
|
| 91 |
-
author={Vargas Vargas, Neithan},
|
| 92 |
-
year={2026}
|
| 93 |
}
|
| 94 |
```
|
|
|
|
| 2 |
language:
|
| 3 |
- en
|
| 4 |
license: mit
|
| 5 |
+
library_name: pytorch
|
| 6 |
+
pipeline_tag: text-generation
|
| 7 |
tags:
|
| 8 |
- arithmetic
|
| 9 |
- length-generalization
|
| 10 |
+
- character-level
|
| 11 |
+
- rope
|
| 12 |
- 2d-rope
|
| 13 |
- tool-use
|
|
|
|
|
|
|
| 14 |
datasets:
|
| 15 |
- synthetic
|
| 16 |
metrics:
|
| 17 |
- accuracy
|
| 18 |
---
|
| 19 |
|
| 20 |
+
# CBT-18.95M
|
| 21 |
|
| 22 |
+
**A 19M-parameter character-level Transformer that adds numbers longer than any it was trained on.**
|
|
|
|
| 23 |
|
| 24 |
+
CBT (Character-Based Transformer) gives every token a two-dimensional position: *which column* it belongs to and *which step* inside that column. With that layout, a model trained on sums of up to 25 digits keeps adding correctly at 26, 40 and 55 digits. An identical twin with ordinary 1D positions gets every one of those wrong.
|
| 25 |
|
| 26 |
+
The same model also does simple tool calling (weather, search, calculator, unit conversion, ...) in a small chat format.
|
| 27 |
|
| 28 |
+
[Paper (PDF)](paper/CBT-18.95M.pdf) · [Paper (Markdown)](paper/CBT-18.95M.md) · [Code on GitHub](https://github.com/NeithanT/CBT)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
---
|
| 31 |
|
| 32 |
+
## Try it
|
| 33 |
|
| 34 |
+
```bash
|
| 35 |
+
git clone https://huggingface.co/Neithanvarvas/CBT && cd CBT
|
| 36 |
+
pip install -r requirements.txt
|
| 37 |
+
python chat.py
|
| 38 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
+
```text
|
| 41 |
+
> 123456789+987654321
|
| 42 |
+
scratchpad : (0+9+1)=10(1+8+2)=11(1+7+3)=11(1+6+4)=11 ... (1+0+0)=01
|
| 43 |
+
answer : 1111111110 [PASS]
|
| 44 |
+
|
| 45 |
+
> What is the weather in Tokyo?
|
| 46 |
+
think : The user requests current weather information. I will query the get_weather tool.
|
| 47 |
+
call : get_weather(city="Tokyo")
|
| 48 |
+
result : {"temp": 22, "condition": "Sunny", "city": "Tokyo"}
|
| 49 |
+
answer : The weather in Tokyo is currently Sunny with a temperature of 22 degrees Celsius.
|
| 50 |
+
```
|
| 51 |
|
| 52 |
+
Runs on a CPU; no GPU needed. Pass `--online` to let the search tool query Wikipedia, or `--raw` to see the token stream.
|
| 53 |
|
| 54 |
+
### From Python
|
| 55 |
|
| 56 |
```python
|
| 57 |
+
from cbt import load, add, chat
|
| 58 |
+
|
| 59 |
+
model, tok = load() # uses this folder, or downloads from the Hub if run elsewhere
|
| 60 |
+
|
| 61 |
+
add(model, tok, 31415926535897932384626433832795, 27182818284590452353602874713527)["answer"]
|
| 62 |
+
# '58598744820488384738229308546322'
|
| 63 |
+
|
| 64 |
+
chat(model, tok, "Convert 10 km to miles")["answer"]
|
|
|
|
| 65 |
```
|
| 66 |
|
| 67 |
+
`chat` runs one user turn and executes at most one tool call. Tools are offline stand-ins in [`cbt/tools.py`](cbt/tools.py); pass `tool_executor=your_function` to connect real ones.
|
| 68 |
|
| 69 |
+
---
|
| 70 |
+
|
| 71 |
+
## Results
|
| 72 |
+
|
| 73 |
+
Greedy decoding, 30 random problems per length. Training covered sums of 1 to 25 digits.
|
| 74 |
+
|
| 75 |
+
### Addition: CBT vs. its 1D-RoPE twin
|
| 76 |
+
|
| 77 |
+
The twin has the same architecture, data and number of training steps. Only the position encoding differs.
|
| 78 |
+
|
| 79 |
+
| Operand digits | Seen in training? | CBT (2D) | 1D twin |
|
| 80 |
+
| :--- | :---: | :---: | :---: |
|
| 81 |
+
| 4, 8, 12, 14, 16 | yes | 150 / 150 | 149 / 150 |
|
| 82 |
+
| 20, 25 | yes | 60 / 60 | 57 / 60 |
|
| 83 |
+
| 26 | no | **30 / 30** | 0 / 30 |
|
| 84 |
+
| 30, 35, 40, 45, 50, 55 | no | **180 / 180** | 0 / 180 |
|
| 85 |
+
| 60 | no | 25 / 30 | 0 / 30 |
|
| 86 |
+
| 70 | no | 19 / 30 | 0 / 30 |
|
| 87 |
+
| **All unseen (26-70)** | no | **254 / 270** | **0 / 270** |
|
| 88 |
+
|
| 89 |
+
### Tool use: exact tool and arguments
|
| 90 |
+
|
| 91 |
+
| Test set | Prompts | CBT (2D) | 1D twin |
|
| 92 |
+
| :--- | :---: | :---: | :---: |
|
| 93 |
+
| Rephrased requests | 26 | 26 | 25 |
|
| 94 |
+
| Out-of-distribution entities | 18 | 18 | 18 |
|
| 95 |
+
| Entities never seen in training | 30 | 30 | 30 |
|
| 96 |
+
|
| 97 |
+
Both models do equally well here, so tool use comes from the training data, not from the 2D layout. The fine-tuning data was broadened after an earlier model failed the first two sets, so the unseen-entity set is the fairer test.
|
| 98 |
+
|
| 99 |
+
All numbers come from single training runs with small test sets. Raw per-problem results are in [`evaluation/`](evaluation/); limitations are discussed in the paper.
|
| 100 |
+
|
| 101 |
+
---
|
| 102 |
+
|
| 103 |
+
## How it works
|
| 104 |
+
|
| 105 |
+
Each token gets a coordinate (p, tau):
|
| 106 |
+
|
| 107 |
+
- **Arithmetic:** p is the decimal place (units, tens, ...) and tau is the step within that column of the scratchpad.
|
| 108 |
+
- **Chat:** p is the dialogue segment (user, think, call, result, answer) and tau is the character position inside it.
|
| 109 |
+
|
| 110 |
+
Half of each attention head is rotated by p and half by tau (Decoupled 2D-RoPE). Aligning digits of the same place becomes one fixed relation, "same p", which does not change as numbers get longer, so the model learns it once. During training, p is randomly shifted by up to 45, so the model has seen place values up to 70.
|
| 111 |
+
|
| 112 |
+
| | |
|
| 113 |
+
| :--- | :--- |
|
| 114 |
+
| Parameters | 18.94M (18.89M non-embedding) |
|
| 115 |
+
| Layers | 6, each applied 3 times with shared weights (effective depth 18) |
|
| 116 |
+
| Width / heads | 512 / 8 (head dim 64) |
|
| 117 |
+
| Context | 1024 characters |
|
| 118 |
+
| Vocabulary | 111: 97 ASCII characters (letters, digits, punctuation, space, tab, newline) + 14 control tags |
|
| 119 |
+
| Positions | Decoupled 2D-RoPE, coordinate shift k = 45 |
|
| 120 |
+
| Training | Pretraining, then several supervised fine-tuning rounds (23,927 steps in the last) |
|
| 121 |
+
|
| 122 |
+
### Prompt formats
|
| 123 |
+
|
| 124 |
+
```text
|
| 125 |
+
Addition: <BOS>123+456=<THINK>(0+3+6)=09(0+2+5)=07(0+1+4)=05</THINK>=579<EOS>
|
| 126 |
+
Chat: <BOS><USER>question</USER><THINK>...</THINK><CALL>tool(arg="...")</CALL><RESULT>{json}</RESULT><ANSWER>...</ANSWER>
|
| 127 |
```
|
| 128 |
|
| 129 |
+
Scratchpad steps read `(carry + digit_a + digit_b) = carry_out digit`, from the units column upward.
|
| 130 |
+
|
| 131 |
---
|
| 132 |
|
| 133 |
+
## Limitations
|
| 134 |
|
| 135 |
+
- Addition of non-negative integers is the only arithmetic it was trained on. Accuracy drops beyond 55 digits, and the 1024-character context caps sums at roughly 75 digits.
|
| 136 |
+
- The chat ability is narrow: it was trained on synthetic tool-calling dialogues with a fixed set of tools. It is not a general assistant and has little world knowledge.
|
| 137 |
+
- Character-level, English-only, ASCII-only. Other characters become `<UNK>`.
|
| 138 |
|
| 139 |
---
|
| 140 |
|
| 141 |
+
## Files
|
| 142 |
+
|
| 143 |
+
```text
|
| 144 |
+
README.md this card
|
| 145 |
+
config.json model hyperparameters
|
| 146 |
+
model.safetensors weights
|
| 147 |
+
cbt/ model, tokenizer, tool stubs, load/add/chat helpers
|
| 148 |
+
chat.py terminal demo
|
| 149 |
+
requirements.txt
|
| 150 |
+
paper/ the paper (PDF and Markdown)
|
| 151 |
+
evaluation/ per-problem results behind the tables above
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
## Citation
|
| 155 |
|
| 156 |
```bibtex
|
| 157 |
@article{vargas2026cbt,
|
| 158 |
+
title = {From Digits to Characters: Two-Dimensional Rotary Coordinates for Length Generalization in a Compact Character-Level Transformer},
|
| 159 |
+
author = {Vargas Vargas, Neithan},
|
| 160 |
+
year = {2026}
|
| 161 |
}
|
| 162 |
```
|
cbt/__init__.py
ADDED
|
@@ -0,0 +1,97 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""CBT-18.95M: character-level Transformer with Decoupled 2D-RoPE.
|
| 2 |
+
|
| 3 |
+
Quick start:
|
| 4 |
+
from cbt import load, add, chat
|
| 5 |
+
model, tok = load() # local folder or the Hugging Face repo
|
| 6 |
+
add(model, tok, 123456789, 987654321) # {'answer': '1111111110', 'correct': True, ...}
|
| 7 |
+
chat(model, tok, "What is the weather in Tokyo?")
|
| 8 |
+
"""
|
| 9 |
+
|
| 10 |
+
import json
|
| 11 |
+
import os
|
| 12 |
+
from dataclasses import fields
|
| 13 |
+
from typing import Any, Callable, Dict, Optional, Tuple, Union
|
| 14 |
+
|
| 15 |
+
import torch
|
| 16 |
+
|
| 17 |
+
from .model import ScaledGPT as CBT, ScaledGPTConfig as CBTConfig
|
| 18 |
+
from .tokenizer import ScaledTokenizer as CBTTokenizer
|
| 19 |
+
from .tools import execute_tool
|
| 20 |
+
|
| 21 |
+
__all__ = ["CBT", "CBTConfig", "CBTTokenizer", "load", "add", "chat", "execute_tool"]
|
| 22 |
+
|
| 23 |
+
REPO_ID = "Neithanvarvas/CBT"
|
| 24 |
+
_PKG_DIR = os.path.dirname(os.path.abspath(__file__))
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def _resolve(source: Optional[str], filename: str) -> str:
|
| 28 |
+
"""Finds `filename` in a local folder (default: the repo root next to this package) or downloads it."""
|
| 29 |
+
folder = source or os.path.dirname(_PKG_DIR)
|
| 30 |
+
local = os.path.join(folder, filename)
|
| 31 |
+
if os.path.isfile(local):
|
| 32 |
+
return local
|
| 33 |
+
from huggingface_hub import hf_hub_download
|
| 34 |
+
return hf_hub_download(repo_id=source if source and not os.path.isdir(source) else REPO_ID, filename=filename)
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def load(source: Optional[str] = None, device: Optional[Union[str, torch.device]] = None) -> Tuple[CBT, CBTTokenizer]:
|
| 38 |
+
"""Loads model and tokenizer from a local folder or a Hugging Face repo id; returns (model, tokenizer) in eval mode."""
|
| 39 |
+
from safetensors.torch import load_file
|
| 40 |
+
|
| 41 |
+
with open(_resolve(source, "config.json"), encoding="utf-8") as f:
|
| 42 |
+
raw: Dict[str, Any] = json.load(f)
|
| 43 |
+
config = CBTConfig(**{k.name: raw[k.name] for k in fields(CBTConfig) if k.name in raw})
|
| 44 |
+
|
| 45 |
+
device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
|
| 46 |
+
tok = CBTTokenizer()
|
| 47 |
+
model = CBT(config, tokenizer=tok)
|
| 48 |
+
model.load_state_dict(load_file(_resolve(source, "model.safetensors")))
|
| 49 |
+
return model.to(device).eval(), tok
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
@torch.no_grad()
|
| 53 |
+
def _generate(model: CBT, ids: torch.Tensor, max_new_tokens: int, stop: list) -> torch.Tensor:
|
| 54 |
+
return model.generate(ids, max_new_tokens=max_new_tokens, temperature=0.0, stop_tokens=stop)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def _encode(model: CBT, tok: CBTTokenizer, text: str, bos: bool = True) -> torch.Tensor:
|
| 58 |
+
device = next(model.parameters()).device
|
| 59 |
+
return torch.tensor([tok.encode(text, add_bos=bos, add_eos=False)], dtype=torch.long, device=device)
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def add(model: CBT, tok: CBTTokenizer, a: Union[int, str], b: Union[int, str]) -> Dict[str, Any]:
|
| 63 |
+
"""Greedy-decodes a + b with the column scratchpad. Trained on 1-25 digit operands."""
|
| 64 |
+
a, b = str(a).strip(), str(b).strip()
|
| 65 |
+
if not (a.isdigit() and b.isdigit()):
|
| 66 |
+
raise ValueError("operands must be non-negative integers")
|
| 67 |
+
budget = 12 * (max(len(a), len(b)) + 2) + 8
|
| 68 |
+
out = _generate(model, _encode(model, tok, f"{a}+{b}=<THINK>"), budget, [tok.eos_id])
|
| 69 |
+
raw = tok.decode(out[0].tolist())
|
| 70 |
+
think, _, tail = raw.partition("<THINK>")[2].partition("</THINK>")
|
| 71 |
+
answer = tail.lstrip("=").replace("<EOS>", "").strip()
|
| 72 |
+
return {"answer": answer, "scratchpad": think, "correct": answer == str(int(a) + int(b)), "raw": raw}
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def chat(
|
| 76 |
+
model: CBT,
|
| 77 |
+
tok: CBTTokenizer,
|
| 78 |
+
message: str,
|
| 79 |
+
tool_executor: Optional[Callable[[str], str]] = execute_tool,
|
| 80 |
+
max_new_tokens: int = 256,
|
| 81 |
+
) -> Dict[str, Any]:
|
| 82 |
+
"""Runs one user turn: <THINK> reasoning, optional <CALL> executed by `tool_executor`, then <ANSWER>."""
|
| 83 |
+
ids = _encode(model, tok, f"<USER>{message}</USER><THINK>")
|
| 84 |
+
out = _generate(model, ids, max_new_tokens, [tok.call_e_id, tok.answer_e_id, tok.eos_id])
|
| 85 |
+
raw = tok.decode(out[0].tolist())
|
| 86 |
+
|
| 87 |
+
if "</CALL>" in raw and tool_executor is not None:
|
| 88 |
+
call = raw.split("<CALL>")[-1].split("</CALL>")[0]
|
| 89 |
+
cont = torch.cat([out, _encode(model, tok, f"<RESULT>{tool_executor(call)}</RESULT>", bos=False)], dim=1)
|
| 90 |
+
out = _generate(model, cont, max_new_tokens, [tok.answer_e_id, tok.eos_id])
|
| 91 |
+
raw = tok.decode(out[0].tolist())
|
| 92 |
+
|
| 93 |
+
def span(tag: str) -> Optional[str]:
|
| 94 |
+
s, e = f"<{tag}>", f"</{tag}>"
|
| 95 |
+
return raw.split(s, 1)[1].split(e, 1)[0].strip() if s in raw and e in raw else None
|
| 96 |
+
|
| 97 |
+
return {"think": span("THINK"), "call": span("CALL"), "result": span("RESULT"), "answer": span("ANSWER"), "raw": raw}
|
model.py → cbt/model.py
RENAMED
|
@@ -22,13 +22,7 @@ import torch
|
|
| 22 |
import torch.nn as nn
|
| 23 |
import torch.nn.functional as F
|
| 24 |
|
| 25 |
-
|
| 26 |
-
from champion_model.scaled_t4.tokenizer import ScaledTokenizer
|
| 27 |
-
except ImportError:
|
| 28 |
-
try:
|
| 29 |
-
from tokenizer import ScaledTokenizer
|
| 30 |
-
except ImportError:
|
| 31 |
-
from champion_model.unified_200k.tokenizer import UnifiedTokenizer as ScaledTokenizer
|
| 32 |
|
| 33 |
|
| 34 |
@dataclass
|
|
|
|
| 22 |
import torch.nn as nn
|
| 23 |
import torch.nn.functional as F
|
| 24 |
|
| 25 |
+
from .tokenizer import ScaledTokenizer
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
|
| 28 |
@dataclass
|
tokenizer.py → cbt/tokenizer.py
RENAMED
|
File without changes
|
cbt/tools.py
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Offline stand-ins for the tools CBT was fine-tuned to call. Replace `execute_tool` with real backends as needed."""
|
| 2 |
+
|
| 3 |
+
import ast
|
| 4 |
+
import json
|
| 5 |
+
import operator
|
| 6 |
+
import re
|
| 7 |
+
from typing import Any, Dict
|
| 8 |
+
|
| 9 |
+
_OPS = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv,
|
| 10 |
+
ast.USub: operator.neg, ast.UAdd: operator.pos}
|
| 11 |
+
|
| 12 |
+
_KNOWLEDGE: Dict[str, str] = {
|
| 13 |
+
"tomato": "the edible berry of the plant Solanum lycopersicum, commonly prepared as a culinary vegetable.",
|
| 14 |
+
"apple": "a round, edible fruit produced by an apple tree (Malus domestica), cultivated worldwide.",
|
| 15 |
+
"potato": "a starchy, tuberous root crop of the nightshade family Solanaceae.",
|
| 16 |
+
"coffee": "a brewed drink prepared from roasted coffee beans of flowering Coffea species.",
|
| 17 |
+
"banana": "an elongated, edible fruit produced by several kinds of large herbaceous plants in the genus Musa.",
|
| 18 |
+
"gravity": "a fundamental interaction causing mutual attraction between all things with mass or energy.",
|
| 19 |
+
"photosynthesis": "a process used by plants to convert light energy into chemical energy stored in glucose.",
|
| 20 |
+
"atom": "the basic unit of a chemical element, consisting of a dense nucleus surrounded by electrons.",
|
| 21 |
+
"speed of light": "approximately 299,792 kilometers per second in a vacuum.",
|
| 22 |
+
"mount everest": "Earth's highest mountain above sea level, located in the Himalayas.",
|
| 23 |
+
}
|
| 24 |
+
|
| 25 |
+
_UNITS = {("km", "miles"): 0.621371, ("miles", "km"): 1.60934, ("kg", "lbs"): 2.20462, ("lbs", "kg"): 0.453592}
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def _calc(node: ast.AST) -> Any:
|
| 29 |
+
if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
|
| 30 |
+
return node.value
|
| 31 |
+
if isinstance(node, ast.BinOp) and type(node.op) in _OPS:
|
| 32 |
+
return _OPS[type(node.op)](_calc(node.left), _calc(node.right))
|
| 33 |
+
if isinstance(node, ast.UnaryOp) and type(node.op) in _OPS:
|
| 34 |
+
return _OPS[type(node.op)](_calc(node.operand))
|
| 35 |
+
raise ValueError("unsupported expression")
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def _search(query: str, online: bool) -> Dict[str, str]:
|
| 39 |
+
q = query.lower().rstrip("s")
|
| 40 |
+
for key, text in _KNOWLEDGE.items():
|
| 41 |
+
if key in q or q in key:
|
| 42 |
+
return {"snippet": text}
|
| 43 |
+
if online:
|
| 44 |
+
try:
|
| 45 |
+
import urllib.parse
|
| 46 |
+
import urllib.request
|
| 47 |
+
url = "https://en.wikipedia.org/api/rest_v1/page/summary/" + urllib.parse.quote(query.replace(" ", "_"))
|
| 48 |
+
req = urllib.request.Request(url, headers={"User-Agent": "CBT-demo/1.0"})
|
| 49 |
+
with urllib.request.urlopen(req, timeout=3) as resp:
|
| 50 |
+
extract = json.loads(resp.read().decode("utf-8")).get("extract")
|
| 51 |
+
if extract:
|
| 52 |
+
return {"snippet": extract[:220]}
|
| 53 |
+
except Exception:
|
| 54 |
+
pass
|
| 55 |
+
return {"snippet": f"{query} is a topic in verified knowledge sources."}
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def execute_tool(call: str, online: bool = False) -> str:
|
| 59 |
+
"""Returns a JSON string for a call such as `get_weather(city="Tokyo")`."""
|
| 60 |
+
name = call.split("(", 1)[0].strip()
|
| 61 |
+
args = dict(re.findall(r'(\w+)="([^"]*)"', call))
|
| 62 |
+
|
| 63 |
+
if name == "calculator":
|
| 64 |
+
try:
|
| 65 |
+
return json.dumps({"result": _calc(ast.parse(args.get("expr", ""), mode="eval").body)})
|
| 66 |
+
except ZeroDivisionError:
|
| 67 |
+
return json.dumps({"error": "ZeroDivisionError: division by zero"})
|
| 68 |
+
except Exception as e:
|
| 69 |
+
return json.dumps({"error": str(e)})
|
| 70 |
+
if name == "get_weather":
|
| 71 |
+
city = args.get("city", "")
|
| 72 |
+
if city.lower() in ("atlantis", "narnia"):
|
| 73 |
+
return json.dumps({"status": "error", "code": 404, "message": f"City '{city}' not found"})
|
| 74 |
+
return json.dumps({"temp": 22, "condition": "Sunny", "city": city})
|
| 75 |
+
if name == "get_time":
|
| 76 |
+
return json.dumps({"time": "14:30", "timezone": args.get("timezone", "UTC")})
|
| 77 |
+
if name == "unit_convert":
|
| 78 |
+
m = re.search(r"value=([\d.]+)", call)
|
| 79 |
+
value = float(m.group(1)) if m else 0.0
|
| 80 |
+
to = args.get("to_unit", "")
|
| 81 |
+
return json.dumps({"value": round(value * _UNITS.get((args.get("from_unit", ""), to), 1.0), 2), "unit": to})
|
| 82 |
+
if name == "search":
|
| 83 |
+
return json.dumps(_search(args.get("query", ""), online))
|
| 84 |
+
if name == "database_query":
|
| 85 |
+
return json.dumps({"status": "success", "count": 3, "records": [{"id": 101, "status": "active"}]})
|
| 86 |
+
if name == "create_event":
|
| 87 |
+
return json.dumps({"event_id": "evt_98214", "status": "confirmed"})
|
| 88 |
+
if name == "file_manager":
|
| 89 |
+
return json.dumps({"status": "ok", "bytes_written": 64})
|
| 90 |
+
if name == "fetch_api":
|
| 91 |
+
return json.dumps({"status": 200, "data": {"status": "healthy"}})
|
| 92 |
+
return json.dumps({"status": "executed", "call": call})
|
cbt_1d_twin_arithmetic_20_70d.json
DELETED
|
@@ -1,224 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"checkpoint": "champion_model/checkpoints/1d/cbt_18m_1d_rope_R3_k45_sftv5.zip",
|
| 3 |
-
"max_train_operand_digits": 25,
|
| 4 |
-
"arithmetic": {
|
| 5 |
-
"per_length": {
|
| 6 |
-
"20": {
|
| 7 |
-
"digits": 20,
|
| 8 |
-
"mode": "teacher_forced",
|
| 9 |
-
"seen_in_training": true,
|
| 10 |
-
"n": 30,
|
| 11 |
-
"exact": 28,
|
| 12 |
-
"exact_ci": [
|
| 13 |
-
78.7,
|
| 14 |
-
98.2
|
| 15 |
-
],
|
| 16 |
-
"think_exact": 28,
|
| 17 |
-
"think_ci": [
|
| 18 |
-
78.7,
|
| 19 |
-
98.2
|
| 20 |
-
],
|
| 21 |
-
"gen_tokens": 0,
|
| 22 |
-
"elapsed_sec": 9.7
|
| 23 |
-
},
|
| 24 |
-
"25": {
|
| 25 |
-
"digits": 25,
|
| 26 |
-
"mode": "teacher_forced",
|
| 27 |
-
"seen_in_training": true,
|
| 28 |
-
"n": 30,
|
| 29 |
-
"exact": 29,
|
| 30 |
-
"exact_ci": [
|
| 31 |
-
83.3,
|
| 32 |
-
99.4
|
| 33 |
-
],
|
| 34 |
-
"think_exact": 29,
|
| 35 |
-
"think_ci": [
|
| 36 |
-
83.3,
|
| 37 |
-
99.4
|
| 38 |
-
],
|
| 39 |
-
"gen_tokens": 0,
|
| 40 |
-
"elapsed_sec": 12.08
|
| 41 |
-
},
|
| 42 |
-
"26": {
|
| 43 |
-
"digits": 26,
|
| 44 |
-
"mode": "teacher_forced",
|
| 45 |
-
"seen_in_training": false,
|
| 46 |
-
"n": 30,
|
| 47 |
-
"exact": 0,
|
| 48 |
-
"exact_ci": [
|
| 49 |
-
0.0,
|
| 50 |
-
11.4
|
| 51 |
-
],
|
| 52 |
-
"think_exact": 0,
|
| 53 |
-
"think_ci": [
|
| 54 |
-
0.0,
|
| 55 |
-
11.4
|
| 56 |
-
],
|
| 57 |
-
"gen_tokens": 0,
|
| 58 |
-
"elapsed_sec": 12.42
|
| 59 |
-
},
|
| 60 |
-
"30": {
|
| 61 |
-
"digits": 30,
|
| 62 |
-
"mode": "teacher_forced",
|
| 63 |
-
"seen_in_training": false,
|
| 64 |
-
"n": 30,
|
| 65 |
-
"exact": 0,
|
| 66 |
-
"exact_ci": [
|
| 67 |
-
0.0,
|
| 68 |
-
11.4
|
| 69 |
-
],
|
| 70 |
-
"think_exact": 0,
|
| 71 |
-
"think_ci": [
|
| 72 |
-
0.0,
|
| 73 |
-
11.4
|
| 74 |
-
],
|
| 75 |
-
"gen_tokens": 0,
|
| 76 |
-
"elapsed_sec": 14.01
|
| 77 |
-
},
|
| 78 |
-
"35": {
|
| 79 |
-
"digits": 35,
|
| 80 |
-
"mode": "teacher_forced",
|
| 81 |
-
"seen_in_training": false,
|
| 82 |
-
"n": 30,
|
| 83 |
-
"exact": 0,
|
| 84 |
-
"exact_ci": [
|
| 85 |
-
0.0,
|
| 86 |
-
11.4
|
| 87 |
-
],
|
| 88 |
-
"think_exact": 0,
|
| 89 |
-
"think_ci": [
|
| 90 |
-
0.0,
|
| 91 |
-
11.4
|
| 92 |
-
],
|
| 93 |
-
"gen_tokens": 0,
|
| 94 |
-
"elapsed_sec": 17.04
|
| 95 |
-
},
|
| 96 |
-
"40": {
|
| 97 |
-
"digits": 40,
|
| 98 |
-
"mode": "teacher_forced",
|
| 99 |
-
"seen_in_training": false,
|
| 100 |
-
"n": 30,
|
| 101 |
-
"exact": 0,
|
| 102 |
-
"exact_ci": [
|
| 103 |
-
0.0,
|
| 104 |
-
11.4
|
| 105 |
-
],
|
| 106 |
-
"think_exact": 0,
|
| 107 |
-
"think_ci": [
|
| 108 |
-
0.0,
|
| 109 |
-
11.4
|
| 110 |
-
],
|
| 111 |
-
"gen_tokens": 0,
|
| 112 |
-
"elapsed_sec": 19.7
|
| 113 |
-
},
|
| 114 |
-
"45": {
|
| 115 |
-
"digits": 45,
|
| 116 |
-
"mode": "teacher_forced",
|
| 117 |
-
"seen_in_training": false,
|
| 118 |
-
"n": 30,
|
| 119 |
-
"exact": 0,
|
| 120 |
-
"exact_ci": [
|
| 121 |
-
0.0,
|
| 122 |
-
11.4
|
| 123 |
-
],
|
| 124 |
-
"think_exact": 0,
|
| 125 |
-
"think_ci": [
|
| 126 |
-
0.0,
|
| 127 |
-
11.4
|
| 128 |
-
],
|
| 129 |
-
"gen_tokens": 0,
|
| 130 |
-
"elapsed_sec": 22.33
|
| 131 |
-
},
|
| 132 |
-
"50": {
|
| 133 |
-
"digits": 50,
|
| 134 |
-
"mode": "teacher_forced",
|
| 135 |
-
"seen_in_training": false,
|
| 136 |
-
"n": 30,
|
| 137 |
-
"exact": 0,
|
| 138 |
-
"exact_ci": [
|
| 139 |
-
0.0,
|
| 140 |
-
11.4
|
| 141 |
-
],
|
| 142 |
-
"think_exact": 0,
|
| 143 |
-
"think_ci": [
|
| 144 |
-
0.0,
|
| 145 |
-
11.4
|
| 146 |
-
],
|
| 147 |
-
"gen_tokens": 0,
|
| 148 |
-
"elapsed_sec": 23.61
|
| 149 |
-
},
|
| 150 |
-
"55": {
|
| 151 |
-
"digits": 55,
|
| 152 |
-
"mode": "teacher_forced",
|
| 153 |
-
"seen_in_training": false,
|
| 154 |
-
"n": 30,
|
| 155 |
-
"exact": 0,
|
| 156 |
-
"exact_ci": [
|
| 157 |
-
0.0,
|
| 158 |
-
11.4
|
| 159 |
-
],
|
| 160 |
-
"think_exact": 0,
|
| 161 |
-
"think_ci": [
|
| 162 |
-
0.0,
|
| 163 |
-
11.4
|
| 164 |
-
],
|
| 165 |
-
"gen_tokens": 0,
|
| 166 |
-
"elapsed_sec": 26.12
|
| 167 |
-
},
|
| 168 |
-
"60": {
|
| 169 |
-
"digits": 60,
|
| 170 |
-
"mode": "teacher_forced",
|
| 171 |
-
"seen_in_training": false,
|
| 172 |
-
"n": 30,
|
| 173 |
-
"exact": 0,
|
| 174 |
-
"exact_ci": [
|
| 175 |
-
0.0,
|
| 176 |
-
11.4
|
| 177 |
-
],
|
| 178 |
-
"think_exact": 0,
|
| 179 |
-
"think_ci": [
|
| 180 |
-
0.0,
|
| 181 |
-
11.4
|
| 182 |
-
],
|
| 183 |
-
"gen_tokens": 0,
|
| 184 |
-
"elapsed_sec": 28.45
|
| 185 |
-
},
|
| 186 |
-
"70": {
|
| 187 |
-
"digits": 70,
|
| 188 |
-
"mode": "teacher_forced",
|
| 189 |
-
"seen_in_training": false,
|
| 190 |
-
"n": 30,
|
| 191 |
-
"exact": 0,
|
| 192 |
-
"exact_ci": [
|
| 193 |
-
0.0,
|
| 194 |
-
11.4
|
| 195 |
-
],
|
| 196 |
-
"think_exact": 0,
|
| 197 |
-
"think_ci": [
|
| 198 |
-
0.0,
|
| 199 |
-
11.4
|
| 200 |
-
],
|
| 201 |
-
"gen_tokens": 0,
|
| 202 |
-
"elapsed_sec": 33.21
|
| 203 |
-
}
|
| 204 |
-
},
|
| 205 |
-
"summary": {
|
| 206 |
-
"unseen": {
|
| 207 |
-
"n": 270,
|
| 208 |
-
"exact": 0,
|
| 209 |
-
"exact_ci": [
|
| 210 |
-
0.0,
|
| 211 |
-
1.4
|
| 212 |
-
]
|
| 213 |
-
},
|
| 214 |
-
"seen": {
|
| 215 |
-
"n": 60,
|
| 216 |
-
"exact": 57,
|
| 217 |
-
"exact_ci": [
|
| 218 |
-
86.3,
|
| 219 |
-
98.3
|
| 220 |
-
]
|
| 221 |
-
}
|
| 222 |
-
}
|
| 223 |
-
}
|
| 224 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
champion_v3_arithmetic.json
DELETED
|
@@ -1,224 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"checkpoint": "champion_model/checkpoints/champion_v3_final.zip",
|
| 3 |
-
"max_train_operand_digits": 25,
|
| 4 |
-
"arithmetic": {
|
| 5 |
-
"per_length": {
|
| 6 |
-
"20": {
|
| 7 |
-
"digits": 20,
|
| 8 |
-
"mode": "teacher_forced",
|
| 9 |
-
"seen_in_training": true,
|
| 10 |
-
"n": 30,
|
| 11 |
-
"exact": 30,
|
| 12 |
-
"exact_ci": [
|
| 13 |
-
88.6,
|
| 14 |
-
100.0
|
| 15 |
-
],
|
| 16 |
-
"think_exact": 30,
|
| 17 |
-
"think_ci": [
|
| 18 |
-
88.6,
|
| 19 |
-
100.0
|
| 20 |
-
],
|
| 21 |
-
"gen_tokens": 0,
|
| 22 |
-
"elapsed_sec": 60.31
|
| 23 |
-
},
|
| 24 |
-
"25": {
|
| 25 |
-
"digits": 25,
|
| 26 |
-
"mode": "teacher_forced",
|
| 27 |
-
"seen_in_training": true,
|
| 28 |
-
"n": 30,
|
| 29 |
-
"exact": 30,
|
| 30 |
-
"exact_ci": [
|
| 31 |
-
88.6,
|
| 32 |
-
100.0
|
| 33 |
-
],
|
| 34 |
-
"think_exact": 30,
|
| 35 |
-
"think_ci": [
|
| 36 |
-
88.6,
|
| 37 |
-
100.0
|
| 38 |
-
],
|
| 39 |
-
"gen_tokens": 0,
|
| 40 |
-
"elapsed_sec": 59.46
|
| 41 |
-
},
|
| 42 |
-
"26": {
|
| 43 |
-
"digits": 26,
|
| 44 |
-
"mode": "teacher_forced",
|
| 45 |
-
"seen_in_training": false,
|
| 46 |
-
"n": 30,
|
| 47 |
-
"exact": 30,
|
| 48 |
-
"exact_ci": [
|
| 49 |
-
88.6,
|
| 50 |
-
100.0
|
| 51 |
-
],
|
| 52 |
-
"think_exact": 30,
|
| 53 |
-
"think_ci": [
|
| 54 |
-
88.6,
|
| 55 |
-
100.0
|
| 56 |
-
],
|
| 57 |
-
"gen_tokens": 0,
|
| 58 |
-
"elapsed_sec": 67.86
|
| 59 |
-
},
|
| 60 |
-
"30": {
|
| 61 |
-
"digits": 30,
|
| 62 |
-
"mode": "teacher_forced",
|
| 63 |
-
"seen_in_training": false,
|
| 64 |
-
"n": 30,
|
| 65 |
-
"exact": 30,
|
| 66 |
-
"exact_ci": [
|
| 67 |
-
88.6,
|
| 68 |
-
100.0
|
| 69 |
-
],
|
| 70 |
-
"think_exact": 30,
|
| 71 |
-
"think_ci": [
|
| 72 |
-
88.6,
|
| 73 |
-
100.0
|
| 74 |
-
],
|
| 75 |
-
"gen_tokens": 0,
|
| 76 |
-
"elapsed_sec": 63.91
|
| 77 |
-
},
|
| 78 |
-
"35": {
|
| 79 |
-
"digits": 35,
|
| 80 |
-
"mode": "teacher_forced",
|
| 81 |
-
"seen_in_training": false,
|
| 82 |
-
"n": 30,
|
| 83 |
-
"exact": 30,
|
| 84 |
-
"exact_ci": [
|
| 85 |
-
88.6,
|
| 86 |
-
100.0
|
| 87 |
-
],
|
| 88 |
-
"think_exact": 30,
|
| 89 |
-
"think_ci": [
|
| 90 |
-
88.6,
|
| 91 |
-
100.0
|
| 92 |
-
],
|
| 93 |
-
"gen_tokens": 0,
|
| 94 |
-
"elapsed_sec": 76.18
|
| 95 |
-
},
|
| 96 |
-
"40": {
|
| 97 |
-
"digits": 40,
|
| 98 |
-
"mode": "teacher_forced",
|
| 99 |
-
"seen_in_training": false,
|
| 100 |
-
"n": 30,
|
| 101 |
-
"exact": 30,
|
| 102 |
-
"exact_ci": [
|
| 103 |
-
88.6,
|
| 104 |
-
100.0
|
| 105 |
-
],
|
| 106 |
-
"think_exact": 30,
|
| 107 |
-
"think_ci": [
|
| 108 |
-
88.6,
|
| 109 |
-
100.0
|
| 110 |
-
],
|
| 111 |
-
"gen_tokens": 0,
|
| 112 |
-
"elapsed_sec": 84.75
|
| 113 |
-
},
|
| 114 |
-
"45": {
|
| 115 |
-
"digits": 45,
|
| 116 |
-
"mode": "teacher_forced",
|
| 117 |
-
"seen_in_training": false,
|
| 118 |
-
"n": 30,
|
| 119 |
-
"exact": 30,
|
| 120 |
-
"exact_ci": [
|
| 121 |
-
88.6,
|
| 122 |
-
100.0
|
| 123 |
-
],
|
| 124 |
-
"think_exact": 30,
|
| 125 |
-
"think_ci": [
|
| 126 |
-
88.6,
|
| 127 |
-
100.0
|
| 128 |
-
],
|
| 129 |
-
"gen_tokens": 0,
|
| 130 |
-
"elapsed_sec": 91.2
|
| 131 |
-
},
|
| 132 |
-
"50": {
|
| 133 |
-
"digits": 50,
|
| 134 |
-
"mode": "teacher_forced",
|
| 135 |
-
"seen_in_training": false,
|
| 136 |
-
"n": 30,
|
| 137 |
-
"exact": 30,
|
| 138 |
-
"exact_ci": [
|
| 139 |
-
88.6,
|
| 140 |
-
100.0
|
| 141 |
-
],
|
| 142 |
-
"think_exact": 30,
|
| 143 |
-
"think_ci": [
|
| 144 |
-
88.6,
|
| 145 |
-
100.0
|
| 146 |
-
],
|
| 147 |
-
"gen_tokens": 0,
|
| 148 |
-
"elapsed_sec": 115.78
|
| 149 |
-
},
|
| 150 |
-
"55": {
|
| 151 |
-
"digits": 55,
|
| 152 |
-
"mode": "teacher_forced",
|
| 153 |
-
"seen_in_training": false,
|
| 154 |
-
"n": 30,
|
| 155 |
-
"exact": 30,
|
| 156 |
-
"exact_ci": [
|
| 157 |
-
88.6,
|
| 158 |
-
100.0
|
| 159 |
-
],
|
| 160 |
-
"think_exact": 30,
|
| 161 |
-
"think_ci": [
|
| 162 |
-
88.6,
|
| 163 |
-
100.0
|
| 164 |
-
],
|
| 165 |
-
"gen_tokens": 0,
|
| 166 |
-
"elapsed_sec": 107.83
|
| 167 |
-
},
|
| 168 |
-
"60": {
|
| 169 |
-
"digits": 60,
|
| 170 |
-
"mode": "teacher_forced",
|
| 171 |
-
"seen_in_training": false,
|
| 172 |
-
"n": 30,
|
| 173 |
-
"exact": 25,
|
| 174 |
-
"exact_ci": [
|
| 175 |
-
66.4,
|
| 176 |
-
92.7
|
| 177 |
-
],
|
| 178 |
-
"think_exact": 25,
|
| 179 |
-
"think_ci": [
|
| 180 |
-
66.4,
|
| 181 |
-
92.7
|
| 182 |
-
],
|
| 183 |
-
"gen_tokens": 0,
|
| 184 |
-
"elapsed_sec": 120.42
|
| 185 |
-
},
|
| 186 |
-
"70": {
|
| 187 |
-
"digits": 70,
|
| 188 |
-
"mode": "teacher_forced",
|
| 189 |
-
"seen_in_training": false,
|
| 190 |
-
"n": 30,
|
| 191 |
-
"exact": 19,
|
| 192 |
-
"exact_ci": [
|
| 193 |
-
45.5,
|
| 194 |
-
78.1
|
| 195 |
-
],
|
| 196 |
-
"think_exact": 19,
|
| 197 |
-
"think_ci": [
|
| 198 |
-
45.5,
|
| 199 |
-
78.1
|
| 200 |
-
],
|
| 201 |
-
"gen_tokens": 0,
|
| 202 |
-
"elapsed_sec": 144.89
|
| 203 |
-
}
|
| 204 |
-
},
|
| 205 |
-
"summary": {
|
| 206 |
-
"unseen": {
|
| 207 |
-
"n": 270,
|
| 208 |
-
"exact": 254,
|
| 209 |
-
"exact_ci": [
|
| 210 |
-
90.6,
|
| 211 |
-
96.3
|
| 212 |
-
]
|
| 213 |
-
},
|
| 214 |
-
"seen": {
|
| 215 |
-
"n": 60,
|
| 216 |
-
"exact": 60,
|
| 217 |
-
"exact_ci": [
|
| 218 |
-
94.0,
|
| 219 |
-
100.0
|
| 220 |
-
]
|
| 221 |
-
}
|
| 222 |
-
}
|
| 223 |
-
}
|
| 224 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
champion_v3_arithmetic_4_16d.json
DELETED
|
@@ -1,108 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"checkpoint": "checkpoints/cbt/cbt_18m_2d_rope_R3_k45_sftv5.pt",
|
| 3 |
-
"max_train_operand_digits": 25,
|
| 4 |
-
"arithmetic": {
|
| 5 |
-
"per_length": {
|
| 6 |
-
"4": {
|
| 7 |
-
"digits": 4,
|
| 8 |
-
"mode": "teacher_forced",
|
| 9 |
-
"seen_in_training": true,
|
| 10 |
-
"n": 30,
|
| 11 |
-
"exact": 30,
|
| 12 |
-
"exact_ci": [
|
| 13 |
-
88.6,
|
| 14 |
-
100.0
|
| 15 |
-
],
|
| 16 |
-
"think_exact": 30,
|
| 17 |
-
"think_ci": [
|
| 18 |
-
88.6,
|
| 19 |
-
100.0
|
| 20 |
-
],
|
| 21 |
-
"gen_tokens": 0,
|
| 22 |
-
"elapsed_sec": 3.29
|
| 23 |
-
},
|
| 24 |
-
"8": {
|
| 25 |
-
"digits": 8,
|
| 26 |
-
"mode": "teacher_forced",
|
| 27 |
-
"seen_in_training": true,
|
| 28 |
-
"n": 30,
|
| 29 |
-
"exact": 30,
|
| 30 |
-
"exact_ci": [
|
| 31 |
-
88.6,
|
| 32 |
-
100.0
|
| 33 |
-
],
|
| 34 |
-
"think_exact": 30,
|
| 35 |
-
"think_ci": [
|
| 36 |
-
88.6,
|
| 37 |
-
100.0
|
| 38 |
-
],
|
| 39 |
-
"gen_tokens": 0,
|
| 40 |
-
"elapsed_sec": 5.23
|
| 41 |
-
},
|
| 42 |
-
"12": {
|
| 43 |
-
"digits": 12,
|
| 44 |
-
"mode": "teacher_forced",
|
| 45 |
-
"seen_in_training": true,
|
| 46 |
-
"n": 30,
|
| 47 |
-
"exact": 30,
|
| 48 |
-
"exact_ci": [
|
| 49 |
-
88.6,
|
| 50 |
-
100.0
|
| 51 |
-
],
|
| 52 |
-
"think_exact": 30,
|
| 53 |
-
"think_ci": [
|
| 54 |
-
88.6,
|
| 55 |
-
100.0
|
| 56 |
-
],
|
| 57 |
-
"gen_tokens": 0,
|
| 58 |
-
"elapsed_sec": 6.68
|
| 59 |
-
},
|
| 60 |
-
"14": {
|
| 61 |
-
"digits": 14,
|
| 62 |
-
"mode": "teacher_forced",
|
| 63 |
-
"seen_in_training": true,
|
| 64 |
-
"n": 30,
|
| 65 |
-
"exact": 30,
|
| 66 |
-
"exact_ci": [
|
| 67 |
-
88.6,
|
| 68 |
-
100.0
|
| 69 |
-
],
|
| 70 |
-
"think_exact": 30,
|
| 71 |
-
"think_ci": [
|
| 72 |
-
88.6,
|
| 73 |
-
100.0
|
| 74 |
-
],
|
| 75 |
-
"gen_tokens": 0,
|
| 76 |
-
"elapsed_sec": 7.93
|
| 77 |
-
},
|
| 78 |
-
"16": {
|
| 79 |
-
"digits": 16,
|
| 80 |
-
"mode": "teacher_forced",
|
| 81 |
-
"seen_in_training": true,
|
| 82 |
-
"n": 30,
|
| 83 |
-
"exact": 30,
|
| 84 |
-
"exact_ci": [
|
| 85 |
-
88.6,
|
| 86 |
-
100.0
|
| 87 |
-
],
|
| 88 |
-
"think_exact": 30,
|
| 89 |
-
"think_ci": [
|
| 90 |
-
88.6,
|
| 91 |
-
100.0
|
| 92 |
-
],
|
| 93 |
-
"gen_tokens": 0,
|
| 94 |
-
"elapsed_sec": 8.82
|
| 95 |
-
}
|
| 96 |
-
},
|
| 97 |
-
"summary": {
|
| 98 |
-
"seen": {
|
| 99 |
-
"n": 150,
|
| 100 |
-
"exact": 150,
|
| 101 |
-
"exact_ci": [
|
| 102 |
-
97.5,
|
| 103 |
-
100.0
|
| 104 |
-
]
|
| 105 |
-
}
|
| 106 |
-
}
|
| 107 |
-
}
|
| 108 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
chat.py
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Terminal demo for CBT-18.95M. Sums like `123+456` go to the adder; anything else is a chat turn.
|
| 3 |
+
|
| 4 |
+
python chat.py # interactive
|
| 5 |
+
python chat.py "98765432109876+1234" # one prompt
|
| 6 |
+
python chat.py --online "Who was Ada Lovelace?"
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
import argparse
|
| 10 |
+
import functools
|
| 11 |
+
import re
|
| 12 |
+
|
| 13 |
+
from cbt import add, chat, execute_tool, load
|
| 14 |
+
|
| 15 |
+
SUM = re.compile(r"^\s*(\d+)\s*\+\s*(\d+)\s*=?\s*$")
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def respond(model, tok, text: str, online: bool, show_raw: bool) -> None:
|
| 19 |
+
m = SUM.match(text)
|
| 20 |
+
if m:
|
| 21 |
+
r = add(model, tok, m.group(1), m.group(2))
|
| 22 |
+
print(f" scratchpad : {r['scratchpad']}")
|
| 23 |
+
print(f" answer : {r['answer']} [{'PASS' if r['correct'] else 'FAIL'}]")
|
| 24 |
+
else:
|
| 25 |
+
r = chat(model, tok, text, tool_executor=functools.partial(execute_tool, online=online))
|
| 26 |
+
for key in ("think", "call", "result", "answer"):
|
| 27 |
+
if r[key] is not None:
|
| 28 |
+
print(f" {key:<10} : {r[key]}")
|
| 29 |
+
if show_raw:
|
| 30 |
+
print(f" raw : {r['raw']}")
|
| 31 |
+
print()
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def main() -> None:
|
| 35 |
+
p = argparse.ArgumentParser(description="CBT-18.95M terminal demo")
|
| 36 |
+
p.add_argument("prompt", nargs="?", help="single prompt; omit for interactive mode")
|
| 37 |
+
p.add_argument("--model", default=None, help="local folder or Hugging Face repo id (default: this folder)")
|
| 38 |
+
p.add_argument("--device", default=None, help="cpu or cuda (default: auto)")
|
| 39 |
+
p.add_argument("--online", action="store_true", help="let the search tool query Wikipedia")
|
| 40 |
+
p.add_argument("--raw", action="store_true", help="also print the raw token stream")
|
| 41 |
+
args = p.parse_args()
|
| 42 |
+
|
| 43 |
+
model, tok = load(args.model, args.device)
|
| 44 |
+
if args.prompt:
|
| 45 |
+
respond(model, tok, args.prompt, args.online, args.raw)
|
| 46 |
+
return
|
| 47 |
+
|
| 48 |
+
print("CBT-18.95M. Type a sum (e.g. 123456789+987654321) or a question. Ctrl-D to quit.\n")
|
| 49 |
+
while True:
|
| 50 |
+
try:
|
| 51 |
+
text = input("> ").strip()
|
| 52 |
+
except (EOFError, KeyboardInterrupt):
|
| 53 |
+
print()
|
| 54 |
+
break
|
| 55 |
+
if text.lower() in ("exit", "quit"):
|
| 56 |
+
break
|
| 57 |
+
if text:
|
| 58 |
+
respond(model, tok, text, args.online, args.raw)
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
if __name__ == "__main__":
|
| 62 |
+
main()
|
config.json
CHANGED
|
@@ -1,6 +1,8 @@
|
|
| 1 |
{
|
| 2 |
-
"model_type": "
|
| 3 |
-
"
|
|
|
|
|
|
|
| 4 |
"block_size": 1024,
|
| 5 |
"n_layer": 6,
|
| 6 |
"n_head": 8,
|
|
@@ -8,24 +10,8 @@
|
|
| 8 |
"n_recur": 3,
|
| 9 |
"dropout": 0.0,
|
| 10 |
"bias": false,
|
| 11 |
-
"pos_emb_type": "rope_2d",
|
| 12 |
"rope_theta": 10000.0,
|
| 13 |
"max_coords": 4096,
|
| 14 |
-
"k_rand_shift":
|
| 15 |
-
"
|
| 16 |
-
|
| 17 |
-
"parameters": {
|
| 18 |
-
"total": 18953216,
|
| 19 |
-
"non_embedding": 18887680,
|
| 20 |
-
"target_scale": "~10M to ~20M",
|
| 21 |
-
"status": "[OK] Scaled 18.95M parameter architecture with 2D-RoPE and R=3 recurrence"
|
| 22 |
-
},
|
| 23 |
-
"t4_optimization": {
|
| 24 |
-
"precision": "FP16 AMP (torch.amp.autocast)",
|
| 25 |
-
"scaler": "torch.amp.GradScaler('cuda')",
|
| 26 |
-
"sdpa": "Scaled Dot-Product Attention with O(1) activation memory",
|
| 27 |
-
"distributed": "PyTorch DistributedDataParallel (DDP) across 2x T4 GPUs",
|
| 28 |
-
"bfloat16_allowed": false,
|
| 29 |
-
"tf32_allowed": false
|
| 30 |
-
}
|
| 31 |
-
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"model_type": "cbt",
|
| 3 |
+
"architecture": "CBT-18.95M",
|
| 4 |
+
"training_step": 23927,
|
| 5 |
+
"vocab_size": 111,
|
| 6 |
"block_size": 1024,
|
| 7 |
"n_layer": 6,
|
| 8 |
"n_head": 8,
|
|
|
|
| 10 |
"n_recur": 3,
|
| 11 |
"dropout": 0.0,
|
| 12 |
"bias": false,
|
|
|
|
| 13 |
"rope_theta": 10000.0,
|
| 14 |
"max_coords": 4096,
|
| 15 |
+
"k_rand_shift": 45,
|
| 16 |
+
"pos_emb_type": "rope_2d"
|
| 17 |
+
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
dataset.py
DELETED
|
@@ -1,1468 +0,0 @@
|
|
| 1 |
-
"""
|
| 2 |
-
Joint Balanced Dataset & Batch Encoder for ScaledGPT (Dual T4 Optimization).
|
| 3 |
-
Supports:
|
| 4 |
-
1. Multi-digit column scratchpad arithmetic (1 to 50+ digits curriculum).
|
| 5 |
-
2. Expanded tool repertoire:
|
| 6 |
-
- Single-tool calling (calculator, weather, search, time, unit_convert)
|
| 7 |
-
- Multi-tool sequential chaining
|
| 8 |
-
- Nested JSON parameter schemas
|
| 9 |
-
- Error handling and recovery traces
|
| 10 |
-
- Multi-turn conversational dialogues
|
| 11 |
-
- Direct conversational routing (no-tool)
|
| 12 |
-
3. Strict prompt and environment output loss masking.
|
| 13 |
-
"""
|
| 14 |
-
|
| 15 |
-
import os
|
| 16 |
-
import sys
|
| 17 |
-
import json
|
| 18 |
-
import random
|
| 19 |
-
from typing import List, Dict, Tuple, Any, Optional
|
| 20 |
-
import torch
|
| 21 |
-
|
| 22 |
-
ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
| 23 |
-
if ROOT_DIR not in sys.path:
|
| 24 |
-
sys.path.insert(0, ROOT_DIR)
|
| 25 |
-
|
| 26 |
-
try:
|
| 27 |
-
from champion_model.scaled_t4.tokenizer import ScaledTokenizer
|
| 28 |
-
except ImportError:
|
| 29 |
-
from tokenizer import ScaledTokenizer
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
def format_addition_char_scratchpad(a: int, b: int) -> str:
|
| 33 |
-
"""Formats addition into column-by-column carry scratchpad for character tokenizer."""
|
| 34 |
-
s_a, s_b = str(a), str(b)
|
| 35 |
-
len_a, len_b = len(s_a), len(s_b)
|
| 36 |
-
max_len = max(len_a, len_b)
|
| 37 |
-
|
| 38 |
-
carry = 0
|
| 39 |
-
steps, digits = [], []
|
| 40 |
-
|
| 41 |
-
for col in range(max_len):
|
| 42 |
-
da = int(s_a[len_a - 1 - col]) if col < len_a else 0
|
| 43 |
-
db = int(s_b[len_b - 1 - col]) if col < len_b else 0
|
| 44 |
-
col_sum = carry + da + db
|
| 45 |
-
next_carry = col_sum // 10
|
| 46 |
-
res_digit = col_sum % 10
|
| 47 |
-
steps.append(f"({carry}+{da}+{db})={next_carry}{res_digit}")
|
| 48 |
-
digits.append(str(res_digit))
|
| 49 |
-
carry = next_carry
|
| 50 |
-
|
| 51 |
-
if carry > 0:
|
| 52 |
-
steps.append(f"({carry}+0+0)=0{carry}")
|
| 53 |
-
digits.append(str(carry))
|
| 54 |
-
|
| 55 |
-
ans = "".join(reversed(digits))
|
| 56 |
-
return f"{a}+{b}=<THINK>" + "".join(steps) + f"</THINK>={ans}"
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
def extract_scratchpad_answer(text: str) -> Tuple[str, str]:
|
| 60 |
-
"""Extracts reasoning scratchpad and predicted answer from generated output."""
|
| 61 |
-
reasoning = ""
|
| 62 |
-
ans = ""
|
| 63 |
-
if "<THINK>" in text and "</THINK>" in text:
|
| 64 |
-
parts = text.split("</THINK>")
|
| 65 |
-
reasoning = parts[0].replace("<THINK>", "")
|
| 66 |
-
after = parts[1]
|
| 67 |
-
if "=" in after:
|
| 68 |
-
ans = after.split("=")[-1].replace("<EOS>", "").replace("<PAD>", "").strip()
|
| 69 |
-
else:
|
| 70 |
-
ans = after.replace("<EOS>", "").replace("<PAD>", "").strip()
|
| 71 |
-
elif "=" in text:
|
| 72 |
-
ans = text.split("=")[-1].replace("<EOS>", "").replace("<PAD>", "").strip()
|
| 73 |
-
return reasoning, ans
|
| 74 |
-
|
| 75 |
-
# Factual databases for synthetic generation
|
| 76 |
-
CITIES = [
|
| 77 |
-
"New York", "London", "Tokyo", "Paris", "Berlin", "Sydney", "Toronto", "Madrid",
|
| 78 |
-
"Rome", "Seoul", "Singapore", "Zurich", "Honolulu", "Reykjavik", "Ouagadougou",
|
| 79 |
-
"Vancouver", "Santiago", "Auckland", "Casablanca", "Vienna", "Stockholm", "Kyoto",
|
| 80 |
-
"Dublin", "Oslo", "Helsinki", "Bangkok", "Cairo", "Lisbon", "Amsterdam", "Athens",
|
| 81 |
-
"Buenos Aires", "Kuala Lumpur", "Rio de Janeiro", "San Francisco", "Hong Kong",
|
| 82 |
-
"Los Angeles", "Mexico City", "Cape Town", "Sao Paulo",
|
| 83 |
-
]
|
| 84 |
-
CONDITIONS = ["Sunny", "Rainy", "Cloudy", "Snowy", "Windy", "Foggy", "Clear"]
|
| 85 |
-
TIMEZONES = ["UTC", "EST", "PST", "GMT", "JST", "CET", "AEST", "CST", "IST", "HST"]
|
| 86 |
-
|
| 87 |
-
# Nonce word phonetic generators to force pure Attention Induction Copying
|
| 88 |
-
NONCE_ONSETS = ["bl", "br", "ch", "cl", "cr", "dr", "fl", "fr", "gl", "gr", "pl", "pr", "sc", "sk", "sl", "sm", "sn", "sp", "st", "sw", "th", "tr", "qu", "v", "z", "k", "j"]
|
| 89 |
-
NONCE_NUCLEI = ["a", "e", "i", "o", "u", "y", "ai", "ea", "ee", "oa", "oo", "ou"]
|
| 90 |
-
NONCE_CODAS = ["b", "d", "f", "g", "k", "l", "m", "n", "p", "r", "s", "t", "x", "z", "ck", "nk", "mp", "sh", "th", "nt", "rt"]
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
def generate_nonce_entity(rng: random.Random, allow_multi_word: bool = True) -> Tuple[str, str, str]:
|
| 94 |
-
"""Generates an arbitrary pseudo-word entity (singular, plural, type), supporting single and multi-word spans."""
|
| 95 |
-
ent_type = rng.choice(["concept", "plant", "creature", "artifact", "substance", "person", "theory"])
|
| 96 |
-
|
| 97 |
-
# 30% chance of multi-word compound nonce (2-3 words) to train multi-word pointer induction
|
| 98 |
-
if allow_multi_word and rng.random() < 0.30:
|
| 99 |
-
n_words = rng.choice([2, 2, 3])
|
| 100 |
-
words = []
|
| 101 |
-
for _ in range(n_words):
|
| 102 |
-
n_syl = rng.choice([1, 2, 2, 3])
|
| 103 |
-
w = "".join(rng.choice(NONCE_ONSETS) + rng.choice(NONCE_NUCLEI) + rng.choice(NONCE_CODAS) for _ in range(n_syl))
|
| 104 |
-
words.append(w.capitalize())
|
| 105 |
-
singular = " ".join(words)
|
| 106 |
-
last_w = words[-1]
|
| 107 |
-
plural_last = last_w + ("es" if last_w[-1] in "sxz" else "s")
|
| 108 |
-
plural = " ".join(words[:-1] + [plural_last])
|
| 109 |
-
return singular, plural, ent_type
|
| 110 |
-
|
| 111 |
-
n_syl = rng.choice([1, 2, 2, 3, 4])
|
| 112 |
-
parts = []
|
| 113 |
-
for _ in range(n_syl):
|
| 114 |
-
parts.append(rng.choice(NONCE_ONSETS) + rng.choice(NONCE_NUCLEI) + rng.choice(NONCE_CODAS))
|
| 115 |
-
singular = "".join(parts)
|
| 116 |
-
|
| 117 |
-
# 20% chance of complex morphological suffix to train long-token copy fidelity
|
| 118 |
-
if rng.random() < 0.20:
|
| 119 |
-
suffix = rng.choice(["ism", "ology", "ification", "ation", "ological", "ity"])
|
| 120 |
-
singular = singular + suffix
|
| 121 |
-
|
| 122 |
-
plural = singular + ("es" if singular[-1] in "sxz" else "s")
|
| 123 |
-
return singular, plural, ent_type
|
| 124 |
-
SEARCH_FACTS = [
|
| 125 |
-
("Python creator", "Guido van Rossum", "creator"),
|
| 126 |
-
("capital of Australia", "Canberra", "capital"),
|
| 127 |
-
("speed of light", "299792 km/s", "constant"),
|
| 128 |
-
("tallest mountain on Earth", "Mount Everest", "geography"),
|
| 129 |
-
("deepest ocean trench", "Mariana Trench", "geography"),
|
| 130 |
-
("inventor of the telephone", "Alexander Graham Bell", "inventor"),
|
| 131 |
-
("currency of Japan", "Yen", "currency"),
|
| 132 |
-
("largest planet in solar system", "Jupiter", "astronomy"),
|
| 133 |
-
("author of Hamlet", "William Shakespeare", "literature"),
|
| 134 |
-
("chemical symbol for Gold", "Au", "chemistry"),
|
| 135 |
-
("Tokyo population", "14000000", "demographics"),
|
| 136 |
-
("Paris land area", "105 sq km", "geography"),
|
| 137 |
-
("distance from Earth to Moon", "384400 km", "astronomy"),
|
| 138 |
-
("human body temperature", "37 Celsius", "biology"),
|
| 139 |
-
]
|
| 140 |
-
|
| 141 |
-
# Open-domain entity database for Universal Search Delegation
|
| 142 |
-
SEARCH_ENTITIES = [
|
| 143 |
-
# Foods & Plants
|
| 144 |
-
{"entity": "tomato", "plural": "tomatoes", "type": "plant", "snippet": "the edible berry of the plant Solanum lycopersicum, commonly prepared as a culinary vegetable."},
|
| 145 |
-
{"entity": "apple", "plural": "apples", "type": "fruit", "snippet": "a round, edible fruit produced by an apple tree, cultivated worldwide."},
|
| 146 |
-
{"entity": "potato", "plural": "potatoes", "type": "vegetable", "snippet": "a starchy, tuberous root crop of the nightshade family Solanaceae."},
|
| 147 |
-
{"entity": "coffee", "plural": "coffee beans", "type": "plant", "snippet": "a brewed beverage prepared from roasted coffee beans of flowering Coffea species."},
|
| 148 |
-
{"entity": "banana", "plural": "bananas", "type": "fruit", "snippet": "an elongated, edible fruit produced by large herbaceous plants in the genus Musa."},
|
| 149 |
-
{"entity": "onion", "plural": "onions", "type": "vegetable", "snippet": "a pungent vegetable that is the most widely cultivated species of the genus Allium."},
|
| 150 |
-
{"entity": "wheat", "plural": "wheat grains", "type": "crop", "snippet": "a cereal grain cultivated worldwide as a major dietary staple."},
|
| 151 |
-
{"entity": "rice", "plural": "rice grains", "type": "crop", "snippet": "the seed of the grass species Oryza sativa, a primary staple food for billions of people."},
|
| 152 |
-
{"entity": "carrot", "plural": "carrots", "type": "vegetable", "snippet": "a root vegetable, typically orange, rich in beta-carotene and dietary fiber."},
|
| 153 |
-
{"entity": "garlic", "plural": "garlic cloves", "type": "plant", "snippet": "a species of bulbous flowering plant in the genus Allium, used for culinary seasoning."},
|
| 154 |
-
# Animals
|
| 155 |
-
{"entity": "dog", "plural": "dogs", "type": "animal", "snippet": "a domesticated descendant of the wolf, widely kept as a companion and working animal."},
|
| 156 |
-
{"entity": "cat", "plural": "cats", "type": "animal", "snippet": "a small, carnivorous mammal known for its agility, flexible body, and companionship with humans."},
|
| 157 |
-
{"entity": "dolphin", "plural": "dolphins", "type": "animal", "snippet": "a highly intelligent marine mammal belonging to the cetacean order, known for echolocation."},
|
| 158 |
-
{"entity": "elephant", "plural": "elephants", "type": "animal", "snippet": "the largest existing land animal, characterized by a long trunk, tusks, and large ear flaps."},
|
| 159 |
-
{"entity": "penguin", "plural": "penguins", "type": "animal", "snippet": "an aquatic, flightless bird living almost exclusively in the Southern Hemisphere."},
|
| 160 |
-
{"entity": "lion", "plural": "lions", "type": "animal", "snippet": "a large cat of the genus Panthera, native to Africa and India, known for social prides."},
|
| 161 |
-
{"entity": "tiger", "plural": "tigers", "type": "animal", "snippet": "the largest living cat species, identifiable by its reddish-orange coat with dark vertical stripes."},
|
| 162 |
-
{"entity": "whale", "plural": "whales", "type": "animal", "snippet": "a widely distributed and diverse group of fully aquatic placental marine mammals."},
|
| 163 |
-
# Science & Nature
|
| 164 |
-
{"entity": "gravity", "plural": "gravitational forces", "type": "physics", "snippet": "a fundamental interaction which causes mutual attraction between all things with mass or energy."},
|
| 165 |
-
{"entity": "photosynthesis", "plural": "photosynthetic processes", "type": "biology", "snippet": "a biological process used by plants to convert light energy into chemical energy stored in glucose."},
|
| 166 |
-
{"entity": "atom", "plural": "atoms", "type": "physics", "snippet": "the basic unit of a chemical element, consisting of a dense central nucleus surrounded by electrons."},
|
| 167 |
-
{"entity": "DNA", "plural": "DNA molecules", "type": "genetics", "snippet": "deoxyribonucleic acid, a polymer composed of polynucleotide chains carrying genetic instructions."},
|
| 168 |
-
{"entity": "black hole", "plural": "black holes", "type": "astronomy", "snippet": "a region of spacetime where gravity is so strong that nothing, including light, can escape from it."},
|
| 169 |
-
{"entity": "water", "plural": "water molecules", "type": "chemistry", "snippet": "a transparent, odorless chemical substance with chemical formula H2O, essential for life."},
|
| 170 |
-
{"entity": "oxygen", "plural": "oxygen molecules", "type": "chemistry", "snippet": "a chemical element with symbol O and atomic number 8, essential for aerobic respiration."},
|
| 171 |
-
{"entity": "electricity", "plural": "electric currents", "type": "physics", "snippet": "physical phenomena associated with the presence and motion of matter possessing electric charge."},
|
| 172 |
-
# Geography & Landmarks
|
| 173 |
-
{"entity": "Eiffel Tower", "plural": "monuments", "type": "landmark", "snippet": "a wrought-iron lattice tower on the Champ de Mars in Paris, France, named after Gustave Eiffel."},
|
| 174 |
-
{"entity": "Great Wall of China", "plural": "ancient walls", "type": "history", "snippet": "a series of fortifications built across the historical northern borders of ancient Chinese states."},
|
| 175 |
-
{"entity": "Mount Everest", "plural": "peaks", "type": "geography", "snippet": "Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas."},
|
| 176 |
-
{"entity": "Amazon River", "plural": "rivers", "type": "geography", "snippet": "the largest river by discharge volume of water in the world, located in South America."},
|
| 177 |
-
{"entity": "Sahara Desert", "plural": "deserts", "type": "geography", "snippet": "the largest hot desert in the world and the third largest desert overall, covering North Africa."},
|
| 178 |
-
# Technology & Society
|
| 179 |
-
{"entity": "computer", "plural": "computers", "type": "technology", "snippet": "an electronic device for storing and processing data, typically according to binary instructions."},
|
| 180 |
-
{"entity": "internet", "plural": "computer networks", "type": "technology", "snippet": "a global system of interconnected computer networks that uses the Internet protocol suite (TCP/IP)."},
|
| 181 |
-
{"entity": "algorithm", "plural": "algorithms", "type": "computer science", "snippet": "a finite sequence of rigorous instructions used to solve problems or perform a computation."},
|
| 182 |
-
{"entity": "telescope", "plural": "telescopes", "type": "instrument", "snippet": "an optical instrument that makes distant objects appear magnified using lenses or curved mirrors."},
|
| 183 |
-
{"entity": "airplane", "plural": "airplanes", "type": "vehicle", "snippet": "a powered, fixed-wing aircraft that is propelled forward by thrust from an engine."},
|
| 184 |
-
# Historical Figures & Creators
|
| 185 |
-
{"entity": "Guido van Rossum", "plural": "programmers", "type": "person", "snippet": "the Dutch programmer best known as the creator of the Python programming language."},
|
| 186 |
-
{"entity": "Albert Einstein", "plural": "physicists", "type": "person", "snippet": "a theoretical physicist who developed the theory of relativity, a pillar of modern physics."},
|
| 187 |
-
{"entity": "William Shakespeare", "plural": "playwrights", "type": "person", "snippet": "an English playwright, poet, and actor widely regarded as the greatest writer in the English language."},
|
| 188 |
-
{"entity": "Alexander Graham Bell", "plural": "inventors", "type": "person", "snippet": "an inventor and scientist credited with patenting the first practical telephone."},
|
| 189 |
-
{"entity": "Isaac Newton", "plural": "scientists", "type": "person", "snippet": "an English mathematician and physicist who formulated the laws of motion and universal gravitation."},
|
| 190 |
-
{"entity": "Nikola Tesla", "plural": "engineers", "type": "person", "snippet": "a Serbian-American engineer known for his contributions to alternating current electricity."},
|
| 191 |
-
{"entity": "Marie Curie", "plural": "scientists", "type": "person", "snippet": "a Polish-French physicist who pioneered radioactivity research and won two Nobel Prizes."},
|
| 192 |
-
{"entity": "Alan Turing", "plural": "mathematicians", "type": "person", "snippet": "an English mathematician widely considered the father of modern computer science and AI."},
|
| 193 |
-
{"entity": "Ada Lovelace", "plural": "mathematicians", "type": "person", "snippet": "an English mathematician celebrated for writing the first computer algorithm."},
|
| 194 |
-
{"entity": "Aristotle", "plural": "philosophers", "type": "person", "snippet": "an ancient Greek philosopher whose work shaped centuries of Western logic and metaphysics."},
|
| 195 |
-
{"entity": "Leonardo da Vinci", "plural": "polymaths", "type": "person", "snippet": "an Italian polymath active as a painter, draughtsman, engineer, and scientist."},
|
| 196 |
-
# Additional Produce & Animals
|
| 197 |
-
{"entity": "lemon", "plural": "lemons", "type": "fruit", "snippet": "a species of small evergreen tree native to Asia, known for its yellow ellipsoidal citrus fruit."},
|
| 198 |
-
{"entity": "strawberry", "plural": "strawberries", "type": "fruit", "snippet": "a widely grown hybrid species of the genus Fragaria, cultivated for its sweet red fruit."},
|
| 199 |
-
{"entity": "grape", "plural": "grapes", "type": "fruit", "snippet": "a botanical berry of the deciduous woody vines of the flowering plant genus Vitis."},
|
| 200 |
-
{"entity": "spinach", "plural": "spinach leaves", "type": "vegetable", "snippet": "an edible flowering plant in the family Amaranthaceae, rich in iron and vitamins."},
|
| 201 |
-
{"entity": "broccoli", "plural": "broccoli florets", "type": "vegetable", "snippet": "an edible green plant in the cabbage family whose large flower head is eaten as a vegetable."},
|
| 202 |
-
{"entity": "avocado", "plural": "avocados", "type": "fruit", "snippet": "a medium-sized, evergreen tree native to the Americas, harvested for its buttery fruit."},
|
| 203 |
-
{"entity": "wolf", "plural": "wolves", "type": "animal", "snippet": "a large canine native to Eurasia and North America, known for pack hunting behavior."},
|
| 204 |
-
{"entity": "eagle", "plural": "eagles", "type": "animal", "snippet": "a large, powerfully built bird of prey belonging to the family Accipitridae."},
|
| 205 |
-
{"entity": "shark", "plural": "sharks", "type": "animal", "snippet": "a group of elasmobranch fish characterized by a cartilaginous skeleton and multiple gill slits."},
|
| 206 |
-
{"entity": "octopus", "plural": "octopuses", "type": "animal", "snippet": "a soft-bodied, eight-limbed mollusc of the order Octopoda, famous for high intelligence."},
|
| 207 |
-
# Science & Computing
|
| 208 |
-
{"entity": "quantum entanglement", "plural": "entangled states", "type": "physics", "snippet": "a physical phenomenon occurring when pairs of particles interact such that quantum states cannot be described independently."},
|
| 209 |
-
{"entity": "thermodynamics", "plural": "thermodynamic laws", "type": "physics", "snippet": "a branch of physics that deals with heat, work, temperature, and statistical mechanics."},
|
| 210 |
-
{"entity": "compiler", "plural": "compilers", "type": "computer science", "snippet": "a computer program that translates computer code written in one language into another."},
|
| 211 |
-
{"entity": "operating system", "plural": "operating systems", "type": "computer science", "snippet": "system software that manages computer hardware, software resources, and common services."},
|
| 212 |
-
{"entity": "neural network", "plural": "neural networks", "type": "computer science", "snippet": "a computational model inspired by the biological neural networks of animal brains."},
|
| 213 |
-
# Alphanumeric, Scientific & Tech Entities
|
| 214 |
-
{"entity": "CRISPR Cas9", "plural": "CRISPR Cas9 systems", "type": "genetics", "snippet": "a gene editing technology using an RNA-guided endonuclease enzyme."},
|
| 215 |
-
{"entity": "Boeing 747", "plural": "Boeing 747 aircraft", "type": "aerospace", "snippet": "a wide-body commercial airliner and cargo transport aircraft."},
|
| 216 |
-
{"entity": "Apollo 11", "plural": "Apollo missions", "type": "spaceflight", "snippet": "the American spaceflight that first landed humans on the Moon in 1969."},
|
| 217 |
-
{"entity": "Web3", "plural": "decentralized protocols", "type": "technology", "snippet": "an idea for an iteration of the World Wide Web based on blockchain technology."},
|
| 218 |
-
{"entity": "Formula 1", "plural": "Formula 1 races", "type": "motorsport", "snippet": "the highest class of international racing for open-wheel single-seater formula cars."},
|
| 219 |
-
{"entity": "IPv6", "plural": "IPv6 addresses", "type": "networking", "snippet": "the most recent version of the Internet Protocol providing a vast numerical address space."},
|
| 220 |
-
# Classical, Philosophical & Cultural Terms
|
| 221 |
-
{"entity": "Marcus Aurelius", "plural": "Roman emperors", "type": "person", "snippet": "a Roman emperor and Stoic philosopher who ruled from 161 to 180 AD."},
|
| 222 |
-
{"entity": "brutalist architecture", "plural": "brutalist structures", "type": "architecture", "snippet": "an architectural style characterized by minimalist constructions showcasing bare building materials."},
|
| 223 |
-
{"entity": "existentialism", "plural": "existentialist doctrines", "type": "philosophy", "snippet": "a philosophical inquiry exploring human existence, freedom, and meaning."},
|
| 224 |
-
{"entity": "Ragnarok", "plural": "mythological battles", "type": "mythology", "snippet": "in Norse mythology, a series of catastrophic events leading to the renewal of the world."},
|
| 225 |
-
{"entity": "theremin", "plural": "theremins", "type": "instrument", "snippet": "an electronic musical instrument controlled without physical contact by metal antennas."},
|
| 226 |
-
{"entity": "hyperinflation", "plural": "hyperinflationary cycles", "type": "economics", "snippet": "rapid, excessive, and out-of-control general price increases in an economy."},
|
| 227 |
-
{"entity": "quantum physics", "plural": "quantum systems", "type": "physics", "snippet": "a fundamental branch of physics that studies nature at the scale of atoms and subatomic particles."},
|
| 228 |
-
{"entity": "interstellar", "plural": "interstellar regions", "type": "astronomy", "snippet": "the matter and radiation that exists in the space between the star systems in a galaxy."},
|
| 229 |
-
{"entity": "tiramisu", "plural": "tiramisus", "type": "cuisine", "snippet": "a coffee-flavoured Italian dessert made of ladyfingers dipped in coffee and layered with mascarpone."},
|
| 230 |
-
{"entity": "neutron stars", "plural": "neutron stars", "type": "astronomy", "snippet": "the collapsed core of a massive supergiant star composed almost entirely of neutrons."},
|
| 231 |
-
{"entity": "transformers in deep learning", "plural": "transformer architectures", "type": "computer science", "snippet": "a deep learning architecture based on multi-head self-attention mechanisms."},
|
| 232 |
-
]
|
| 233 |
-
DEPARTMENTS = ["engineering", "marketing", "sales", "finance", "legal", "product"]
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
# -----------------------------------------------------------------------------
|
| 237 |
-
# 1. Search Delegation & Copy Induction Dataset Generator
|
| 238 |
-
# -----------------------------------------------------------------------------
|
| 239 |
-
|
| 240 |
-
def generate_search_delegation_dataset(
|
| 241 |
-
n_samples: int = 15000,
|
| 242 |
-
seed: int = 42,
|
| 243 |
-
) -> List[Dict[str, str]]:
|
| 244 |
-
"""
|
| 245 |
-
Generates concentrated Search Delegation samples for Induction Head copy training.
|
| 246 |
-
Curriculum:
|
| 247 |
-
- 65% Nonce Words (1-3 syllables) forcing pure Attention Pointer Copying (memorization impossible).
|
| 248 |
-
- 35% Real-World Entities (tomatoes, photosynthesis, etc.) establishing lexical alignment.
|
| 249 |
-
- Symmetric punctuation curriculum (50% with '?', '.', 50% raw text without punctuation).
|
| 250 |
-
- Diverse question framing: What are, What is a, Define, Tell me about, Search for, Who is, Explain.
|
| 251 |
-
"""
|
| 252 |
-
rng = random.Random(seed)
|
| 253 |
-
samples: List[Dict[str, str]] = []
|
| 254 |
-
|
| 255 |
-
for _ in range(n_samples):
|
| 256 |
-
is_nonce = (rng.random() < 0.65)
|
| 257 |
-
has_punct = (rng.random() < 0.50)
|
| 258 |
-
|
| 259 |
-
if is_nonce:
|
| 260 |
-
ent, plural, ent_type = generate_nonce_entity(rng)
|
| 261 |
-
snippet = f"a hypothetical {ent_type} characterized in verified theoretical studies."
|
| 262 |
-
else:
|
| 263 |
-
ent_item = rng.choice(SEARCH_ENTITIES)
|
| 264 |
-
ent = ent_item["entity"]
|
| 265 |
-
plural = ent_item.get("plural", ent + "s")
|
| 266 |
-
snippet = ent_item["snippet"]
|
| 267 |
-
ent_type = ent_item.get("type", "concept")
|
| 268 |
-
|
| 269 |
-
# Choose query framing
|
| 270 |
-
frame = rng.choice([
|
| 271 |
-
"what_are_plural",
|
| 272 |
-
"what_is_singular",
|
| 273 |
-
"what_is_concept",
|
| 274 |
-
"define",
|
| 275 |
-
"tell_me_about",
|
| 276 |
-
"search_for",
|
| 277 |
-
"who_is",
|
| 278 |
-
"explain",
|
| 279 |
-
"lookup",
|
| 280 |
-
])
|
| 281 |
-
|
| 282 |
-
if frame == "what_are_plural":
|
| 283 |
-
q_mark = "?" if has_punct else ""
|
| 284 |
-
user_prompt = f"What are {plural}{q_mark}"
|
| 285 |
-
search_q = plural
|
| 286 |
-
thought = rng.choice([
|
| 287 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 288 |
-
"The user requests factual information. Querying search.",
|
| 289 |
-
"The user is seeking knowledge on this topic. I will search the web.",
|
| 290 |
-
])
|
| 291 |
-
answer = f"{plural.capitalize()} are {snippet}"
|
| 292 |
-
elif frame == "what_is_singular" and ent_type in ["plant", "fruit", "vegetable", "animal", "creature", "artifact", "instrument", "vehicle"]:
|
| 293 |
-
article = "an" if ent[0].lower() in "aeiou" else "a"
|
| 294 |
-
q_mark = "?" if has_punct else ""
|
| 295 |
-
user_prompt = f"What is {article} {ent}{q_mark}"
|
| 296 |
-
search_q = ent
|
| 297 |
-
thought = rng.choice([
|
| 298 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 299 |
-
"The user is inquiring about this entity. I will query the search tool.",
|
| 300 |
-
"The user requests factual information. Querying search.",
|
| 301 |
-
])
|
| 302 |
-
answer = f"{ent.capitalize()} is {snippet}"
|
| 303 |
-
elif frame == "define":
|
| 304 |
-
p_mark = "." if has_punct else ""
|
| 305 |
-
user_prompt = f"Define {ent}{p_mark}"
|
| 306 |
-
search_q = ent
|
| 307 |
-
thought = rng.choice([
|
| 308 |
-
"The user asks for a definition. Calling search tool.",
|
| 309 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 310 |
-
"The user requests factual information. Querying search.",
|
| 311 |
-
])
|
| 312 |
-
answer = f"{ent.capitalize()} is defined as {snippet}"
|
| 313 |
-
elif frame == "tell_me_about":
|
| 314 |
-
p_mark = "." if has_punct else ""
|
| 315 |
-
user_prompt = f"Tell me about {ent}{p_mark}"
|
| 316 |
-
search_q = ent
|
| 317 |
-
thought = rng.choice([
|
| 318 |
-
"The user requests factual background information. Querying search.",
|
| 319 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 320 |
-
"The user is seeking knowledge on this topic. I will search the web.",
|
| 321 |
-
])
|
| 322 |
-
answer = f"According to verified search results, {ent} is {snippet}"
|
| 323 |
-
elif frame == "search_for":
|
| 324 |
-
p_mark = "." if has_punct else ""
|
| 325 |
-
user_prompt = f"Search for {ent}{p_mark}"
|
| 326 |
-
search_q = ent
|
| 327 |
-
thought = rng.choice([
|
| 328 |
-
"The user explicitly requests a search. Calling search tool.",
|
| 329 |
-
"The user requests factual information. Querying search.",
|
| 330 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 331 |
-
])
|
| 332 |
-
answer = f"Search results indicate {ent} is {snippet}"
|
| 333 |
-
elif frame == "who_is" and ent_type in ["person", "creature"]:
|
| 334 |
-
q_mark = "?" if has_punct else ""
|
| 335 |
-
user_prompt = f"Who was {ent}{q_mark}" if rng.random() < 0.5 else f"Who is {ent}{q_mark}"
|
| 336 |
-
search_q = ent
|
| 337 |
-
thought = rng.choice([
|
| 338 |
-
"The user requests biographical information. Calling search.",
|
| 339 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 340 |
-
"The user requests factual information. Querying search.",
|
| 341 |
-
])
|
| 342 |
-
answer = f"{ent} was {snippet}" if "was" in user_prompt else f"{ent} is {snippet}"
|
| 343 |
-
elif frame == "explain":
|
| 344 |
-
p_mark = "." if has_punct else ""
|
| 345 |
-
user_prompt = f"Explain {ent}{p_mark}"
|
| 346 |
-
search_q = ent
|
| 347 |
-
thought = rng.choice([
|
| 348 |
-
"The user asks for an explanation of this subject. I will query the search tool.",
|
| 349 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 350 |
-
"The user requests factual information. Querying search.",
|
| 351 |
-
])
|
| 352 |
-
answer = f"{ent.capitalize()} is {snippet}"
|
| 353 |
-
else:
|
| 354 |
-
q_mark = "?" if has_punct else ""
|
| 355 |
-
user_prompt = f"What is {ent}{q_mark}"
|
| 356 |
-
search_q = ent
|
| 357 |
-
thought = rng.choice([
|
| 358 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 359 |
-
"The user requests factual information. Querying search.",
|
| 360 |
-
"The user is seeking knowledge on this topic. I will search the web.",
|
| 361 |
-
])
|
| 362 |
-
answer = f"{ent.capitalize()} is {snippet}"
|
| 363 |
-
|
| 364 |
-
call = f"search(query=\"{search_q}\")"
|
| 365 |
-
result_json = json.dumps({"snippet": snippet})
|
| 366 |
-
full_text = (
|
| 367 |
-
f"<USER>{user_prompt}</USER>"
|
| 368 |
-
f"<THINK>{thought}</THINK>"
|
| 369 |
-
f"<CALL>{call}</CALL>"
|
| 370 |
-
f"<RESULT>{result_json}</RESULT>"
|
| 371 |
-
f"<ANSWER>{answer}</ANSWER>"
|
| 372 |
-
)
|
| 373 |
-
|
| 374 |
-
samples.append({
|
| 375 |
-
"task_type": "tool",
|
| 376 |
-
"category": "search_delegation",
|
| 377 |
-
"prompt": f"<USER>{user_prompt}</USER>",
|
| 378 |
-
"full_text": full_text,
|
| 379 |
-
})
|
| 380 |
-
|
| 381 |
-
return samples
|
| 382 |
-
|
| 383 |
-
|
| 384 |
-
# -----------------------------------------------------------------------------
|
| 385 |
-
# 1. High-Digit Arithmetic Dataset Generator
|
| 386 |
-
# -----------------------------------------------------------------------------
|
| 387 |
-
|
| 388 |
-
def generate_arithmetic_equations(
|
| 389 |
-
n_samples: int = 4000,
|
| 390 |
-
seed: int = 42,
|
| 391 |
-
max_train_digits: int = 15,
|
| 392 |
-
) -> List[Dict[str, str]]:
|
| 393 |
-
"""
|
| 394 |
-
Generates balanced multi-digit addition equations with column carry scratchpad.
|
| 395 |
-
Curriculum: 1 to max_train_digits (default 15) digits, plus edge cases:
|
| 396 |
-
- Zero padding
|
| 397 |
-
- Cascade carries (e.g. 9999 + 1)
|
| 398 |
-
- Equal length and asymmetric lengths
|
| 399 |
-
"""
|
| 400 |
-
rng = random.Random(seed)
|
| 401 |
-
samples: List[Dict[str, str]] = []
|
| 402 |
-
|
| 403 |
-
for _ in range(n_samples):
|
| 404 |
-
mode = rng.random()
|
| 405 |
-
|
| 406 |
-
if mode < 0.20:
|
| 407 |
-
# Cascade carry stress cases (999... + 1 or 999... + small)
|
| 408 |
-
digits = rng.randint(2, max_train_digits)
|
| 409 |
-
a = int("9" * digits)
|
| 410 |
-
b = rng.randint(1, 9)
|
| 411 |
-
elif mode < 0.35:
|
| 412 |
-
# Round numbers / power of 10 zero tracking
|
| 413 |
-
d1 = rng.randint(2, max_train_digits)
|
| 414 |
-
d2 = rng.randint(2, max_train_digits)
|
| 415 |
-
a = rng.randint(1, 9) * (10 ** (d1 - 1))
|
| 416 |
-
b = rng.randint(1, 9) * (10 ** (d2 - 1))
|
| 417 |
-
elif mode < 0.60:
|
| 418 |
-
# Uniform length random digits (1 to max_train_digits)
|
| 419 |
-
d = rng.randint(1, max_train_digits)
|
| 420 |
-
min_v = 10 ** (d - 1) if d > 1 else 0
|
| 421 |
-
max_v = (10 ** d) - 1
|
| 422 |
-
a = rng.randint(min_v, max_v)
|
| 423 |
-
b = rng.randint(min_v, max_v)
|
| 424 |
-
else:
|
| 425 |
-
# Asymmetric length random digits
|
| 426 |
-
d1 = rng.randint(1, max_train_digits)
|
| 427 |
-
d2 = rng.randint(1, max_train_digits)
|
| 428 |
-
min_v1 = 10 ** (d1 - 1) if d1 > 1 else 0
|
| 429 |
-
max_v1 = (10 ** d1) - 1
|
| 430 |
-
min_v2 = 10 ** (d2 - 1) if d2 > 1 else 0
|
| 431 |
-
max_v2 = (10 ** d2) - 1
|
| 432 |
-
a = rng.randint(min_v1, max_v1)
|
| 433 |
-
b = rng.randint(min_v2, max_v2)
|
| 434 |
-
|
| 435 |
-
full_text = format_addition_char_scratchpad(a, b)
|
| 436 |
-
prompt = full_text.split("<THINK>")[0]
|
| 437 |
-
samples.append({
|
| 438 |
-
"task_type": "math",
|
| 439 |
-
"prompt": prompt,
|
| 440 |
-
"full_text": full_text,
|
| 441 |
-
})
|
| 442 |
-
|
| 443 |
-
return samples
|
| 444 |
-
|
| 445 |
-
|
| 446 |
-
# -----------------------------------------------------------------------------
|
| 447 |
-
# 2. Expanded Tool Repertoire Generator
|
| 448 |
-
# -----------------------------------------------------------------------------
|
| 449 |
-
|
| 450 |
-
def generate_tool_example(rng: random.Random, category: Optional[str] = None) -> Dict[str, str]:
|
| 451 |
-
"""
|
| 452 |
-
Generates a structured synthetic tool interaction from 6 expanded categories:
|
| 453 |
-
1. Single-tool calling (calculator, weather, search, time, unit_convert)
|
| 454 |
-
2. Multi-tool sequential chaining
|
| 455 |
-
3. Nested JSON parameter schemas
|
| 456 |
-
4. Error handling and recovery
|
| 457 |
-
5. Multi-turn dialogues
|
| 458 |
-
6. Direct conversational routing (no-tool)
|
| 459 |
-
|
| 460 |
-
Enforces 1-Hop Induction Copying:
|
| 461 |
-
- <THINK> emits a clean, entity-agnostic routing intent (no variable copying in <THINK>).
|
| 462 |
-
- <CALL> copies arguments directly from <USER> via 1-hop attention pointers.
|
| 463 |
-
- Symmetric punctuation curriculum (50% with '?'/'.', 50% raw conversational text).
|
| 464 |
-
`category` forces a specific category instead of sampling one.
|
| 465 |
-
"""
|
| 466 |
-
category = category or rng.choices(
|
| 467 |
-
[
|
| 468 |
-
"single_tool",
|
| 469 |
-
"multi_tool_chain",
|
| 470 |
-
"nested_json_schema",
|
| 471 |
-
"error_recovery",
|
| 472 |
-
"multi_turn",
|
| 473 |
-
"no_tool",
|
| 474 |
-
],
|
| 475 |
-
weights=[0.55, 0.12, 0.12, 0.08, 0.06, 0.07],
|
| 476 |
-
k=1,
|
| 477 |
-
)[0]
|
| 478 |
-
has_punct = (rng.random() < 0.50)
|
| 479 |
-
q_mark = "?" if has_punct else ""
|
| 480 |
-
p_mark = "." if has_punct else ""
|
| 481 |
-
|
| 482 |
-
if category == "single_tool":
|
| 483 |
-
sub = rng.choices(
|
| 484 |
-
["weather", "calculator", "time", "unit_convert", "search"],
|
| 485 |
-
weights=[0.28, 0.24, 0.20, 0.20, 0.08],
|
| 486 |
-
k=1,
|
| 487 |
-
)[0]
|
| 488 |
-
if sub == "calculator":
|
| 489 |
-
op = rng.choice(["+", "-", "*", "/"])
|
| 490 |
-
if op == "+":
|
| 491 |
-
a, b = rng.randint(10, 9999), rng.randint(10, 9999)
|
| 492 |
-
res = a + b
|
| 493 |
-
elif op == "-":
|
| 494 |
-
a, b = rng.randint(100, 9999), rng.randint(10, 99)
|
| 495 |
-
res = a - b
|
| 496 |
-
elif op == "*":
|
| 497 |
-
a, b = rng.randint(2, 250), rng.randint(2, 99)
|
| 498 |
-
res = a * b
|
| 499 |
-
else:
|
| 500 |
-
b = rng.randint(2, 50)
|
| 501 |
-
res = rng.randint(2, 200)
|
| 502 |
-
a = res * b
|
| 503 |
-
|
| 504 |
-
use_spaces = (rng.random() < 0.50)
|
| 505 |
-
expr_prompt = f"{a} {op} {b}" if use_spaces else f"{a}{op}{b}"
|
| 506 |
-
# Keep call expression identical to prompt expression or normalized so 1-hop pointer copies cleanly
|
| 507 |
-
call_expr = expr_prompt if rng.random() < 0.65 else f"{a}{op}{b}"
|
| 508 |
-
user_prompt = rng.choice([
|
| 509 |
-
f"Calculate {expr_prompt}{p_mark}",
|
| 510 |
-
f"What is {expr_prompt}{q_mark}",
|
| 511 |
-
f"Compute {expr_prompt}{p_mark}",
|
| 512 |
-
f"Evaluate {expr_prompt}{p_mark}",
|
| 513 |
-
f"Evaluate the mathematical expression {expr_prompt}{p_mark}",
|
| 514 |
-
f"Solve {expr_prompt}{p_mark}",
|
| 515 |
-
])
|
| 516 |
-
thought = rng.choice([
|
| 517 |
-
"The user requests an arithmetic calculation. I will invoke the calculator tool.",
|
| 518 |
-
"The user is asking to evaluate a mathematical expression. Calling calculator.",
|
| 519 |
-
])
|
| 520 |
-
call = f"calculator(expr=\"{call_expr}\")"
|
| 521 |
-
result = json.dumps({"result": res})
|
| 522 |
-
answer = f"The evaluated result of {call_expr} is {res}."
|
| 523 |
-
|
| 524 |
-
elif sub == "weather":
|
| 525 |
-
if rng.random() < 0.35:
|
| 526 |
-
# Nonce city to force pure copy induction (single or multi-word)
|
| 527 |
-
nonce_city, _, _ = generate_nonce_entity(rng, allow_multi_word=True)
|
| 528 |
-
city = " ".join(w.capitalize() for w in nonce_city.split())
|
| 529 |
-
else:
|
| 530 |
-
city = rng.choice(CITIES)
|
| 531 |
-
temp = rng.randint(-5, 35)
|
| 532 |
-
cond = rng.choice(CONDITIONS)
|
| 533 |
-
user_prompt = rng.choice([
|
| 534 |
-
f"What is the weather in {city}{q_mark}",
|
| 535 |
-
f"What is the weather on {city}{q_mark}",
|
| 536 |
-
f"What is the weather for {city}{q_mark}",
|
| 537 |
-
f"What is the current weather forecast for {city}{q_mark}",
|
| 538 |
-
f"What is the weather like in {city}{q_mark}",
|
| 539 |
-
f"How is the weather in {city}{q_mark}",
|
| 540 |
-
f"How is the weather on {city}{q_mark}",
|
| 541 |
-
f"How is the weather for {city}{q_mark}",
|
| 542 |
-
f"Weather in {city}{p_mark}",
|
| 543 |
-
f"Weather on {city}{p_mark}",
|
| 544 |
-
f"Weather for {city}{p_mark}",
|
| 545 |
-
f"Check the weather in {city}{p_mark}",
|
| 546 |
-
f"Check the weather for {city}{p_mark}",
|
| 547 |
-
f"Tell me the weather in {city}{p_mark}",
|
| 548 |
-
f"Forecast for {city}{p_mark}",
|
| 549 |
-
f"{city} weather{p_mark}",
|
| 550 |
-
])
|
| 551 |
-
thought = rng.choice([
|
| 552 |
-
"The user requests current weather information. I will query the get_weather tool.",
|
| 553 |
-
"The user is asking for weather conditions. Querying get_weather.",
|
| 554 |
-
])
|
| 555 |
-
call = f"get_weather(city=\"{city}\")"
|
| 556 |
-
result = json.dumps({"temp": temp, "condition": cond, "units": "celsius"})
|
| 557 |
-
answer = f"The weather in {city} is currently {cond} with a temperature of {temp} degrees Celsius."
|
| 558 |
-
|
| 559 |
-
elif sub == "search":
|
| 560 |
-
search_mode = rng.random()
|
| 561 |
-
if search_mode < 0.50:
|
| 562 |
-
ent, plural, ent_type = generate_nonce_entity(rng, allow_multi_word=True)
|
| 563 |
-
snippet = f"a hypothetical {ent_type} characterized in general knowledge."
|
| 564 |
-
elif search_mode < 0.85 and bool(SEARCH_ENTITIES):
|
| 565 |
-
ent_item = rng.choice(SEARCH_ENTITIES)
|
| 566 |
-
ent = ent_item["entity"]
|
| 567 |
-
plural = ent_item.get("plural", ent + "s")
|
| 568 |
-
snippet = ent_item["snippet"]
|
| 569 |
-
ent_type = ent_item.get("type", "concept")
|
| 570 |
-
else:
|
| 571 |
-
ent = None
|
| 572 |
-
|
| 573 |
-
if ent is not None:
|
| 574 |
-
variant = rng.choice([
|
| 575 |
-
"what_are_plural",
|
| 576 |
-
"what_is_a",
|
| 577 |
-
"what_is",
|
| 578 |
-
"tell_me_about",
|
| 579 |
-
"explain",
|
| 580 |
-
"define",
|
| 581 |
-
"search_for",
|
| 582 |
-
"lookup",
|
| 583 |
-
"who_is",
|
| 584 |
-
])
|
| 585 |
-
|
| 586 |
-
if variant == "what_are_plural":
|
| 587 |
-
user_prompt = f"What are {plural}{q_mark}"
|
| 588 |
-
search_q = plural
|
| 589 |
-
answer = f"{plural.capitalize()} are {snippet}"
|
| 590 |
-
elif variant == "what_is_a" and ent_type in ["plant", "fruit", "vegetable", "animal", "creature", "artifact", "instrument", "vehicle"]:
|
| 591 |
-
article = "an" if ent[0].lower() in "aeiou" else "a"
|
| 592 |
-
user_prompt = f"What is {article} {ent}{q_mark}"
|
| 593 |
-
search_q = ent
|
| 594 |
-
answer = f"{ent.capitalize()} is {snippet}"
|
| 595 |
-
elif variant == "tell_me_about":
|
| 596 |
-
user_prompt = f"Tell me about {ent}{p_mark}"
|
| 597 |
-
search_q = ent
|
| 598 |
-
answer = f"According to verified search results, {ent} is {snippet}"
|
| 599 |
-
elif variant == "define":
|
| 600 |
-
user_prompt = f"Define {ent}{p_mark}"
|
| 601 |
-
search_q = ent
|
| 602 |
-
answer = f"{ent.capitalize()} is defined as {snippet}"
|
| 603 |
-
elif variant == "explain":
|
| 604 |
-
user_prompt = f"Explain {ent}{p_mark}"
|
| 605 |
-
search_q = ent
|
| 606 |
-
answer = f"{ent.capitalize()} is {snippet}"
|
| 607 |
-
elif variant == "search_for":
|
| 608 |
-
user_prompt = f"Search for {ent}{p_mark}"
|
| 609 |
-
search_q = ent
|
| 610 |
-
answer = f"Search results indicate {ent} is {snippet}"
|
| 611 |
-
elif variant == "lookup":
|
| 612 |
-
user_prompt = f"Look up {ent}{p_mark}"
|
| 613 |
-
search_q = ent
|
| 614 |
-
answer = f"Information for {ent}: {snippet}"
|
| 615 |
-
elif variant == "who_is" and ent_type in ["person", "creature"]:
|
| 616 |
-
prefix = "Who was" if rng.random() < 0.5 else "Who is"
|
| 617 |
-
user_prompt = f"{prefix} {ent}{q_mark}"
|
| 618 |
-
search_q = ent
|
| 619 |
-
answer = f"{ent} is {snippet}"
|
| 620 |
-
else:
|
| 621 |
-
user_prompt = f"What is {ent}{q_mark}"
|
| 622 |
-
search_q = ent
|
| 623 |
-
answer = f"{ent.capitalize()} is {snippet}"
|
| 624 |
-
|
| 625 |
-
thought = rng.choice([
|
| 626 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 627 |
-
"The user requests factual information. Querying search.",
|
| 628 |
-
"The user is seeking knowledge on this topic. I will search the web.",
|
| 629 |
-
])
|
| 630 |
-
call = f"search(query=\"{search_q}\")"
|
| 631 |
-
result = json.dumps({"snippet": snippet})
|
| 632 |
-
else:
|
| 633 |
-
topic, fact, _ = rng.choice(SEARCH_FACTS)
|
| 634 |
-
user_prompt = rng.choice([
|
| 635 |
-
f"Who or what is {topic}{q_mark}",
|
| 636 |
-
f"What is {topic}{q_mark}",
|
| 637 |
-
f"Tell me about {topic}{p_mark}",
|
| 638 |
-
])
|
| 639 |
-
thought = "The user is asking a factual question. I will query the search tool."
|
| 640 |
-
call = f"search(query=\"{topic}\")"
|
| 641 |
-
result = json.dumps({"snippet": fact})
|
| 642 |
-
answer = f"According to verified search results, {topic} is {fact}."
|
| 643 |
-
|
| 644 |
-
elif sub == "time":
|
| 645 |
-
tz = rng.choice(TIMEZONES)
|
| 646 |
-
hr = rng.randint(0, 23)
|
| 647 |
-
mn = rng.randint(0, 59)
|
| 648 |
-
time_str = f"{hr:02d}:{mn:02d}"
|
| 649 |
-
user_prompt = rng.choice([
|
| 650 |
-
f"What is the current time in {tz}{q_mark}",
|
| 651 |
-
f"What is the time in {tz}{q_mark}",
|
| 652 |
-
f"What time is it in {tz}{q_mark}",
|
| 653 |
-
f"Current time in {tz}{p_mark}",
|
| 654 |
-
f"Time in {tz}{p_mark}",
|
| 655 |
-
f"Tell me the time in {tz}{p_mark}",
|
| 656 |
-
f"Check the time for {tz}{p_mark}",
|
| 657 |
-
])
|
| 658 |
-
thought = rng.choice([
|
| 659 |
-
"The user requests the current time for a timezone. I will query the get_time tool.",
|
| 660 |
-
"The user is asking for the current time. Querying get_time.",
|
| 661 |
-
])
|
| 662 |
-
call = f"get_time(timezone=\"{tz}\")"
|
| 663 |
-
result = json.dumps({"time": time_str, "timezone": tz})
|
| 664 |
-
answer = f"The current time in {tz} is {time_str}."
|
| 665 |
-
|
| 666 |
-
else: # unit_convert
|
| 667 |
-
val = rng.randint(5, 500)
|
| 668 |
-
from_u, to_u, factor = rng.choice([
|
| 669 |
-
("km", "miles", 0.621371),
|
| 670 |
-
("miles", "km", 1.60934),
|
| 671 |
-
("kg", "lbs", 2.20462),
|
| 672 |
-
("lbs", "kg", 0.453592),
|
| 673 |
-
])
|
| 674 |
-
converted = round(val * factor, 2)
|
| 675 |
-
user_prompt = rng.choice([
|
| 676 |
-
f"Convert {val} {from_u} to {to_u}{p_mark}",
|
| 677 |
-
f"How many {to_u} is {val} {from_u}{q_mark}",
|
| 678 |
-
f"How many {to_u} are in {val} {from_u}{q_mark}",
|
| 679 |
-
f"What is {val} {from_u} in {to_u}{q_mark}",
|
| 680 |
-
f"Change {val} {from_u} into {to_u}{p_mark}",
|
| 681 |
-
])
|
| 682 |
-
thought = rng.choice([
|
| 683 |
-
"The user requests a unit conversion. I will invoke the unit_convert tool.",
|
| 684 |
-
"The user is asking to convert between measurement units. Calling unit_convert.",
|
| 685 |
-
])
|
| 686 |
-
call = f"unit_convert(value={val}, from_unit=\"{from_u}\", to_unit=\"{to_u}\")"
|
| 687 |
-
result = json.dumps({"value": converted, "unit": to_u})
|
| 688 |
-
answer = f"{val} {from_u} is equal to {converted} {to_u}."
|
| 689 |
-
|
| 690 |
-
full_text = (
|
| 691 |
-
f"<USER>{user_prompt}</USER>"
|
| 692 |
-
f"<THINK>{thought}</THINK>"
|
| 693 |
-
f"<CALL>{call}</CALL>"
|
| 694 |
-
f"<RESULT>{result}</RESULT>"
|
| 695 |
-
f"<ANSWER>{answer}</ANSWER>"
|
| 696 |
-
)
|
| 697 |
-
|
| 698 |
-
elif category == "multi_tool_chain":
|
| 699 |
-
# Chained sequential reasoning: Step 1 -> Tool 1 -> Result 1 -> Step 2 -> Tool 2 -> Result 2 -> Final Answer
|
| 700 |
-
chain_type = rng.choice(["fact_calc", "dual_weather", "tz_calc"])
|
| 701 |
-
if chain_type == "fact_calc":
|
| 702 |
-
pop = 14000000
|
| 703 |
-
area = 2194
|
| 704 |
-
density = round(pop / area, 2)
|
| 705 |
-
user_prompt = f"What is the population density of Tokyo based on its population and area of 2194 sq km{q_mark}"
|
| 706 |
-
thought1 = "First, I need to search for Tokyo's population."
|
| 707 |
-
call1 = "search(query=\"Tokyo population\")"
|
| 708 |
-
result1 = json.dumps({"population": pop})
|
| 709 |
-
thought2 = "Now I will calculate the population density using the calculator tool."
|
| 710 |
-
call2 = f"calculator(expr=\"{pop} / {area}\")"
|
| 711 |
-
result2 = json.dumps({"result": density})
|
| 712 |
-
answer = f"Tokyo's population is {pop:,}. Dividing by 2,194 sq km gives a density of approximately {density} people per sq km."
|
| 713 |
-
|
| 714 |
-
elif chain_type == "dual_weather":
|
| 715 |
-
c1, c2 = rng.sample(CITIES, 2)
|
| 716 |
-
t1, t2 = rng.randint(0, 30), rng.randint(0, 30)
|
| 717 |
-
diff = abs(t1 - t2)
|
| 718 |
-
user_prompt = rng.choice([
|
| 719 |
-
f"Compare the temperature between {c1} and {c2}{p_mark}",
|
| 720 |
-
f"Compare the weather in {c1} and {c2}{p_mark}",
|
| 721 |
-
f"What is the temperature difference between {c1} and {c2}{q_mark}",
|
| 722 |
-
])
|
| 723 |
-
thought1 = "I will first query the weather for the first city."
|
| 724 |
-
call1 = f"get_weather(city=\"{c1}\")"
|
| 725 |
-
result1 = json.dumps({"temp": t1, "condition": "Cloudy"})
|
| 726 |
-
thought2 = "Next, I will query the weather for the second city."
|
| 727 |
-
call2 = f"get_weather(city=\"{c2}\")"
|
| 728 |
-
result2 = json.dumps({"temp": t2, "condition": "Sunny"})
|
| 729 |
-
answer = f"{c1} is {t1}C and {c2} is {t2}C, with a temperature difference of {diff} degrees."
|
| 730 |
-
|
| 731 |
-
else: # tz_calc
|
| 732 |
-
tz1, tz2 = rng.sample(TIMEZONES[:6], 2)
|
| 733 |
-
user_prompt = f"What is the time difference between {tz1} and {tz2}{q_mark}"
|
| 734 |
-
thought1 = "First, I will query the current time in the first timezone."
|
| 735 |
-
call1 = f"get_time(timezone=\"{tz1}\")"
|
| 736 |
-
result1 = json.dumps({"time": "08:00", "utc_offset": -5})
|
| 737 |
-
thought2 = "Next, I will query the current time in the second timezone."
|
| 738 |
-
call2 = f"get_time(timezone=\"{tz2}\")"
|
| 739 |
-
result2 = json.dumps({"time": "22:00", "utc_offset": 9})
|
| 740 |
-
answer = f"{tz1} is UTC-5 and {tz2} is UTC+9, representing a 14-hour time difference."
|
| 741 |
-
|
| 742 |
-
full_text = (
|
| 743 |
-
f"<USER>{user_prompt}</USER>"
|
| 744 |
-
f"<THINK>{thought1}</THINK>"
|
| 745 |
-
f"<CALL>{call1}</CALL>"
|
| 746 |
-
f"<RESULT>{result1}</RESULT>"
|
| 747 |
-
f"<THINK>{thought2}</THINK>"
|
| 748 |
-
f"<CALL>{call2}</CALL>"
|
| 749 |
-
f"<RESULT>{result2}</RESULT>"
|
| 750 |
-
f"<ANSWER>{answer}</ANSWER>"
|
| 751 |
-
)
|
| 752 |
-
|
| 753 |
-
elif category == "nested_json_schema":
|
| 754 |
-
# Complex tools with nested dictionaries, lists, and boolean flags
|
| 755 |
-
schema_type = rng.choice(["db_query", "event_create", "file_manage", "api_fetch"])
|
| 756 |
-
if schema_type == "db_query":
|
| 757 |
-
dept = rng.choice(DEPARTMENTS)
|
| 758 |
-
limit = rng.randint(3, 10)
|
| 759 |
-
user_prompt = rng.choice([
|
| 760 |
-
f"Query the top {limit} active records in the {dept} department{p_mark}",
|
| 761 |
-
f"Query active records in the {dept} department with limit {limit}{p_mark}",
|
| 762 |
-
f"Fetch the top {limit} active entries from {dept}{p_mark}",
|
| 763 |
-
])
|
| 764 |
-
thought = "I will invoke database_query with a nested filter and options schema."
|
| 765 |
-
call = f"database_query(filter={{\"department\": \"{dept}\", \"status\": \"active\"}}, options={{\"limit\": {limit}, \"sort\": \"desc\"}})"
|
| 766 |
-
result = json.dumps({"status": "success", "count": limit, "records": [{"id": 100 + i, "dept": dept} for i in range(min(limit, 3))]})
|
| 767 |
-
answer = f"Found {limit} active records in {dept} according to your filter specifications."
|
| 768 |
-
|
| 769 |
-
elif schema_type == "event_create":
|
| 770 |
-
attendees = ["alice@corp.internal", "bob@corp.internal"]
|
| 771 |
-
user_prompt = f"Schedule a Sprint Sync meeting for tomorrow from 10:00 to 11:00 UTC with Alice and Bob{p_mark}"
|
| 772 |
-
thought = "I need to call create_event with a nested event schedule and attendee list."
|
| 773 |
-
call = "create_event(event={\"title\": \"Sprint Sync\", \"schedule\": {\"date\": \"2026-11-01\", \"start\": \"10:00\", \"end\": \"11:00\", \"tz\": \"UTC\"}}, attendees=[\"alice@corp.internal\", \"bob@corp.internal\"])"
|
| 774 |
-
result = json.dumps({"event_id": "evt_98214", "status": "confirmed"})
|
| 775 |
-
answer = "Successfully scheduled the Sprint Sync meeting with Alice and Bob for 10:00-11:00 UTC (Event ID: evt_98214)."
|
| 776 |
-
|
| 777 |
-
elif schema_type == "file_manage":
|
| 778 |
-
path = "/configs/model_t4.json"
|
| 779 |
-
user_prompt = f"Save model configuration to {path} with fp16 enabled and 2 workers{p_mark}"
|
| 780 |
-
thought = "Calling file_manager with action write and a nested file payload."
|
| 781 |
-
call = f"file_manager(action=\"write\", file={{\"path\": \"{path}\", \"format\": \"json\"}}, content={{\"fp16\": true, \"workers\": 2}})"
|
| 782 |
-
result = json.dumps({"status": "ok", "bytes_written": 48})
|
| 783 |
-
answer = f"Configuration file has been written to {path} with fp16=True and 2 workers."
|
| 784 |
-
|
| 785 |
-
else: # api_fetch
|
| 786 |
-
endpoint = "/v2/metrics/latency"
|
| 787 |
-
user_prompt = f"Fetch latency metrics for EU region from {endpoint}{p_mark}"
|
| 788 |
-
thought = "Calling fetch_api with endpoint and nested filter parameters."
|
| 789 |
-
call = f"fetch_api(endpoint=\"{endpoint}\", params={{\"region\": \"EU\", \"metric\": \"p99\"}}, method=\"GET\")"
|
| 790 |
-
result = json.dumps({"status": 200, "data": {"p99_ms": 14.2}})
|
| 791 |
-
answer = f"Retrieved p99 latency for EU region from {endpoint}: 14.2 ms."
|
| 792 |
-
|
| 793 |
-
full_text = (
|
| 794 |
-
f"<USER>{user_prompt}</USER>"
|
| 795 |
-
f"<THINK>{thought}</THINK>"
|
| 796 |
-
f"<CALL>{call}</CALL>"
|
| 797 |
-
f"<RESULT>{result}</RESULT>"
|
| 798 |
-
f"<ANSWER>{answer}</ANSWER>"
|
| 799 |
-
)
|
| 800 |
-
|
| 801 |
-
elif category == "error_recovery":
|
| 802 |
-
# Robust error reasoning & recovery paths
|
| 803 |
-
err_type = rng.choice(["div_zero", "city_not_found", "api_503_fallback", "table_missing"])
|
| 804 |
-
if err_type == "div_zero":
|
| 805 |
-
num = rng.choice([100, 150, 250, 500])
|
| 806 |
-
user_prompt = f"Calculate {num} / 0"
|
| 807 |
-
thought1 = "The user requests an arithmetic calculation. I will invoke the calculator tool."
|
| 808 |
-
call1 = f"calculator(expr=\"{num}/0\")"
|
| 809 |
-
result1 = json.dumps({"error": "ZeroDivisionError: division by zero"})
|
| 810 |
-
thought2 = "The calculator returned ZeroDivisionError. Division by zero is mathematically undefined. I will explain this clearly."
|
| 811 |
-
answer = "Error: Division by zero is undefined in mathematics. Please specify a non-zero denominator."
|
| 812 |
-
full_text = (
|
| 813 |
-
f"<USER>{user_prompt}</USER>"
|
| 814 |
-
f"<THINK>{thought1}</THINK>"
|
| 815 |
-
f"<CALL>{call1}</CALL>"
|
| 816 |
-
f"<RESULT>{result1}</RESULT>"
|
| 817 |
-
f"<THINK>{thought2}</THINK>"
|
| 818 |
-
f"<ANSWER>{answer}</ANSWER>"
|
| 819 |
-
)
|
| 820 |
-
|
| 821 |
-
elif err_type == "city_not_found":
|
| 822 |
-
city = rng.choice(["Atlantis", "Narnia"])
|
| 823 |
-
user_prompt = f"What is the weather in {city}{q_mark}"
|
| 824 |
-
thought1 = "The user requests current weather information. I will query the get_weather tool."
|
| 825 |
-
call1 = f"get_weather(city=\"{city}\")"
|
| 826 |
-
result1 = json.dumps({"status": "error", "code": 404, "message": f"City '{city}' not found"})
|
| 827 |
-
thought2 = "get_weather returned 404. The requested city is not recognized. I must inform the user."
|
| 828 |
-
answer = f"I could not retrieve weather data for {city} because it is not recognized in our weather database."
|
| 829 |
-
full_text = (
|
| 830 |
-
f"<USER>{user_prompt}</USER>"
|
| 831 |
-
f"<THINK>{thought1}</THINK>"
|
| 832 |
-
f"<CALL>{call1}</CALL>"
|
| 833 |
-
f"<RESULT>{result1}</RESULT>"
|
| 834 |
-
f"<THINK>{thought2}</THINK>"
|
| 835 |
-
f"<ANSWER>{answer}</ANSWER>"
|
| 836 |
-
)
|
| 837 |
-
|
| 838 |
-
elif err_type == "api_503_fallback":
|
| 839 |
-
city = "Tokyo"
|
| 840 |
-
user_prompt = f"Check the primary API forecast for {city}{p_mark}"
|
| 841 |
-
thought1 = "Attempting primary weather API fetch."
|
| 842 |
-
call1 = f"fetch_api(endpoint=\"/v1/weather\", params={{\"city\": \"{city}\"}}, method=\"GET\")"
|
| 843 |
-
result1 = json.dumps({"status": "error", "code": 503, "message": "Service Unavailable"})
|
| 844 |
-
thought2 = "Primary API failed with 503. Recovering via fallback tool get_weather."
|
| 845 |
-
call2 = f"get_weather(city=\"{city}\")"
|
| 846 |
-
result2 = json.dumps({"temp": 22, "condition": "Sunny"})
|
| 847 |
-
answer = "Primary service was temporarily unavailable, but fallback weather confirmed Tokyo is Sunny at 22C."
|
| 848 |
-
full_text = (
|
| 849 |
-
f"<USER>{user_prompt}</USER>"
|
| 850 |
-
f"<THINK>{thought1}</THINK>"
|
| 851 |
-
f"<CALL>{call1}</CALL>"
|
| 852 |
-
f"<RESULT>{result1}</RESULT>"
|
| 853 |
-
f"<THINK>{thought2}</THINK>"
|
| 854 |
-
f"<CALL>{call2}</CALL>"
|
| 855 |
-
f"<RESULT>{result2}</RESULT>"
|
| 856 |
-
f"<ANSWER>{answer}</ANSWER>"
|
| 857 |
-
)
|
| 858 |
-
|
| 859 |
-
else: # table_missing
|
| 860 |
-
user_prompt = f"Query records from table 'legacy_metrics'{p_mark}"
|
| 861 |
-
thought1 = "Querying database for the requested table."
|
| 862 |
-
call1 = "database_query(filter={\"table\": \"legacy_metrics\"}, options={\"limit\": 5})"
|
| 863 |
-
result1 = json.dumps({"status": "error", "code": "TABLE_NOT_FOUND", "message": "Table does not exist"})
|
| 864 |
-
thought2 = "TABLE_NOT_FOUND error encountered. Informing user of invalid table name."
|
| 865 |
-
answer = "The requested database table 'legacy_metrics' was not found. Please verify table name."
|
| 866 |
-
full_text = (
|
| 867 |
-
f"<USER>{user_prompt}</USER>"
|
| 868 |
-
f"<THINK>{thought1}</THINK>"
|
| 869 |
-
f"<CALL>{call1}</CALL>"
|
| 870 |
-
f"<RESULT>{result1}</RESULT>"
|
| 871 |
-
f"<THINK>{thought2}</THINK>"
|
| 872 |
-
f"<ANSWER>{answer}</ANSWER>"
|
| 873 |
-
)
|
| 874 |
-
|
| 875 |
-
elif category == "multi_turn":
|
| 876 |
-
# Multi-turn conversational dialogue with state continuity
|
| 877 |
-
city = rng.choice(CITIES)
|
| 878 |
-
temp = rng.randint(10, 25)
|
| 879 |
-
w_q = rng.choice([
|
| 880 |
-
f"How is the weather in {city}{q_mark}",
|
| 881 |
-
f"What is the weather in {city}{q_mark}",
|
| 882 |
-
f"What is the weather on {city}{q_mark}",
|
| 883 |
-
])
|
| 884 |
-
full_text = (
|
| 885 |
-
f"<USER>Hi there!</USER>"
|
| 886 |
-
f"<THINK>Conversational greeting without tool.</THINK>"
|
| 887 |
-
f"<ANSWER>Hello! How can I assist you today?</ANSWER>"
|
| 888 |
-
f"<USER>{w_q}</USER>"
|
| 889 |
-
f"<THINK>The user requests current weather information. I will query the get_weather tool.</THINK>"
|
| 890 |
-
f"<CALL>get_weather(city=\"{city}\")</CALL>"
|
| 891 |
-
f"<RESULT>{{\"temp\": {temp}, \"condition\": \"Clear\"}}</RESULT>"
|
| 892 |
-
f"<ANSWER>{city} is currently Clear with a temperature of {temp} degrees Celsius.</ANSWER>"
|
| 893 |
-
f"<USER>Thank you!</USER>"
|
| 894 |
-
f"<THINK>User acknowledges. Polite closing response.</THINK>"
|
| 895 |
-
f"<ANSWER>You are very welcome! Let me know if you need anything else.</ANSWER>"
|
| 896 |
-
)
|
| 897 |
-
|
| 898 |
-
else: # no_tool
|
| 899 |
-
greeting = rng.choice([
|
| 900 |
-
("Hello! How are you?", "I am doing well, thank you! How may I assist you today?"),
|
| 901 |
-
("Hi there!", "Hello! What can I help you with today?"),
|
| 902 |
-
("Thank you for your assistance.", "You are welcome! Let me know if you have any more questions."),
|
| 903 |
-
("Thank you!", "You are very welcome! Let me know if you need anything else."),
|
| 904 |
-
("Good morning!", "Good morning! How may I help you?"),
|
| 905 |
-
("Who are you?", "I am ScaledGPT, an AI assistant with structured tool-calling and high-digit arithmetic capabilities."),
|
| 906 |
-
("Explain the concept of gradient descent.", "Gradient descent is an optimization algorithm that iteratively moves model weights in the direction of steepest loss decrease."),
|
| 907 |
-
])
|
| 908 |
-
user_prompt, ans = greeting
|
| 909 |
-
thought = "Conversational request that does not require any tool. I will answer directly."
|
| 910 |
-
full_text = (
|
| 911 |
-
f"<USER>{user_prompt}</USER>"
|
| 912 |
-
f"<THINK>{thought}</THINK>"
|
| 913 |
-
f"<ANSWER>{ans}</ANSWER>"
|
| 914 |
-
)
|
| 915 |
-
|
| 916 |
-
return {
|
| 917 |
-
"category": category,
|
| 918 |
-
"full_text": full_text,
|
| 919 |
-
}
|
| 920 |
-
|
| 921 |
-
|
| 922 |
-
def generate_tool_dataset(n_samples: int = 4000, seed: int = 1337) -> List[Dict[str, str]]:
|
| 923 |
-
"""Generates an expanded tool interaction dataset across all 6 categories."""
|
| 924 |
-
rng = random.Random(seed)
|
| 925 |
-
dataset: List[Dict[str, str]] = []
|
| 926 |
-
for _ in range(n_samples):
|
| 927 |
-
ex = generate_tool_example(rng)
|
| 928 |
-
full_text = ex["full_text"]
|
| 929 |
-
prompt = full_text.split("</USER>")[0] + "</USER>"
|
| 930 |
-
dataset.append({
|
| 931 |
-
"task_type": "tools",
|
| 932 |
-
"prompt": prompt,
|
| 933 |
-
"full_text": full_text,
|
| 934 |
-
"category": ex["category"],
|
| 935 |
-
})
|
| 936 |
-
return dataset
|
| 937 |
-
|
| 938 |
-
|
| 939 |
-
# -----------------------------------------------------------------------------
|
| 940 |
-
# 2b. Curriculum V5: Broad-Distribution Copy Spans & Parameterized Schemas
|
| 941 |
-
# -----------------------------------------------------------------------------
|
| 942 |
-
# V4 copy spans came from one narrow phonotactic inventory, so real-world
|
| 943 |
-
# spellings outside it (doubled vowels, lowercase connectors, leading articles)
|
| 944 |
-
# drifted at copy time. V5 widens the span distribution, strips articles from
|
| 945 |
-
# arguments, parameterizes every nested schema, and keeps <THINK> short so
|
| 946 |
-
# nested calls close within a 140-token decode budget.
|
| 947 |
-
|
| 948 |
-
SPAN_ONSETS_EXT = NONCE_ONSETS + [
|
| 949 |
-
"", "", "b", "c", "d", "f", "g", "h", "l", "m", "n", "p", "r", "s", "t", "w", "y",
|
| 950 |
-
"ph", "sh", "kh", "zh", "ts", "ng", "nj", "sr", "vr", "hy", "mb", "nd", "rh", "x",
|
| 951 |
-
]
|
| 952 |
-
SPAN_NUCLEI_EXT = NONCE_NUCLEI + ["aa", "ii", "uu", "ae", "ia", "io", "ua", "ei", "au", "eo", "ie", "ya"]
|
| 953 |
-
SPAN_CODAS_EXT = NONCE_CODAS + ["", "", "", "n", "r", "s", "l", "ng", "rm", "nd", "st", "x", "ch", "lt", "ss"]
|
| 954 |
-
SPAN_CONNECTORS = ["of", "van", "de", "da", "del", "es", "la", "al", "bin", "von", "le", "and", "y", "el"]
|
| 955 |
-
LETTERS = "abcdefghijklmnopqrstuvwxyz"
|
| 956 |
-
|
| 957 |
-
V5_TIMEZONES = TIMEZONES + ["MST", "EET", "BST", "SGT", "NZST", "PDT", "EDT", "WET", "KST", "MSK"]
|
| 958 |
-
V5_REGIONS = ["EU", "US", "APAC", "LATAM", "MEA", "UK", "JP", "IN", "BR", "CA"]
|
| 959 |
-
V5_METRICS = ["latency", "errors", "uptime", "cpu", "memory", "requests", "traffic", "bandwidth"]
|
| 960 |
-
V5_DAYS = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday", "tomorrow", "next week"]
|
| 961 |
-
V5_FILE_EXTS = ["txt", "json", "csv", "md", "log", "py", "yaml", "cfg"]
|
| 962 |
-
V5_RECORD_STATUSES = ["active", "inactive", "pending", "archived"]
|
| 963 |
-
|
| 964 |
-
NO_TOOL_OPENERS = [
|
| 965 |
-
"Hello", "Hi", "Hey", "Good morning", "Good afternoon", "Good evening",
|
| 966 |
-
"Greetings", "Hi there", "Hello there", "Hey there", "Howdy",
|
| 967 |
-
]
|
| 968 |
-
NO_TOOL_BODIES = [
|
| 969 |
-
"how are you", "how are you today", "how are you doing", "how is it going",
|
| 970 |
-
"how have you been", "what's up", "nice to meet you", "how is your day",
|
| 971 |
-
"how is your day going", "how are things", "are you there", "how are you doing today",
|
| 972 |
-
]
|
| 973 |
-
NO_TOOL_STANDALONE = [
|
| 974 |
-
("thanks", "Thank you"), ("thanks", "Thanks a lot"), ("thanks", "Thanks for the help"),
|
| 975 |
-
("thanks", "Thank you so much"), ("thanks", "Much appreciated"), ("thanks", "Great, thanks"),
|
| 976 |
-
("bye", "Goodbye"), ("bye", "See you later"), ("bye", "Good night"), ("bye", "Bye for now"),
|
| 977 |
-
("identity", "Who are you"), ("identity", "What can you do"), ("identity", "What is your name"),
|
| 978 |
-
("identity", "Are you a robot"), ("ack", "Okay"), ("ack", "Nice work"), ("ack", "That was helpful"),
|
| 979 |
-
("ack", "Cool"), ("ack", "Sounds good"), ("ack", "Got it"),
|
| 980 |
-
]
|
| 981 |
-
NO_TOOL_REPLIES = {
|
| 982 |
-
"greeting": [
|
| 983 |
-
"Hello! I am doing well, thank you. How can I help you today?",
|
| 984 |
-
"Hi! Everything is running smoothly. What can I do for you?",
|
| 985 |
-
"Hello! How may I assist you today?",
|
| 986 |
-
],
|
| 987 |
-
"thanks": [
|
| 988 |
-
"You are welcome! Let me know if you need anything else.",
|
| 989 |
-
"Happy to help! Feel free to ask another question.",
|
| 990 |
-
],
|
| 991 |
-
"bye": ["Goodbye! Have a great day.", "See you soon! Take care."],
|
| 992 |
-
"identity": [
|
| 993 |
-
"I am CBT, a compact assistant that can call tools for weather, time, search, unit conversion, and arithmetic.",
|
| 994 |
-
],
|
| 995 |
-
"ack": ["Great! Let me know if there is anything else I can do.", "Glad to hear it. What else can I help with?"],
|
| 996 |
-
}
|
| 997 |
-
|
| 998 |
-
|
| 999 |
-
def _span_word(rng: random.Random) -> str:
|
| 1000 |
-
"""Samples one lowercase word from a mixture of phonotactic, nonce, random-letter, and alphanumeric shapes."""
|
| 1001 |
-
r = rng.random()
|
| 1002 |
-
if r < 0.55:
|
| 1003 |
-
w = ""
|
| 1004 |
-
while not 2 <= len(w) <= 14:
|
| 1005 |
-
n_syl = rng.choice([1, 2, 2, 2, 3, 3, 4])
|
| 1006 |
-
w = "".join(
|
| 1007 |
-
rng.choice(SPAN_ONSETS_EXT) + rng.choice(SPAN_NUCLEI_EXT)
|
| 1008 |
-
+ (rng.choice(SPAN_CODAS_EXT) if rng.random() < 0.5 else "")
|
| 1009 |
-
for _ in range(n_syl)
|
| 1010 |
-
)
|
| 1011 |
-
elif r < 0.75:
|
| 1012 |
-
w, _, _ = generate_nonce_entity(rng, allow_multi_word=False)
|
| 1013 |
-
w = w.lower()
|
| 1014 |
-
elif r < 0.90:
|
| 1015 |
-
w = "".join(rng.choice(LETTERS) for _ in range(rng.randint(3, 12)))
|
| 1016 |
-
else:
|
| 1017 |
-
stem = "".join(rng.choice(LETTERS) for _ in range(rng.randint(1, 6)))
|
| 1018 |
-
num = str(rng.randint(0, 999))
|
| 1019 |
-
w = rng.choice([stem + num, stem + "-" + num, num + stem, stem.upper() + num])
|
| 1020 |
-
return w if len(w) >= 2 else w + rng.choice(LETTERS)
|
| 1021 |
-
|
| 1022 |
-
|
| 1023 |
-
def generate_copy_span(rng: random.Random, max_words: int = 4, title_prob: float = 0.5) -> str:
|
| 1024 |
-
"""Samples a 1-4 word span with mixed casing and optional lowercase connectors ('X de Y' shape)."""
|
| 1025 |
-
n_words = rng.choices([1, 2, 3, 4][:max_words], weights=[0.50, 0.30, 0.15, 0.05][:max_words], k=1)[0]
|
| 1026 |
-
words = [_span_word(rng) for _ in range(n_words)]
|
| 1027 |
-
while len(" ".join(words)) > 36:
|
| 1028 |
-
words = [_span_word(rng) for _ in range(n_words)]
|
| 1029 |
-
casing = rng.random()
|
| 1030 |
-
if casing < title_prob:
|
| 1031 |
-
words = [w[0].upper() + w[1:] for w in words]
|
| 1032 |
-
elif casing < title_prob + 0.05:
|
| 1033 |
-
words = [w.upper() for w in words]
|
| 1034 |
-
elif casing < title_prob + 0.10:
|
| 1035 |
-
words[0] = words[0][0].upper() + words[0][1:]
|
| 1036 |
-
if n_words >= 2 and rng.random() < 0.25:
|
| 1037 |
-
words.insert(rng.randint(1, n_words - 1), rng.choice(SPAN_CONNECTORS))
|
| 1038 |
-
return " ".join(words)
|
| 1039 |
-
|
| 1040 |
-
|
| 1041 |
-
def _v5_entity(rng: random.Random) -> Tuple[str, str]:
|
| 1042 |
-
"""Returns (entity, snippet), mixing broad copy spans with case-perturbed known entities."""
|
| 1043 |
-
if rng.random() < 0.70:
|
| 1044 |
-
ent = generate_copy_span(rng)
|
| 1045 |
-
return ent, "a topic described in verified reference sources."
|
| 1046 |
-
item = rng.choice(SEARCH_ENTITIES)
|
| 1047 |
-
ent = item["entity"]
|
| 1048 |
-
r = rng.random()
|
| 1049 |
-
if r < 0.25:
|
| 1050 |
-
ent = ent.lower()
|
| 1051 |
-
elif r < 0.35:
|
| 1052 |
-
ent = ent.title()
|
| 1053 |
-
return ent, item["snippet"]
|
| 1054 |
-
|
| 1055 |
-
|
| 1056 |
-
def _wrap(user_prompt: str, thought: str, call: str, result: str, answer: str) -> str:
|
| 1057 |
-
return (
|
| 1058 |
-
f"<USER>{user_prompt}</USER><THINK>{thought}</THINK><CALL>{call}</CALL>"
|
| 1059 |
-
f"<RESULT>{result}</RESULT><ANSWER>{answer}</ANSWER>"
|
| 1060 |
-
)
|
| 1061 |
-
|
| 1062 |
-
|
| 1063 |
-
def generate_search_v5_example(rng: random.Random) -> Dict[str, str]:
|
| 1064 |
-
"""Search delegation with article-stripped arguments ('Tell me about the X' -> query='X')."""
|
| 1065 |
-
ent, snippet = _v5_entity(rng)
|
| 1066 |
-
q = "?" if rng.random() < 0.5 else ""
|
| 1067 |
-
p = "." if rng.random() < 0.5 else ""
|
| 1068 |
-
article = "an" if ent[0].lower() in "aeiou" else "a"
|
| 1069 |
-
user_prompt = rng.choice([
|
| 1070 |
-
f"What is {ent}{q}", f"What is the {ent}{q}", f"What is {article} {ent}{q}", f"What are {ent}{q}",
|
| 1071 |
-
f"What are the {ent}{q}", f"Tell me about {ent}{p}", f"Tell me about the {ent}{p}",
|
| 1072 |
-
f"Tell me about {article} {ent}{p}", f"Define {ent}{p}", f"Define the term {ent}{p}",
|
| 1073 |
-
f"Explain {ent}{p}", f"Explain the {ent}{p}", f"Search for {ent}{p}", f"Look up {ent}{p}",
|
| 1074 |
-
f"Who is {ent}{q}", f"Who was {ent}{q}", f"Find information about {ent}{p}",
|
| 1075 |
-
f"Give me information on {ent}{p}", f"I want to learn about {ent}{p}",
|
| 1076 |
-
])
|
| 1077 |
-
thought = rng.choice([
|
| 1078 |
-
"The user is asking a factual question. I will query the search tool.",
|
| 1079 |
-
"The user requests factual information. Querying search.",
|
| 1080 |
-
"The user is seeking knowledge on this topic. I will search the web.",
|
| 1081 |
-
])
|
| 1082 |
-
full_text = _wrap(
|
| 1083 |
-
user_prompt, thought, f"search(query=\"{ent}\")",
|
| 1084 |
-
json.dumps({"snippet": snippet}), f"According to search results, {ent} is {snippet}",
|
| 1085 |
-
)
|
| 1086 |
-
return {"category": "search_v5", "full_text": full_text}
|
| 1087 |
-
|
| 1088 |
-
|
| 1089 |
-
def generate_single_tool_v5_example(rng: random.Random) -> Dict[str, str]:
|
| 1090 |
-
"""Weather / calculator / time / unit_convert with broad spans and verbatim calculator expressions."""
|
| 1091 |
-
q = "?" if rng.random() < 0.5 else ""
|
| 1092 |
-
p = "." if rng.random() < 0.5 else ""
|
| 1093 |
-
sub = rng.choices(["weather", "calculator", "time", "unit_convert"], weights=[0.35, 0.35, 0.15, 0.15], k=1)[0]
|
| 1094 |
-
|
| 1095 |
-
if sub == "weather":
|
| 1096 |
-
city = generate_copy_span(rng, max_words=3, title_prob=0.85) if rng.random() < 0.55 else rng.choice(CITIES)
|
| 1097 |
-
temp, cond = rng.randint(-10, 40), rng.choice(CONDITIONS)
|
| 1098 |
-
user_prompt = rng.choice([
|
| 1099 |
-
f"What is the weather in {city}{q}", f"What is the weather on {city}{q}",
|
| 1100 |
-
f"What is the weather for {city}{q}", f"What's the weather in {city}{q}",
|
| 1101 |
-
f"What is the weather like in {city}{q}", f"How is the weather in {city}{q}",
|
| 1102 |
-
f"How is the weather on {city}{q}", f"Weather in {city}{p}", f"Weather on {city}{p}",
|
| 1103 |
-
f"Weather for {city}{p}", f"Check the weather in {city}{p}", f"Check the weather for {city}{p}",
|
| 1104 |
-
f"Tell me the weather in {city}{p}", f"Forecast for {city}{p}", f"{city} weather{p}",
|
| 1105 |
-
f"Is it raining in {city}{q}", f"Temperature in {city}{p}",
|
| 1106 |
-
])
|
| 1107 |
-
thought = rng.choice([
|
| 1108 |
-
"The user requests current weather information. I will query the get_weather tool.",
|
| 1109 |
-
"The user is asking for weather conditions. Querying get_weather.",
|
| 1110 |
-
])
|
| 1111 |
-
call = f"get_weather(city=\"{city}\")"
|
| 1112 |
-
result = json.dumps({"temp": temp, "condition": cond, "units": "celsius"})
|
| 1113 |
-
answer = f"The weather in {city} is currently {cond} with a temperature of {temp} degrees Celsius."
|
| 1114 |
-
|
| 1115 |
-
elif sub == "calculator":
|
| 1116 |
-
op = rng.choice(["+", "-", "*", "/"])
|
| 1117 |
-
if op == "/":
|
| 1118 |
-
b = rng.randint(2, 99)
|
| 1119 |
-
res = rng.randint(2, 999)
|
| 1120 |
-
a = res * b
|
| 1121 |
-
else:
|
| 1122 |
-
a = rng.randint(1, 10 ** rng.randint(1, 5) - 1)
|
| 1123 |
-
b = rng.randint(1, 10 ** rng.randint(1, 4) - 1)
|
| 1124 |
-
res = a + b if op == "+" else (a - b if op == "-" else a * b)
|
| 1125 |
-
expr = rng.choice([f"{a} {op} {b}", f"{a}{op}{b}"])
|
| 1126 |
-
user_prompt = rng.choice([
|
| 1127 |
-
f"Calculate {expr}{p}", f"What is {expr}{q}", f"Compute {expr}{p}", f"Evaluate {expr}{p}",
|
| 1128 |
-
f"Solve {expr}{p}", f"What's {expr}{q}", f"How much is {expr}{q}",
|
| 1129 |
-
])
|
| 1130 |
-
thought = rng.choice([
|
| 1131 |
-
"The user requests an arithmetic calculation. I will invoke the calculator tool.",
|
| 1132 |
-
"The user is asking to evaluate a mathematical expression. Calling calculator.",
|
| 1133 |
-
])
|
| 1134 |
-
call = f"calculator(expr=\"{expr}\")"
|
| 1135 |
-
result = json.dumps({"result": res})
|
| 1136 |
-
answer = f"The evaluated result of {expr} is {res}."
|
| 1137 |
-
|
| 1138 |
-
elif sub == "time":
|
| 1139 |
-
# 40% synthetic uppercase codes so timezone is copied from the prompt, not recalled from a list
|
| 1140 |
-
tz = rng.choice(V5_TIMEZONES) if rng.random() < 0.6 else "".join(rng.choice(LETTERS) for _ in range(rng.randint(2, 5))).upper()
|
| 1141 |
-
time_str = f"{rng.randint(0, 23):02d}:{rng.randint(0, 59):02d}"
|
| 1142 |
-
user_prompt = rng.choice([
|
| 1143 |
-
f"What is the current time in {tz}{q}", f"What is the time in {tz}{q}", f"What time is it in {tz}{q}",
|
| 1144 |
-
f"Current time in {tz}{p}", f"Time in {tz}{p}", f"Tell me the time in {tz}{p}",
|
| 1145 |
-
f"Check the time for {tz}{p}",
|
| 1146 |
-
])
|
| 1147 |
-
thought = "The user is asking for the current time. Querying get_time."
|
| 1148 |
-
call = f"get_time(timezone=\"{tz}\")"
|
| 1149 |
-
result = json.dumps({"time": time_str, "timezone": tz})
|
| 1150 |
-
answer = f"The current time in {tz} is {time_str}."
|
| 1151 |
-
|
| 1152 |
-
else:
|
| 1153 |
-
val = rng.randint(1, 1000)
|
| 1154 |
-
from_u, to_u, factor = rng.choice([
|
| 1155 |
-
("km", "miles", 0.621371), ("miles", "km", 1.60934), ("kg", "lbs", 2.20462), ("lbs", "kg", 0.453592),
|
| 1156 |
-
])
|
| 1157 |
-
user_prompt = rng.choice([
|
| 1158 |
-
f"Convert {val} {from_u} to {to_u}{p}", f"How many {to_u} is {val} {from_u}{q}",
|
| 1159 |
-
f"How many {to_u} are in {val} {from_u}{q}", f"What is {val} {from_u} in {to_u}{q}",
|
| 1160 |
-
f"Change {val} {from_u} into {to_u}{p}",
|
| 1161 |
-
])
|
| 1162 |
-
thought = "The user requests a unit conversion. I will invoke the unit_convert tool."
|
| 1163 |
-
call = f"unit_convert(value={val}, from_unit=\"{from_u}\", to_unit=\"{to_u}\")"
|
| 1164 |
-
result = json.dumps({"value": round(val * factor, 2), "unit": to_u})
|
| 1165 |
-
answer = f"{val} {from_u} is equal to {round(val * factor, 2)} {to_u}."
|
| 1166 |
-
|
| 1167 |
-
return {"category": f"single_{sub}", "full_text": _wrap(user_prompt, thought, call, result, answer)}
|
| 1168 |
-
|
| 1169 |
-
|
| 1170 |
-
def generate_nested_v5_example(rng: random.Random) -> Dict[str, str]:
|
| 1171 |
-
"""Parameterized compact nested schemas; short <THINK> keeps the full call under the decode budget."""
|
| 1172 |
-
p = "." if rng.random() < 0.5 else ""
|
| 1173 |
-
schema = rng.choice(["db_query", "event_create", "file_manage", "api_fetch"])
|
| 1174 |
-
|
| 1175 |
-
if schema == "db_query":
|
| 1176 |
-
dept = rng.choice(DEPARTMENTS) if rng.random() < 0.6 else _span_word(rng).lower()[:12]
|
| 1177 |
-
n = rng.randint(1, 20)
|
| 1178 |
-
status = rng.choice(V5_RECORD_STATUSES)
|
| 1179 |
-
user_prompt = rng.choice([
|
| 1180 |
-
f"Query the top {n} {status} records in the {dept} department{p}",
|
| 1181 |
-
f"Query top {n} {status} records in {dept}{p}",
|
| 1182 |
-
f"Get {n} {status} records from {dept}{p}",
|
| 1183 |
-
f"Show the top {n} {status} {dept} records{p}",
|
| 1184 |
-
f"Fetch the top {n} {status} entries from {dept}{p}",
|
| 1185 |
-
f"List {n} {status} records in the {dept} team{p}",
|
| 1186 |
-
])
|
| 1187 |
-
thought = "Calling database_query."
|
| 1188 |
-
call = f"database_query(filter={{\"department\": \"{dept}\", \"status\": \"{status}\"}}, options={{\"limit\": {n}}})"
|
| 1189 |
-
result = json.dumps({"status": "success", "count": n})
|
| 1190 |
-
answer = f"Found {n} {status} records in {dept}."
|
| 1191 |
-
|
| 1192 |
-
elif schema == "event_create":
|
| 1193 |
-
title = generate_copy_span(rng, max_words=3, title_prob=0.9)
|
| 1194 |
-
h = rng.randint(0, 22)
|
| 1195 |
-
start = f"{h:02d}:{rng.choice([0, 15, 30, 45]):02d}"
|
| 1196 |
-
end = f"{h + 1:02d}:{start[-2:]}"
|
| 1197 |
-
mode = rng.random()
|
| 1198 |
-
if mode < 0.3:
|
| 1199 |
-
day = rng.choice(V5_DAYS)
|
| 1200 |
-
on = "" if day in ("tomorrow", "next week") else "on "
|
| 1201 |
-
user_prompt = rng.choice([
|
| 1202 |
-
f"Create a calendar event for {title} {on}{day}{p}",
|
| 1203 |
-
f"Schedule {title} {on}{day}{p}",
|
| 1204 |
-
f"Add {title} to my calendar {on}{day}{p}",
|
| 1205 |
-
f"Book {title} {on}{day}{p}",
|
| 1206 |
-
])
|
| 1207 |
-
call = f"create_event(event={{\"title\": \"{title}\", \"date\": \"{day}\"}})"
|
| 1208 |
-
start = day
|
| 1209 |
-
elif mode < 0.55:
|
| 1210 |
-
user_prompt = rng.choice([
|
| 1211 |
-
f"Schedule {title} from {start} to {end}{p}",
|
| 1212 |
-
f"Create a calendar event for {title} from {start} to {end}{p}",
|
| 1213 |
-
f"Book {title} between {start} and {end}{p}",
|
| 1214 |
-
])
|
| 1215 |
-
call = f"create_event(event={{\"title\": \"{title}\", \"start\": \"{start}\", \"end\": \"{end}\"}})"
|
| 1216 |
-
else:
|
| 1217 |
-
user_prompt = rng.choice([
|
| 1218 |
-
f"Create a calendar event for {title} at {start}{p}",
|
| 1219 |
-
f"Create an event called {title} at {start}{p}",
|
| 1220 |
-
f"Schedule {title} at {start}{p}",
|
| 1221 |
-
f"Add {title} to my calendar at {start}{p}",
|
| 1222 |
-
f"Set up {title} at {start} tomorrow{p}",
|
| 1223 |
-
])
|
| 1224 |
-
call = f"create_event(event={{\"title\": \"{title}\", \"start\": \"{start}\"}})"
|
| 1225 |
-
thought = "Calling create_event."
|
| 1226 |
-
result = json.dumps({"event_id": f"evt_{rng.randint(10000, 99999)}", "status": "confirmed"})
|
| 1227 |
-
answer = f"Scheduled {title} for {start}."
|
| 1228 |
-
|
| 1229 |
-
elif schema == "file_manage":
|
| 1230 |
-
fname = f"{_span_word(rng).lower()[:10]}.{rng.choice(V5_FILE_EXTS)}"
|
| 1231 |
-
if rng.random() < 0.3:
|
| 1232 |
-
fname = f"{rng.choice(['docs', 'data', 'logs', 'tmp', 'notes'])}/{fname}"
|
| 1233 |
-
action = rng.choices(["write", "read", "delete"], weights=[0.7, 0.15, 0.15], k=1)[0]
|
| 1234 |
-
if action == "write":
|
| 1235 |
-
content = generate_copy_span(rng, max_words=3, title_prob=0.6)
|
| 1236 |
-
user_prompt = rng.choice([
|
| 1237 |
-
f"Save {fname} with content {content}{p}",
|
| 1238 |
-
f"Write {content} to {fname}{p}",
|
| 1239 |
-
f"Create file {fname} containing {content}{p}",
|
| 1240 |
-
f"Save the text {content} in {fname}{p}",
|
| 1241 |
-
])
|
| 1242 |
-
call = f"file_manager(action=\"write\", file={{\"path\": \"{fname}\"}}, content=\"{content}\")"
|
| 1243 |
-
else:
|
| 1244 |
-
verb = "Read" if action == "read" else "Delete"
|
| 1245 |
-
user_prompt = rng.choice([f"{verb} {fname}{p}", f"{verb} the file {fname}{p}"])
|
| 1246 |
-
call = f"file_manager(action=\"{action}\", file={{\"path\": \"{fname}\"}})"
|
| 1247 |
-
thought = "Calling file_manager."
|
| 1248 |
-
result = json.dumps({"status": "ok"})
|
| 1249 |
-
answer = f"File operation {action} completed for {fname}."
|
| 1250 |
-
|
| 1251 |
-
else:
|
| 1252 |
-
segs = [_span_word(rng).lower()[:8] for _ in range(rng.randint(1, 2))]
|
| 1253 |
-
endpoint = f"/v{rng.randint(1, 3)}/" + "/".join(segs)
|
| 1254 |
-
region, metric = rng.choice(V5_REGIONS), rng.choice(V5_METRICS)
|
| 1255 |
-
user_prompt = rng.choice([
|
| 1256 |
-
f"Fetch {metric} metrics for {region} from {endpoint}{p}",
|
| 1257 |
-
f"Fetch {metric} metrics for {region} region from {endpoint}{p}",
|
| 1258 |
-
f"Get {metric} data from {endpoint} for {region}{p}",
|
| 1259 |
-
f"Call {endpoint} for {region} {metric}{p}",
|
| 1260 |
-
])
|
| 1261 |
-
thought = "Calling fetch_api."
|
| 1262 |
-
call = f"fetch_api(endpoint=\"{endpoint}\", params={{\"region\": \"{region}\", \"metric\": \"{metric}\"}})"
|
| 1263 |
-
result = json.dumps({"status": 200, "data": {"value": round(rng.random() * 100, 1)}})
|
| 1264 |
-
answer = f"Retrieved {metric} for {region} from {endpoint}."
|
| 1265 |
-
|
| 1266 |
-
return {"category": f"nested_{schema}", "full_text": _wrap(user_prompt, thought, call, result, answer)}
|
| 1267 |
-
|
| 1268 |
-
|
| 1269 |
-
def generate_error_v5_example(rng: random.Random) -> Dict[str, str]:
|
| 1270 |
-
"""Error recovery traces using the V5 compact schemas."""
|
| 1271 |
-
err = rng.choice(["div_zero", "city_not_found", "api_503_fallback", "table_missing"])
|
| 1272 |
-
if err == "div_zero":
|
| 1273 |
-
num = rng.randint(1, 9999)
|
| 1274 |
-
user_prompt = f"Calculate {num} / 0"
|
| 1275 |
-
steps = [
|
| 1276 |
-
("The user requests an arithmetic calculation. I will invoke the calculator tool.",
|
| 1277 |
-
f"calculator(expr=\"{num} / 0\")", json.dumps({"error": "ZeroDivisionError: division by zero"})),
|
| 1278 |
-
]
|
| 1279 |
-
final_thought = "The calculator returned ZeroDivisionError. I will explain this clearly."
|
| 1280 |
-
answer = "Division by zero is undefined. Please provide a non-zero denominator."
|
| 1281 |
-
elif err == "city_not_found":
|
| 1282 |
-
city = generate_copy_span(rng, max_words=2, title_prob=1.0)
|
| 1283 |
-
user_prompt = f"What is the weather in {city}"
|
| 1284 |
-
steps = [
|
| 1285 |
-
("The user requests current weather information. I will query the get_weather tool.",
|
| 1286 |
-
f"get_weather(city=\"{city}\")", json.dumps({"status": "error", "code": 404})),
|
| 1287 |
-
]
|
| 1288 |
-
final_thought = "get_weather returned 404. I must inform the user."
|
| 1289 |
-
answer = f"I could not retrieve weather data for {city} because it is not recognized."
|
| 1290 |
-
elif err == "api_503_fallback":
|
| 1291 |
-
city = rng.choice(CITIES)
|
| 1292 |
-
user_prompt = f"Check the primary API forecast for {city}"
|
| 1293 |
-
steps = [
|
| 1294 |
-
("Calling fetch_api.", f"fetch_api(endpoint=\"/v1/weather\", params={{\"city\": \"{city}\"}})",
|
| 1295 |
-
json.dumps({"status": 503})),
|
| 1296 |
-
("Primary API failed with 503. Falling back to get_weather.", f"get_weather(city=\"{city}\")",
|
| 1297 |
-
json.dumps({"temp": 22, "condition": "Sunny"})),
|
| 1298 |
-
]
|
| 1299 |
-
final_thought = ""
|
| 1300 |
-
answer = f"The primary service was unavailable, but the fallback reports {city} is Sunny at 22C."
|
| 1301 |
-
else:
|
| 1302 |
-
table = _span_word(rng).lower()[:12]
|
| 1303 |
-
user_prompt = f"Query records from table {table}"
|
| 1304 |
-
steps = [
|
| 1305 |
-
("Calling database_query.", f"database_query(filter={{\"table\": \"{table}\"}}, options={{\"limit\": 5}})",
|
| 1306 |
-
json.dumps({"status": "error", "code": "TABLE_NOT_FOUND"})),
|
| 1307 |
-
]
|
| 1308 |
-
final_thought = "TABLE_NOT_FOUND. Informing the user."
|
| 1309 |
-
answer = f"The table {table} was not found. Please verify the table name."
|
| 1310 |
-
|
| 1311 |
-
text = f"<USER>{user_prompt}</USER>"
|
| 1312 |
-
for thought, call, result in steps:
|
| 1313 |
-
text += f"<THINK>{thought}</THINK><CALL>{call}</CALL><RESULT>{result}</RESULT>"
|
| 1314 |
-
if final_thought:
|
| 1315 |
-
text += f"<THINK>{final_thought}</THINK>"
|
| 1316 |
-
text += f"<ANSWER>{answer}</ANSWER>"
|
| 1317 |
-
return {"category": "error_v5", "full_text": text}
|
| 1318 |
-
|
| 1319 |
-
|
| 1320 |
-
def generate_no_tool_v5_example(rng: random.Random) -> Dict[str, str]:
|
| 1321 |
-
"""Compositional chit-chat (greetings with 'today'/'doing' etc.) routed directly to <ANSWER>."""
|
| 1322 |
-
if rng.random() < 0.6:
|
| 1323 |
-
opener, body = rng.choice(NO_TOOL_OPENERS), rng.choice(NO_TOOL_BODIES)
|
| 1324 |
-
sep = rng.choice([", ", "! ", " ", ". "])
|
| 1325 |
-
user_prompt = f"{opener}{sep}{body}{rng.choice(['?', '', '!'])}"
|
| 1326 |
-
if rng.random() < 0.15:
|
| 1327 |
-
user_prompt = user_prompt.lower()
|
| 1328 |
-
kind = "greeting"
|
| 1329 |
-
else:
|
| 1330 |
-
kind, user_prompt = rng.choice(NO_TOOL_STANDALONE)
|
| 1331 |
-
user_prompt += rng.choice(["", ".", "!", "?"] if kind == "identity" else ["", ".", "!"])
|
| 1332 |
-
thought = "Conversational request that does not require any tool. I will answer directly."
|
| 1333 |
-
full_text = f"<USER>{user_prompt}</USER><THINK>{thought}</THINK><ANSWER>{rng.choice(NO_TOOL_REPLIES[kind])}</ANSWER>"
|
| 1334 |
-
return {"category": "no_tool_v5", "full_text": full_text}
|
| 1335 |
-
|
| 1336 |
-
|
| 1337 |
-
def generate_v5_tool_example(rng: random.Random) -> Dict[str, str]:
|
| 1338 |
-
"""Samples one non-search V5 tool trace; chains and multi-turn reuse the V4 generator."""
|
| 1339 |
-
category = rng.choices(
|
| 1340 |
-
["single", "nested", "no_tool", "error", "chain", "multi_turn"],
|
| 1341 |
-
weights=[0.42, 0.24, 0.14, 0.08, 0.06, 0.06],
|
| 1342 |
-
k=1,
|
| 1343 |
-
)[0]
|
| 1344 |
-
if category == "single":
|
| 1345 |
-
return generate_single_tool_v5_example(rng)
|
| 1346 |
-
if category == "nested":
|
| 1347 |
-
return generate_nested_v5_example(rng)
|
| 1348 |
-
if category == "no_tool":
|
| 1349 |
-
return generate_no_tool_v5_example(rng)
|
| 1350 |
-
if category == "error":
|
| 1351 |
-
return generate_error_v5_example(rng)
|
| 1352 |
-
return generate_tool_example(rng, category="multi_tool_chain" if category == "chain" else "multi_turn")
|
| 1353 |
-
|
| 1354 |
-
|
| 1355 |
-
def to_tool_item(example: Dict[str, str]) -> Dict[str, str]:
|
| 1356 |
-
"""Wraps a generated trace into the encode_joint_batch item format."""
|
| 1357 |
-
full_text = example["full_text"]
|
| 1358 |
-
return {
|
| 1359 |
-
"task_type": "tools",
|
| 1360 |
-
"prompt": full_text.split("</USER>")[0] + "</USER>",
|
| 1361 |
-
"full_text": full_text,
|
| 1362 |
-
"category": example["category"],
|
| 1363 |
-
}
|
| 1364 |
-
|
| 1365 |
-
|
| 1366 |
-
def generate_joint_dataset(
|
| 1367 |
-
n_math: int = 4000,
|
| 1368 |
-
n_tools: int = 4000,
|
| 1369 |
-
seed: int = 42,
|
| 1370 |
-
max_train_digits: int = 15,
|
| 1371 |
-
) -> Tuple[List[Dict[str, str]], List[Dict[str, str]]]:
|
| 1372 |
-
"""Generates dual datasets for math curriculum and expanded tools."""
|
| 1373 |
-
math_samples = generate_arithmetic_equations(n_samples=n_math, seed=seed, max_train_digits=max_train_digits)
|
| 1374 |
-
tool_samples = generate_tool_dataset(n_samples=n_tools, seed=seed + 100)
|
| 1375 |
-
return math_samples, tool_samples
|
| 1376 |
-
|
| 1377 |
-
|
| 1378 |
-
# -----------------------------------------------------------------------------
|
| 1379 |
-
# 3. Batch Encoder & Strict Loss Masker
|
| 1380 |
-
# -----------------------------------------------------------------------------
|
| 1381 |
-
|
| 1382 |
-
def encode_joint_batch(
|
| 1383 |
-
math_items: List[Dict[str, str]],
|
| 1384 |
-
tool_items: List[Dict[str, str]],
|
| 1385 |
-
tokenizer: ScaledTokenizer,
|
| 1386 |
-
max_len: Optional[int] = None,
|
| 1387 |
-
device: Optional[torch.device] = None,
|
| 1388 |
-
) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 1389 |
-
"""
|
| 1390 |
-
Encodes balanced joint batch of math and tool sequences.
|
| 1391 |
-
Strict Prompt & Environment Loss Masking:
|
| 1392 |
-
- Math: masks everything up to and including '<THINK>' (loss on scratchpad + answer).
|
| 1393 |
-
- Tools:
|
| 1394 |
-
- Masks everything inside '<USER>...</USER>' except at '</USER>' (where target is '<THINK>').
|
| 1395 |
-
- Masks everything inside '<RESULT>...</RESULT>' except at '</RESULT>' (where target is '<THINK>' or '<ANSWER>').
|
| 1396 |
-
- Supervised: reasoning traces '<THINK>', tool calls '<CALL>', and answers '<ANSWER>'.
|
| 1397 |
-
"""
|
| 1398 |
-
items = math_items + tool_items
|
| 1399 |
-
B = len(items)
|
| 1400 |
-
|
| 1401 |
-
encoded_seqs = [tokenizer.encode(item["full_text"], add_bos=True, add_eos=True) for item in items]
|
| 1402 |
-
calc_max_len = max(len(s) for s in encoded_seqs) - 1
|
| 1403 |
-
if max_len is not None:
|
| 1404 |
-
batch_max = min(calc_max_len, max_len)
|
| 1405 |
-
else:
|
| 1406 |
-
batch_max = calc_max_len
|
| 1407 |
-
|
| 1408 |
-
x_tensor = torch.full((B, batch_max), tokenizer.pad_id, dtype=torch.long)
|
| 1409 |
-
y_tensor = torch.full((B, batch_max), -100, dtype=torch.long)
|
| 1410 |
-
|
| 1411 |
-
think_s_id = tokenizer.think_s_id
|
| 1412 |
-
user_s_id = tokenizer.user_s_id
|
| 1413 |
-
user_e_id = tokenizer.user_e_id
|
| 1414 |
-
result_s_id = tokenizer.result_s_id
|
| 1415 |
-
result_e_id = tokenizer.result_e_id
|
| 1416 |
-
|
| 1417 |
-
for i, (item, tokens) in enumerate(zip(items, encoded_seqs)):
|
| 1418 |
-
# Truncate if exceeding batch_max + 1
|
| 1419 |
-
if len(tokens) > batch_max + 1:
|
| 1420 |
-
tokens = tokens[: batch_max + 1]
|
| 1421 |
-
|
| 1422 |
-
n = len(tokens) - 1
|
| 1423 |
-
t_tok = torch.tensor(tokens, dtype=torch.long)
|
| 1424 |
-
x_tensor[i, :n] = t_tok[:-1]
|
| 1425 |
-
y_tensor[i, :n] = t_tok[1:].clone()
|
| 1426 |
-
|
| 1427 |
-
if item["task_type"] == "math":
|
| 1428 |
-
# Mask prompt up to '<THINK>'
|
| 1429 |
-
think_matches = (t_tok[:-1] == think_s_id).nonzero(as_tuple=True)[0]
|
| 1430 |
-
if len(think_matches) > 0:
|
| 1431 |
-
mask_until = think_matches[0].item()
|
| 1432 |
-
y_tensor[i, :mask_until] = -100
|
| 1433 |
-
else:
|
| 1434 |
-
# Segment-aware masking for single-turn, multi-turn, and tool responses
|
| 1435 |
-
mask = torch.zeros(n, dtype=torch.bool)
|
| 1436 |
-
in_user = False
|
| 1437 |
-
in_result = False
|
| 1438 |
-
|
| 1439 |
-
for pos in range(n):
|
| 1440 |
-
tok_id = t_tok[pos].item()
|
| 1441 |
-
if tok_id == user_s_id:
|
| 1442 |
-
in_user = True
|
| 1443 |
-
elif tok_id == result_s_id:
|
| 1444 |
-
in_result = True
|
| 1445 |
-
|
| 1446 |
-
if in_user:
|
| 1447 |
-
if tok_id == user_e_id:
|
| 1448 |
-
in_user = False
|
| 1449 |
-
mask[pos] = False
|
| 1450 |
-
else:
|
| 1451 |
-
mask[pos] = True
|
| 1452 |
-
elif in_result:
|
| 1453 |
-
if tok_id == result_e_id:
|
| 1454 |
-
in_result = False
|
| 1455 |
-
mask[pos] = False
|
| 1456 |
-
else:
|
| 1457 |
-
mask[pos] = True
|
| 1458 |
-
elif pos == 0:
|
| 1459 |
-
# BOS token preceding <USER>
|
| 1460 |
-
mask[pos] = True
|
| 1461 |
-
|
| 1462 |
-
y_tensor[i, :n][mask] = -100
|
| 1463 |
-
|
| 1464 |
-
if device is not None:
|
| 1465 |
-
x_tensor = x_tensor.to(device)
|
| 1466 |
-
y_tensor = y_tensor.to(device)
|
| 1467 |
-
|
| 1468 |
-
return x_tensor, y_tensor
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
equalized_15d_1024_and_ruoss_sweep.json
DELETED
|
@@ -1,890 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"part3_18m_1d_at_1024_context": {
|
| 3 |
-
"25d_cascade": {
|
| 4 |
-
"passed": false,
|
| 5 |
-
"zero_loop": false,
|
| 6 |
-
"extracted_answer_prefix": "9991000900999",
|
| 7 |
-
"gen_tail": "+9+0)=10(1+9+0)=10(1+0+0)=01(0+9+0)=09(0+9+0)=09(0+9+0)=09</THINK>=9991000900999"
|
| 8 |
-
},
|
| 9 |
-
"35d_ones_eights": {
|
| 10 |
-
"passed": false,
|
| 11 |
-
"zero_loop": false,
|
| 12 |
-
"extracted_answer_prefix": "08(0+0+0+0101999",
|
| 13 |
-
"gen_tail": "+8)=09(0+1+8)=09(0+8+8)=16(1+1+0)=02(0+1+8)=09(0+1+8)=09(0+0+8)=08(0+0+0+0101999"
|
| 14 |
-
},
|
| 15 |
-
"50d_comp": {
|
| 16 |
-
"passed": false,
|
| 17 |
-
"zero_loop": false,
|
| 18 |
-
"extracted_answer_prefix": "10880+08+8+6+5)=1",
|
| 19 |
-
"gen_tail": "10(1+6+0)=07(0+6+0)=06(0+0+0)=00(0+1+0+0+0+4+1+0+0)=01</THINK>=10880+08+8+6+5)=1"
|
| 20 |
-
},
|
| 21 |
-
"50d_cascade": {
|
| 22 |
-
"passed": false,
|
| 23 |
-
"zero_loop": false,
|
| 24 |
-
"extracted_answer_prefix": "<THINK>99999999999999999999999",
|
| 25 |
-
"gen_tail": "99999999999999999999999999999999999999999999999999999999999999999999999999999999"
|
| 26 |
-
}
|
| 27 |
-
},
|
| 28 |
-
"equalized_15d_1024_comparison": {
|
| 29 |
-
"1d_standard_15d_1024": {
|
| 30 |
-
"meta": {
|
| 31 |
-
"id": "1d_standard_15d_1024",
|
| 32 |
-
"label": "1D-RoPE Standard (1-15d, T=1024)",
|
| 33 |
-
"pos_mode": "1d_standard",
|
| 34 |
-
"k_shift": 0,
|
| 35 |
-
"final_loss": 0.09822,
|
| 36 |
-
"train_sec": 238.36
|
| 37 |
-
},
|
| 38 |
-
"pooled_seen_8_15d": {
|
| 39 |
-
"passed": 0,
|
| 40 |
-
"n": 600,
|
| 41 |
-
"accuracy": 0.0,
|
| 42 |
-
"wilson_95_ci": [
|
| 43 |
-
0.0,
|
| 44 |
-
0.6
|
| 45 |
-
]
|
| 46 |
-
},
|
| 47 |
-
"pooled_unseen_16_40d": {
|
| 48 |
-
"passed": 0,
|
| 49 |
-
"n": 1200,
|
| 50 |
-
"accuracy": 0.0,
|
| 51 |
-
"wilson_95_ci": [
|
| 52 |
-
0.0,
|
| 53 |
-
0.3
|
| 54 |
-
]
|
| 55 |
-
},
|
| 56 |
-
"pooled_unseen_16_50d": {
|
| 57 |
-
"passed": 0,
|
| 58 |
-
"n": 1600,
|
| 59 |
-
"accuracy": 0.0,
|
| 60 |
-
"wilson_95_ci": [
|
| 61 |
-
0.0,
|
| 62 |
-
0.2
|
| 63 |
-
]
|
| 64 |
-
},
|
| 65 |
-
"per_length": {
|
| 66 |
-
"8": {
|
| 67 |
-
"digits": 8,
|
| 68 |
-
"n": 200,
|
| 69 |
-
"passed": 0,
|
| 70 |
-
"accuracy": 0.0,
|
| 71 |
-
"wilson_95_ci": [
|
| 72 |
-
0.0,
|
| 73 |
-
1.9
|
| 74 |
-
],
|
| 75 |
-
"elapsed_sec": 1.39
|
| 76 |
-
},
|
| 77 |
-
"12": {
|
| 78 |
-
"digits": 12,
|
| 79 |
-
"n": 200,
|
| 80 |
-
"passed": 0,
|
| 81 |
-
"accuracy": 0.0,
|
| 82 |
-
"wilson_95_ci": [
|
| 83 |
-
0.0,
|
| 84 |
-
1.9
|
| 85 |
-
],
|
| 86 |
-
"elapsed_sec": 0.85
|
| 87 |
-
},
|
| 88 |
-
"15": {
|
| 89 |
-
"digits": 15,
|
| 90 |
-
"n": 200,
|
| 91 |
-
"passed": 0,
|
| 92 |
-
"accuracy": 0.0,
|
| 93 |
-
"wilson_95_ci": [
|
| 94 |
-
0.0,
|
| 95 |
-
1.9
|
| 96 |
-
],
|
| 97 |
-
"elapsed_sec": 1.2
|
| 98 |
-
},
|
| 99 |
-
"16": {
|
| 100 |
-
"digits": 16,
|
| 101 |
-
"n": 200,
|
| 102 |
-
"passed": 0,
|
| 103 |
-
"accuracy": 0.0,
|
| 104 |
-
"wilson_95_ci": [
|
| 105 |
-
0.0,
|
| 106 |
-
1.9
|
| 107 |
-
],
|
| 108 |
-
"elapsed_sec": 1.28
|
| 109 |
-
},
|
| 110 |
-
"20": {
|
| 111 |
-
"digits": 20,
|
| 112 |
-
"n": 200,
|
| 113 |
-
"passed": 0,
|
| 114 |
-
"accuracy": 0.0,
|
| 115 |
-
"wilson_95_ci": [
|
| 116 |
-
0.0,
|
| 117 |
-
1.9
|
| 118 |
-
],
|
| 119 |
-
"elapsed_sec": 1.75
|
| 120 |
-
},
|
| 121 |
-
"25": {
|
| 122 |
-
"digits": 25,
|
| 123 |
-
"n": 200,
|
| 124 |
-
"passed": 0,
|
| 125 |
-
"accuracy": 0.0,
|
| 126 |
-
"wilson_95_ci": [
|
| 127 |
-
0.0,
|
| 128 |
-
1.9
|
| 129 |
-
],
|
| 130 |
-
"elapsed_sec": 2.53
|
| 131 |
-
},
|
| 132 |
-
"30": {
|
| 133 |
-
"digits": 30,
|
| 134 |
-
"n": 200,
|
| 135 |
-
"passed": 0,
|
| 136 |
-
"accuracy": 0.0,
|
| 137 |
-
"wilson_95_ci": [
|
| 138 |
-
0.0,
|
| 139 |
-
1.9
|
| 140 |
-
],
|
| 141 |
-
"elapsed_sec": 0.39
|
| 142 |
-
},
|
| 143 |
-
"35": {
|
| 144 |
-
"digits": 35,
|
| 145 |
-
"n": 200,
|
| 146 |
-
"passed": 0,
|
| 147 |
-
"accuracy": 0.0,
|
| 148 |
-
"wilson_95_ci": [
|
| 149 |
-
0.0,
|
| 150 |
-
1.9
|
| 151 |
-
],
|
| 152 |
-
"elapsed_sec": 0.37
|
| 153 |
-
},
|
| 154 |
-
"40": {
|
| 155 |
-
"digits": 40,
|
| 156 |
-
"n": 200,
|
| 157 |
-
"passed": 0,
|
| 158 |
-
"accuracy": 0.0,
|
| 159 |
-
"wilson_95_ci": [
|
| 160 |
-
0.0,
|
| 161 |
-
1.9
|
| 162 |
-
],
|
| 163 |
-
"elapsed_sec": 0.44
|
| 164 |
-
},
|
| 165 |
-
"45": {
|
| 166 |
-
"digits": 45,
|
| 167 |
-
"n": 200,
|
| 168 |
-
"passed": 0,
|
| 169 |
-
"accuracy": 0.0,
|
| 170 |
-
"wilson_95_ci": [
|
| 171 |
-
0.0,
|
| 172 |
-
1.9
|
| 173 |
-
],
|
| 174 |
-
"elapsed_sec": 0.54
|
| 175 |
-
},
|
| 176 |
-
"50": {
|
| 177 |
-
"digits": 50,
|
| 178 |
-
"n": 200,
|
| 179 |
-
"passed": 0,
|
| 180 |
-
"accuracy": 0.0,
|
| 181 |
-
"wilson_95_ci": [
|
| 182 |
-
0.0,
|
| 183 |
-
1.9
|
| 184 |
-
],
|
| 185 |
-
"elapsed_sec": 0.63
|
| 186 |
-
},
|
| 187 |
-
"55": {
|
| 188 |
-
"digits": 55,
|
| 189 |
-
"n": 200,
|
| 190 |
-
"passed": 0,
|
| 191 |
-
"accuracy": 0.0,
|
| 192 |
-
"wilson_95_ci": [
|
| 193 |
-
0.0,
|
| 194 |
-
1.9
|
| 195 |
-
],
|
| 196 |
-
"elapsed_sec": 0.65
|
| 197 |
-
},
|
| 198 |
-
"60": {
|
| 199 |
-
"digits": 60,
|
| 200 |
-
"n": 200,
|
| 201 |
-
"passed": 0,
|
| 202 |
-
"accuracy": 0.0,
|
| 203 |
-
"wilson_95_ci": [
|
| 204 |
-
0.0,
|
| 205 |
-
1.9
|
| 206 |
-
],
|
| 207 |
-
"elapsed_sec": 0.89
|
| 208 |
-
},
|
| 209 |
-
"70": {
|
| 210 |
-
"digits": 70,
|
| 211 |
-
"n": 200,
|
| 212 |
-
"passed": 0,
|
| 213 |
-
"accuracy": 0.0,
|
| 214 |
-
"wilson_95_ci": [
|
| 215 |
-
0.0,
|
| 216 |
-
1.9
|
| 217 |
-
],
|
| 218 |
-
"elapsed_sec": 0.7
|
| 219 |
-
},
|
| 220 |
-
"80": {
|
| 221 |
-
"digits": 80,
|
| 222 |
-
"n": 200,
|
| 223 |
-
"passed": 0,
|
| 224 |
-
"accuracy": 0.0,
|
| 225 |
-
"wilson_95_ci": [
|
| 226 |
-
0.0,
|
| 227 |
-
1.9
|
| 228 |
-
],
|
| 229 |
-
"elapsed_sec": 0.84
|
| 230 |
-
},
|
| 231 |
-
"100": {
|
| 232 |
-
"digits": 100,
|
| 233 |
-
"n": 200,
|
| 234 |
-
"passed": 0,
|
| 235 |
-
"accuracy": 0.0,
|
| 236 |
-
"wilson_95_ci": [
|
| 237 |
-
0.0,
|
| 238 |
-
1.9
|
| 239 |
-
],
|
| 240 |
-
"elapsed_sec": 0.37
|
| 241 |
-
}
|
| 242 |
-
}
|
| 243 |
-
},
|
| 244 |
-
"1d_shift512_15d_1024": {
|
| 245 |
-
"meta": {
|
| 246 |
-
"id": "1d_shift512_15d_1024",
|
| 247 |
-
"label": "1D-RoPE + Random Offset (1-15d, T=1024, Shift=512)",
|
| 248 |
-
"pos_mode": "1d_shift",
|
| 249 |
-
"k_shift": 512,
|
| 250 |
-
"final_loss": 0.09582,
|
| 251 |
-
"train_sec": 237.03
|
| 252 |
-
},
|
| 253 |
-
"pooled_seen_8_15d": {
|
| 254 |
-
"passed": 0,
|
| 255 |
-
"n": 600,
|
| 256 |
-
"accuracy": 0.0,
|
| 257 |
-
"wilson_95_ci": [
|
| 258 |
-
0.0,
|
| 259 |
-
0.6
|
| 260 |
-
]
|
| 261 |
-
},
|
| 262 |
-
"pooled_unseen_16_40d": {
|
| 263 |
-
"passed": 0,
|
| 264 |
-
"n": 1200,
|
| 265 |
-
"accuracy": 0.0,
|
| 266 |
-
"wilson_95_ci": [
|
| 267 |
-
0.0,
|
| 268 |
-
0.3
|
| 269 |
-
]
|
| 270 |
-
},
|
| 271 |
-
"pooled_unseen_16_50d": {
|
| 272 |
-
"passed": 0,
|
| 273 |
-
"n": 1600,
|
| 274 |
-
"accuracy": 0.0,
|
| 275 |
-
"wilson_95_ci": [
|
| 276 |
-
0.0,
|
| 277 |
-
0.2
|
| 278 |
-
]
|
| 279 |
-
},
|
| 280 |
-
"per_length": {
|
| 281 |
-
"8": {
|
| 282 |
-
"digits": 8,
|
| 283 |
-
"n": 200,
|
| 284 |
-
"passed": 0,
|
| 285 |
-
"accuracy": 0.0,
|
| 286 |
-
"wilson_95_ci": [
|
| 287 |
-
0.0,
|
| 288 |
-
1.9
|
| 289 |
-
],
|
| 290 |
-
"elapsed_sec": 0.46
|
| 291 |
-
},
|
| 292 |
-
"12": {
|
| 293 |
-
"digits": 12,
|
| 294 |
-
"n": 200,
|
| 295 |
-
"passed": 0,
|
| 296 |
-
"accuracy": 0.0,
|
| 297 |
-
"wilson_95_ci": [
|
| 298 |
-
0.0,
|
| 299 |
-
1.9
|
| 300 |
-
],
|
| 301 |
-
"elapsed_sec": 0.84
|
| 302 |
-
},
|
| 303 |
-
"15": {
|
| 304 |
-
"digits": 15,
|
| 305 |
-
"n": 200,
|
| 306 |
-
"passed": 0,
|
| 307 |
-
"accuracy": 0.0,
|
| 308 |
-
"wilson_95_ci": [
|
| 309 |
-
0.0,
|
| 310 |
-
1.9
|
| 311 |
-
],
|
| 312 |
-
"elapsed_sec": 1.19
|
| 313 |
-
},
|
| 314 |
-
"16": {
|
| 315 |
-
"digits": 16,
|
| 316 |
-
"n": 200,
|
| 317 |
-
"passed": 0,
|
| 318 |
-
"accuracy": 0.0,
|
| 319 |
-
"wilson_95_ci": [
|
| 320 |
-
0.0,
|
| 321 |
-
1.9
|
| 322 |
-
],
|
| 323 |
-
"elapsed_sec": 1.17
|
| 324 |
-
},
|
| 325 |
-
"20": {
|
| 326 |
-
"digits": 20,
|
| 327 |
-
"n": 200,
|
| 328 |
-
"passed": 0,
|
| 329 |
-
"accuracy": 0.0,
|
| 330 |
-
"wilson_95_ci": [
|
| 331 |
-
0.0,
|
| 332 |
-
1.9
|
| 333 |
-
],
|
| 334 |
-
"elapsed_sec": 1.66
|
| 335 |
-
},
|
| 336 |
-
"25": {
|
| 337 |
-
"digits": 25,
|
| 338 |
-
"n": 200,
|
| 339 |
-
"passed": 0,
|
| 340 |
-
"accuracy": 0.0,
|
| 341 |
-
"wilson_95_ci": [
|
| 342 |
-
0.0,
|
| 343 |
-
1.9
|
| 344 |
-
],
|
| 345 |
-
"elapsed_sec": 2.38
|
| 346 |
-
},
|
| 347 |
-
"30": {
|
| 348 |
-
"digits": 30,
|
| 349 |
-
"n": 200,
|
| 350 |
-
"passed": 0,
|
| 351 |
-
"accuracy": 0.0,
|
| 352 |
-
"wilson_95_ci": [
|
| 353 |
-
0.0,
|
| 354 |
-
1.9
|
| 355 |
-
],
|
| 356 |
-
"elapsed_sec": 3.0
|
| 357 |
-
},
|
| 358 |
-
"35": {
|
| 359 |
-
"digits": 35,
|
| 360 |
-
"n": 200,
|
| 361 |
-
"passed": 0,
|
| 362 |
-
"accuracy": 0.0,
|
| 363 |
-
"wilson_95_ci": [
|
| 364 |
-
0.0,
|
| 365 |
-
1.9
|
| 366 |
-
],
|
| 367 |
-
"elapsed_sec": 3.87
|
| 368 |
-
},
|
| 369 |
-
"40": {
|
| 370 |
-
"digits": 40,
|
| 371 |
-
"n": 200,
|
| 372 |
-
"passed": 0,
|
| 373 |
-
"accuracy": 0.0,
|
| 374 |
-
"wilson_95_ci": [
|
| 375 |
-
0.0,
|
| 376 |
-
1.9
|
| 377 |
-
],
|
| 378 |
-
"elapsed_sec": 4.56
|
| 379 |
-
},
|
| 380 |
-
"45": {
|
| 381 |
-
"digits": 45,
|
| 382 |
-
"n": 200,
|
| 383 |
-
"passed": 0,
|
| 384 |
-
"accuracy": 0.0,
|
| 385 |
-
"wilson_95_ci": [
|
| 386 |
-
0.0,
|
| 387 |
-
1.9
|
| 388 |
-
],
|
| 389 |
-
"elapsed_sec": 5.78
|
| 390 |
-
},
|
| 391 |
-
"50": {
|
| 392 |
-
"digits": 50,
|
| 393 |
-
"n": 200,
|
| 394 |
-
"passed": 0,
|
| 395 |
-
"accuracy": 0.0,
|
| 396 |
-
"wilson_95_ci": [
|
| 397 |
-
0.0,
|
| 398 |
-
1.9
|
| 399 |
-
],
|
| 400 |
-
"elapsed_sec": 6.28
|
| 401 |
-
},
|
| 402 |
-
"55": {
|
| 403 |
-
"digits": 55,
|
| 404 |
-
"n": 200,
|
| 405 |
-
"passed": 0,
|
| 406 |
-
"accuracy": 0.0,
|
| 407 |
-
"wilson_95_ci": [
|
| 408 |
-
0.0,
|
| 409 |
-
1.9
|
| 410 |
-
],
|
| 411 |
-
"elapsed_sec": 8.48
|
| 412 |
-
},
|
| 413 |
-
"60": {
|
| 414 |
-
"digits": 60,
|
| 415 |
-
"n": 200,
|
| 416 |
-
"passed": 0,
|
| 417 |
-
"accuracy": 0.0,
|
| 418 |
-
"wilson_95_ci": [
|
| 419 |
-
0.0,
|
| 420 |
-
1.9
|
| 421 |
-
],
|
| 422 |
-
"elapsed_sec": 9.89
|
| 423 |
-
},
|
| 424 |
-
"70": {
|
| 425 |
-
"digits": 70,
|
| 426 |
-
"n": 200,
|
| 427 |
-
"passed": 0,
|
| 428 |
-
"accuracy": 0.0,
|
| 429 |
-
"wilson_95_ci": [
|
| 430 |
-
0.0,
|
| 431 |
-
1.9
|
| 432 |
-
],
|
| 433 |
-
"elapsed_sec": 14.4
|
| 434 |
-
},
|
| 435 |
-
"80": {
|
| 436 |
-
"digits": 80,
|
| 437 |
-
"n": 200,
|
| 438 |
-
"passed": 0,
|
| 439 |
-
"accuracy": 0.0,
|
| 440 |
-
"wilson_95_ci": [
|
| 441 |
-
0.0,
|
| 442 |
-
1.9
|
| 443 |
-
],
|
| 444 |
-
"elapsed_sec": 16.89
|
| 445 |
-
},
|
| 446 |
-
"100": {
|
| 447 |
-
"digits": 100,
|
| 448 |
-
"n": 200,
|
| 449 |
-
"passed": 0,
|
| 450 |
-
"accuracy": 0.0,
|
| 451 |
-
"wilson_95_ci": [
|
| 452 |
-
0.0,
|
| 453 |
-
1.9
|
| 454 |
-
],
|
| 455 |
-
"elapsed_sec": 26.57
|
| 456 |
-
}
|
| 457 |
-
}
|
| 458 |
-
},
|
| 459 |
-
"1d_ruoss_15d_1024": {
|
| 460 |
-
"meta": {
|
| 461 |
-
"id": "1d_ruoss_15d_1024",
|
| 462 |
-
"label": "1D-RoPE + Randomized Positions (Ruoss et al. 2023, 1-15d, M=1024)",
|
| 463 |
-
"pos_mode": "1d_ruoss",
|
| 464 |
-
"k_shift": 0,
|
| 465 |
-
"final_loss": 0.23179,
|
| 466 |
-
"train_sec": 271.63
|
| 467 |
-
},
|
| 468 |
-
"pooled_seen_8_15d": {
|
| 469 |
-
"passed": 0,
|
| 470 |
-
"n": 600,
|
| 471 |
-
"accuracy": 0.0,
|
| 472 |
-
"wilson_95_ci": [
|
| 473 |
-
0.0,
|
| 474 |
-
0.6
|
| 475 |
-
]
|
| 476 |
-
},
|
| 477 |
-
"pooled_unseen_16_40d": {
|
| 478 |
-
"passed": 0,
|
| 479 |
-
"n": 1200,
|
| 480 |
-
"accuracy": 0.0,
|
| 481 |
-
"wilson_95_ci": [
|
| 482 |
-
0.0,
|
| 483 |
-
0.3
|
| 484 |
-
]
|
| 485 |
-
},
|
| 486 |
-
"pooled_unseen_16_50d": {
|
| 487 |
-
"passed": 0,
|
| 488 |
-
"n": 1600,
|
| 489 |
-
"accuracy": 0.0,
|
| 490 |
-
"wilson_95_ci": [
|
| 491 |
-
0.0,
|
| 492 |
-
0.2
|
| 493 |
-
]
|
| 494 |
-
},
|
| 495 |
-
"per_length": {
|
| 496 |
-
"8": {
|
| 497 |
-
"digits": 8,
|
| 498 |
-
"n": 200,
|
| 499 |
-
"passed": 0,
|
| 500 |
-
"accuracy": 0.0,
|
| 501 |
-
"wilson_95_ci": [
|
| 502 |
-
0.0,
|
| 503 |
-
1.9
|
| 504 |
-
],
|
| 505 |
-
"elapsed_sec": 0.58
|
| 506 |
-
},
|
| 507 |
-
"12": {
|
| 508 |
-
"digits": 12,
|
| 509 |
-
"n": 200,
|
| 510 |
-
"passed": 0,
|
| 511 |
-
"accuracy": 0.0,
|
| 512 |
-
"wilson_95_ci": [
|
| 513 |
-
0.0,
|
| 514 |
-
1.9
|
| 515 |
-
],
|
| 516 |
-
"elapsed_sec": 0.81
|
| 517 |
-
},
|
| 518 |
-
"15": {
|
| 519 |
-
"digits": 15,
|
| 520 |
-
"n": 200,
|
| 521 |
-
"passed": 0,
|
| 522 |
-
"accuracy": 0.0,
|
| 523 |
-
"wilson_95_ci": [
|
| 524 |
-
0.0,
|
| 525 |
-
1.9
|
| 526 |
-
],
|
| 527 |
-
"elapsed_sec": 1.11
|
| 528 |
-
},
|
| 529 |
-
"16": {
|
| 530 |
-
"digits": 16,
|
| 531 |
-
"n": 200,
|
| 532 |
-
"passed": 0,
|
| 533 |
-
"accuracy": 0.0,
|
| 534 |
-
"wilson_95_ci": [
|
| 535 |
-
0.0,
|
| 536 |
-
1.9
|
| 537 |
-
],
|
| 538 |
-
"elapsed_sec": 1.18
|
| 539 |
-
},
|
| 540 |
-
"20": {
|
| 541 |
-
"digits": 20,
|
| 542 |
-
"n": 200,
|
| 543 |
-
"passed": 0,
|
| 544 |
-
"accuracy": 0.0,
|
| 545 |
-
"wilson_95_ci": [
|
| 546 |
-
0.0,
|
| 547 |
-
1.9
|
| 548 |
-
],
|
| 549 |
-
"elapsed_sec": 1.72
|
| 550 |
-
},
|
| 551 |
-
"25": {
|
| 552 |
-
"digits": 25,
|
| 553 |
-
"n": 200,
|
| 554 |
-
"passed": 0,
|
| 555 |
-
"accuracy": 0.0,
|
| 556 |
-
"wilson_95_ci": [
|
| 557 |
-
0.0,
|
| 558 |
-
1.9
|
| 559 |
-
],
|
| 560 |
-
"elapsed_sec": 2.32
|
| 561 |
-
},
|
| 562 |
-
"30": {
|
| 563 |
-
"digits": 30,
|
| 564 |
-
"n": 200,
|
| 565 |
-
"passed": 0,
|
| 566 |
-
"accuracy": 0.0,
|
| 567 |
-
"wilson_95_ci": [
|
| 568 |
-
0.0,
|
| 569 |
-
1.9
|
| 570 |
-
],
|
| 571 |
-
"elapsed_sec": 3.05
|
| 572 |
-
},
|
| 573 |
-
"35": {
|
| 574 |
-
"digits": 35,
|
| 575 |
-
"n": 200,
|
| 576 |
-
"passed": 0,
|
| 577 |
-
"accuracy": 0.0,
|
| 578 |
-
"wilson_95_ci": [
|
| 579 |
-
0.0,
|
| 580 |
-
1.9
|
| 581 |
-
],
|
| 582 |
-
"elapsed_sec": 3.78
|
| 583 |
-
},
|
| 584 |
-
"40": {
|
| 585 |
-
"digits": 40,
|
| 586 |
-
"n": 200,
|
| 587 |
-
"passed": 0,
|
| 588 |
-
"accuracy": 0.0,
|
| 589 |
-
"wilson_95_ci": [
|
| 590 |
-
0.0,
|
| 591 |
-
1.9
|
| 592 |
-
],
|
| 593 |
-
"elapsed_sec": 4.09
|
| 594 |
-
},
|
| 595 |
-
"45": {
|
| 596 |
-
"digits": 45,
|
| 597 |
-
"n": 200,
|
| 598 |
-
"passed": 0,
|
| 599 |
-
"accuracy": 0.0,
|
| 600 |
-
"wilson_95_ci": [
|
| 601 |
-
0.0,
|
| 602 |
-
1.9
|
| 603 |
-
],
|
| 604 |
-
"elapsed_sec": 4.98
|
| 605 |
-
},
|
| 606 |
-
"50": {
|
| 607 |
-
"digits": 50,
|
| 608 |
-
"n": 200,
|
| 609 |
-
"passed": 0,
|
| 610 |
-
"accuracy": 0.0,
|
| 611 |
-
"wilson_95_ci": [
|
| 612 |
-
0.0,
|
| 613 |
-
1.9
|
| 614 |
-
],
|
| 615 |
-
"elapsed_sec": 7.21
|
| 616 |
-
},
|
| 617 |
-
"55": {
|
| 618 |
-
"digits": 55,
|
| 619 |
-
"n": 200,
|
| 620 |
-
"passed": 0,
|
| 621 |
-
"accuracy": 0.0,
|
| 622 |
-
"wilson_95_ci": [
|
| 623 |
-
0.0,
|
| 624 |
-
1.9
|
| 625 |
-
],
|
| 626 |
-
"elapsed_sec": 7.39
|
| 627 |
-
},
|
| 628 |
-
"60": {
|
| 629 |
-
"digits": 60,
|
| 630 |
-
"n": 200,
|
| 631 |
-
"passed": 0,
|
| 632 |
-
"accuracy": 0.0,
|
| 633 |
-
"wilson_95_ci": [
|
| 634 |
-
0.0,
|
| 635 |
-
1.9
|
| 636 |
-
],
|
| 637 |
-
"elapsed_sec": 10.29
|
| 638 |
-
},
|
| 639 |
-
"70": {
|
| 640 |
-
"digits": 70,
|
| 641 |
-
"n": 200,
|
| 642 |
-
"passed": 0,
|
| 643 |
-
"accuracy": 0.0,
|
| 644 |
-
"wilson_95_ci": [
|
| 645 |
-
0.0,
|
| 646 |
-
1.9
|
| 647 |
-
],
|
| 648 |
-
"elapsed_sec": 14.0
|
| 649 |
-
},
|
| 650 |
-
"80": {
|
| 651 |
-
"digits": 80,
|
| 652 |
-
"n": 200,
|
| 653 |
-
"passed": 0,
|
| 654 |
-
"accuracy": 0.0,
|
| 655 |
-
"wilson_95_ci": [
|
| 656 |
-
0.0,
|
| 657 |
-
1.9
|
| 658 |
-
],
|
| 659 |
-
"elapsed_sec": 16.05
|
| 660 |
-
},
|
| 661 |
-
"100": {
|
| 662 |
-
"digits": 100,
|
| 663 |
-
"n": 200,
|
| 664 |
-
"passed": 0,
|
| 665 |
-
"accuracy": 0.0,
|
| 666 |
-
"wilson_95_ci": [
|
| 667 |
-
0.0,
|
| 668 |
-
1.9
|
| 669 |
-
],
|
| 670 |
-
"elapsed_sec": 24.49
|
| 671 |
-
}
|
| 672 |
-
}
|
| 673 |
-
},
|
| 674 |
-
"2d_shift32_15d_1024": {
|
| 675 |
-
"meta": {
|
| 676 |
-
"id": "2d_shift32_15d_1024",
|
| 677 |
-
"label": "Decoupled 2D-RoPE + Shift (Ours, 1-15d, T=1024, k=32)",
|
| 678 |
-
"pos_mode": "2d_rope",
|
| 679 |
-
"k_shift": 32,
|
| 680 |
-
"final_loss": 0.00597,
|
| 681 |
-
"train_sec": 352.24
|
| 682 |
-
},
|
| 683 |
-
"pooled_seen_8_15d": {
|
| 684 |
-
"passed": 417,
|
| 685 |
-
"n": 600,
|
| 686 |
-
"accuracy": 69.5,
|
| 687 |
-
"wilson_95_ci": [
|
| 688 |
-
65.7,
|
| 689 |
-
73.1
|
| 690 |
-
]
|
| 691 |
-
},
|
| 692 |
-
"pooled_unseen_16_40d": {
|
| 693 |
-
"passed": 977,
|
| 694 |
-
"n": 1200,
|
| 695 |
-
"accuracy": 81.42,
|
| 696 |
-
"wilson_95_ci": [
|
| 697 |
-
79.1,
|
| 698 |
-
83.5
|
| 699 |
-
]
|
| 700 |
-
},
|
| 701 |
-
"pooled_unseen_16_50d": {
|
| 702 |
-
"passed": 1037,
|
| 703 |
-
"n": 1600,
|
| 704 |
-
"accuracy": 64.81,
|
| 705 |
-
"wilson_95_ci": [
|
| 706 |
-
62.4,
|
| 707 |
-
67.1
|
| 708 |
-
]
|
| 709 |
-
},
|
| 710 |
-
"per_length": {
|
| 711 |
-
"8": {
|
| 712 |
-
"digits": 8,
|
| 713 |
-
"n": 200,
|
| 714 |
-
"passed": 108,
|
| 715 |
-
"accuracy": 54.0,
|
| 716 |
-
"wilson_95_ci": [
|
| 717 |
-
47.1,
|
| 718 |
-
60.8
|
| 719 |
-
],
|
| 720 |
-
"elapsed_sec": 1.99
|
| 721 |
-
},
|
| 722 |
-
"12": {
|
| 723 |
-
"digits": 12,
|
| 724 |
-
"n": 200,
|
| 725 |
-
"passed": 137,
|
| 726 |
-
"accuracy": 68.5,
|
| 727 |
-
"wilson_95_ci": [
|
| 728 |
-
61.8,
|
| 729 |
-
74.5
|
| 730 |
-
],
|
| 731 |
-
"elapsed_sec": 1.53
|
| 732 |
-
},
|
| 733 |
-
"15": {
|
| 734 |
-
"digits": 15,
|
| 735 |
-
"n": 200,
|
| 736 |
-
"passed": 172,
|
| 737 |
-
"accuracy": 86.0,
|
| 738 |
-
"wilson_95_ci": [
|
| 739 |
-
80.5,
|
| 740 |
-
90.1
|
| 741 |
-
],
|
| 742 |
-
"elapsed_sec": 2.02
|
| 743 |
-
},
|
| 744 |
-
"16": {
|
| 745 |
-
"digits": 16,
|
| 746 |
-
"n": 200,
|
| 747 |
-
"passed": 184,
|
| 748 |
-
"accuracy": 92.0,
|
| 749 |
-
"wilson_95_ci": [
|
| 750 |
-
87.4,
|
| 751 |
-
95.0
|
| 752 |
-
],
|
| 753 |
-
"elapsed_sec": 3.12
|
| 754 |
-
},
|
| 755 |
-
"20": {
|
| 756 |
-
"digits": 20,
|
| 757 |
-
"n": 200,
|
| 758 |
-
"passed": 180,
|
| 759 |
-
"accuracy": 90.0,
|
| 760 |
-
"wilson_95_ci": [
|
| 761 |
-
85.1,
|
| 762 |
-
93.4
|
| 763 |
-
],
|
| 764 |
-
"elapsed_sec": 3.89
|
| 765 |
-
},
|
| 766 |
-
"25": {
|
| 767 |
-
"digits": 25,
|
| 768 |
-
"n": 200,
|
| 769 |
-
"passed": 184,
|
| 770 |
-
"accuracy": 92.0,
|
| 771 |
-
"wilson_95_ci": [
|
| 772 |
-
87.4,
|
| 773 |
-
95.0
|
| 774 |
-
],
|
| 775 |
-
"elapsed_sec": 3.71
|
| 776 |
-
},
|
| 777 |
-
"30": {
|
| 778 |
-
"digits": 30,
|
| 779 |
-
"n": 200,
|
| 780 |
-
"passed": 170,
|
| 781 |
-
"accuracy": 85.0,
|
| 782 |
-
"wilson_95_ci": [
|
| 783 |
-
79.4,
|
| 784 |
-
89.3
|
| 785 |
-
],
|
| 786 |
-
"elapsed_sec": 5.5
|
| 787 |
-
},
|
| 788 |
-
"35": {
|
| 789 |
-
"digits": 35,
|
| 790 |
-
"n": 200,
|
| 791 |
-
"passed": 156,
|
| 792 |
-
"accuracy": 78.0,
|
| 793 |
-
"wilson_95_ci": [
|
| 794 |
-
71.8,
|
| 795 |
-
83.2
|
| 796 |
-
],
|
| 797 |
-
"elapsed_sec": 8.06
|
| 798 |
-
},
|
| 799 |
-
"40": {
|
| 800 |
-
"digits": 40,
|
| 801 |
-
"n": 200,
|
| 802 |
-
"passed": 103,
|
| 803 |
-
"accuracy": 51.5,
|
| 804 |
-
"wilson_95_ci": [
|
| 805 |
-
44.6,
|
| 806 |
-
58.3
|
| 807 |
-
],
|
| 808 |
-
"elapsed_sec": 9.16
|
| 809 |
-
},
|
| 810 |
-
"45": {
|
| 811 |
-
"digits": 45,
|
| 812 |
-
"n": 200,
|
| 813 |
-
"passed": 60,
|
| 814 |
-
"accuracy": 30.0,
|
| 815 |
-
"wilson_95_ci": [
|
| 816 |
-
24.1,
|
| 817 |
-
36.7
|
| 818 |
-
],
|
| 819 |
-
"elapsed_sec": 11.37
|
| 820 |
-
},
|
| 821 |
-
"50": {
|
| 822 |
-
"digits": 50,
|
| 823 |
-
"n": 200,
|
| 824 |
-
"passed": 0,
|
| 825 |
-
"accuracy": 0.0,
|
| 826 |
-
"wilson_95_ci": [
|
| 827 |
-
0.0,
|
| 828 |
-
1.9
|
| 829 |
-
],
|
| 830 |
-
"elapsed_sec": 11.7
|
| 831 |
-
},
|
| 832 |
-
"55": {
|
| 833 |
-
"digits": 55,
|
| 834 |
-
"n": 200,
|
| 835 |
-
"passed": 0,
|
| 836 |
-
"accuracy": 0.0,
|
| 837 |
-
"wilson_95_ci": [
|
| 838 |
-
0.0,
|
| 839 |
-
1.9
|
| 840 |
-
],
|
| 841 |
-
"elapsed_sec": 12.22
|
| 842 |
-
},
|
| 843 |
-
"60": {
|
| 844 |
-
"digits": 60,
|
| 845 |
-
"n": 200,
|
| 846 |
-
"passed": 0,
|
| 847 |
-
"accuracy": 0.0,
|
| 848 |
-
"wilson_95_ci": [
|
| 849 |
-
0.0,
|
| 850 |
-
1.9
|
| 851 |
-
],
|
| 852 |
-
"elapsed_sec": 14.38
|
| 853 |
-
},
|
| 854 |
-
"70": {
|
| 855 |
-
"digits": 70,
|
| 856 |
-
"n": 200,
|
| 857 |
-
"passed": 0,
|
| 858 |
-
"accuracy": 0.0,
|
| 859 |
-
"wilson_95_ci": [
|
| 860 |
-
0.0,
|
| 861 |
-
1.9
|
| 862 |
-
],
|
| 863 |
-
"elapsed_sec": 23.93
|
| 864 |
-
},
|
| 865 |
-
"80": {
|
| 866 |
-
"digits": 80,
|
| 867 |
-
"n": 200,
|
| 868 |
-
"passed": 0,
|
| 869 |
-
"accuracy": 0.0,
|
| 870 |
-
"wilson_95_ci": [
|
| 871 |
-
0.0,
|
| 872 |
-
1.9
|
| 873 |
-
],
|
| 874 |
-
"elapsed_sec": 30.33
|
| 875 |
-
},
|
| 876 |
-
"100": {
|
| 877 |
-
"digits": 100,
|
| 878 |
-
"n": 200,
|
| 879 |
-
"passed": 0,
|
| 880 |
-
"accuracy": 0.0,
|
| 881 |
-
"wilson_95_ci": [
|
| 882 |
-
0.0,
|
| 883 |
-
1.9
|
| 884 |
-
],
|
| 885 |
-
"elapsed_sec": 35.26
|
| 886 |
-
}
|
| 887 |
-
}
|
| 888 |
-
}
|
| 889 |
-
}
|
| 890 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
cbt_2d_arithmetic_20_70d.json → evaluation/cbt_2d/addition_20-70_digits.json
RENAMED
|
File without changes
|
cbt_2d_arithmetic_4_16d.json → evaluation/cbt_2d/addition_4-16_digits.json
RENAMED
|
File without changes
|
cbt_2d_ood.json → evaluation/cbt_2d/tools_out_of_distribution.json
RENAMED
|
File without changes
|
cbt_2d_phrasing.json → evaluation/cbt_2d/tools_rephrased.json
RENAMED
|
File without changes
|
cbt_2d_unseen_copy.json → evaluation/cbt_2d/tools_unseen_entities.json
RENAMED
|
File without changes
|
cbt1d_v5_arithmetic.json → evaluation/twin_1d/addition_20-70_digits.json
RENAMED
|
File without changes
|
cbt_1d_twin_arithmetic_4_16d.json → evaluation/twin_1d/addition_4-16_digits.json
RENAMED
|
File without changes
|
champion_v3_ood.json → evaluation/twin_1d/tools_out_of_distribution.json
RENAMED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"checkpoint": "
|
| 3 |
"max_train_operand_digits": 25,
|
| 4 |
"ood": {
|
| 5 |
"total": 18,
|
|
|
|
| 1 |
{
|
| 2 |
+
"checkpoint": "checkpoints/cbt/cbt_18m_1d_rope_R3_k45_sftv5_best.pt",
|
| 3 |
"max_train_operand_digits": 25,
|
| 4 |
"ood": {
|
| 5 |
"total": 18,
|
champion_v3_phrasing.json → evaluation/twin_1d/tools_rephrased.json
RENAMED
|
@@ -1,17 +1,17 @@
|
|
| 1 |
{
|
| 2 |
-
"checkpoint": "
|
| 3 |
"max_train_operand_digits": 25,
|
| 4 |
"phrasing": {
|
| 5 |
"total": 26,
|
| 6 |
-
"routing_passed":
|
| 7 |
"routing_ci": [
|
| 8 |
-
|
| 9 |
-
|
| 10 |
],
|
| 11 |
-
"exact_args_passed":
|
| 12 |
"exact_args_ci": [
|
| 13 |
-
|
| 14 |
-
|
| 15 |
]
|
| 16 |
}
|
| 17 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"checkpoint": "checkpoints/cbt/cbt_18m_1d_rope_R3_k45_sftv5_best.pt",
|
| 3 |
"max_train_operand_digits": 25,
|
| 4 |
"phrasing": {
|
| 5 |
"total": 26,
|
| 6 |
+
"routing_passed": 25,
|
| 7 |
"routing_ci": [
|
| 8 |
+
81.1,
|
| 9 |
+
99.3
|
| 10 |
],
|
| 11 |
+
"exact_args_passed": 25,
|
| 12 |
"exact_args_ci": [
|
| 13 |
+
81.1,
|
| 14 |
+
99.3
|
| 15 |
]
|
| 16 |
}
|
| 17 |
}
|
champion_v3_unseen_ood.json → evaluation/twin_1d/tools_unseen_entities.json
RENAMED
|
@@ -37,7 +37,7 @@
|
|
| 37 |
"prompt": "<USER>Who was Hypatia of Alexandria?</USER><THINK>",
|
| 38 |
"expected_tool": "search",
|
| 39 |
"expected_arg": "Hypatia of Alexandria",
|
| 40 |
-
"raw_response": "The user is
|
| 41 |
"parsed_call": "search(query=\"Hypatia of Alexandria\")",
|
| 42 |
"route_ok": true,
|
| 43 |
"exact_copy_ok": true
|
|
@@ -48,7 +48,7 @@
|
|
| 48 |
"prompt": "<USER>Who was Ludwig van Beethoven?</USER><THINK>",
|
| 49 |
"expected_tool": "search",
|
| 50 |
"expected_arg": "Ludwig van Beethoven",
|
| 51 |
-
"raw_response": "The user is
|
| 52 |
"parsed_call": "search(query=\"Ludwig van Beethoven\")",
|
| 53 |
"route_ok": true,
|
| 54 |
"exact_copy_ok": true
|
|
@@ -59,7 +59,7 @@
|
|
| 59 |
"prompt": "<USER>Who was Johannes Vermeer?</USER><THINK>",
|
| 60 |
"expected_tool": "search",
|
| 61 |
"expected_arg": "Johannes Vermeer",
|
| 62 |
-
"raw_response": "The user is
|
| 63 |
"parsed_call": "search(query=\"Johannes Vermeer\")",
|
| 64 |
"route_ok": true,
|
| 65 |
"exact_copy_ok": true
|
|
@@ -70,7 +70,7 @@
|
|
| 70 |
"prompt": "<USER>Who is Ada Yonath?</USER><THINK>",
|
| 71 |
"expected_tool": "search",
|
| 72 |
"expected_arg": "Ada Yonath",
|
| 73 |
-
"raw_response": "The user is
|
| 74 |
"parsed_call": "search(query=\"Ada Yonath\")",
|
| 75 |
"route_ok": true,
|
| 76 |
"exact_copy_ok": true
|
|
@@ -81,7 +81,7 @@
|
|
| 81 |
"prompt": "<USER>Who was Srinivasa Ramanujan?</USER><THINK>",
|
| 82 |
"expected_tool": "search",
|
| 83 |
"expected_arg": "Srinivasa Ramanujan",
|
| 84 |
-
"raw_response": "The user is
|
| 85 |
"parsed_call": "search(query=\"Srinivasa Ramanujan\")",
|
| 86 |
"route_ok": true,
|
| 87 |
"exact_copy_ok": true
|
|
@@ -114,7 +114,7 @@
|
|
| 114 |
"prompt": "<USER>Weather on Antananarivo</USER><THINK>",
|
| 115 |
"expected_tool": "get_weather",
|
| 116 |
"expected_arg": "Antananarivo",
|
| 117 |
-
"raw_response": "The user
|
| 118 |
"parsed_call": "get_weather(city=\"Antananarivo\")",
|
| 119 |
"route_ok": true,
|
| 120 |
"exact_copy_ok": true
|
|
@@ -202,7 +202,7 @@
|
|
| 202 |
"prompt": "<USER>Explain bose einstein condensate</USER><THINK>",
|
| 203 |
"expected_tool": "search",
|
| 204 |
"expected_arg": "bose einstein condensate",
|
| 205 |
-
"raw_response": "The user is
|
| 206 |
"parsed_call": "search(query=\"bose einstein condensate\")",
|
| 207 |
"route_ok": true,
|
| 208 |
"exact_copy_ok": true
|
|
@@ -235,7 +235,7 @@
|
|
| 235 |
"prompt": "<USER>Define baklava</USER><THINK>",
|
| 236 |
"expected_tool": "search",
|
| 237 |
"expected_arg": "baklava",
|
| 238 |
-
"raw_response": "The user is
|
| 239 |
"parsed_call": "search(query=\"baklava\")",
|
| 240 |
"route_ok": true,
|
| 241 |
"exact_copy_ok": true
|
|
@@ -246,7 +246,7 @@
|
|
| 246 |
"prompt": "<USER>What is shakshuka?</USER><THINK>",
|
| 247 |
"expected_tool": "search",
|
| 248 |
"expected_arg": "shakshuka",
|
| 249 |
-
"raw_response": "The user is
|
| 250 |
"parsed_call": "search(query=\"shakshuka\")",
|
| 251 |
"route_ok": true,
|
| 252 |
"exact_copy_ok": true
|
|
@@ -301,7 +301,7 @@
|
|
| 301 |
"prompt": "<USER>Define Zyntral Krelvok</USER><THINK>",
|
| 302 |
"expected_tool": "search",
|
| 303 |
"expected_arg": "Zyntral Krelvok",
|
| 304 |
-
"raw_response": "The user is
|
| 305 |
"parsed_call": "search(query=\"Zyntral Krelvok\")",
|
| 306 |
"route_ok": true,
|
| 307 |
"exact_copy_ok": true
|
|
|
|
| 37 |
"prompt": "<USER>Who was Hypatia of Alexandria?</USER><THINK>",
|
| 38 |
"expected_tool": "search",
|
| 39 |
"expected_arg": "Hypatia of Alexandria",
|
| 40 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Hypatia of Alexandria\")</CALL>",
|
| 41 |
"parsed_call": "search(query=\"Hypatia of Alexandria\")",
|
| 42 |
"route_ok": true,
|
| 43 |
"exact_copy_ok": true
|
|
|
|
| 48 |
"prompt": "<USER>Who was Ludwig van Beethoven?</USER><THINK>",
|
| 49 |
"expected_tool": "search",
|
| 50 |
"expected_arg": "Ludwig van Beethoven",
|
| 51 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Ludwig van Beethoven\")</CALL>",
|
| 52 |
"parsed_call": "search(query=\"Ludwig van Beethoven\")",
|
| 53 |
"route_ok": true,
|
| 54 |
"exact_copy_ok": true
|
|
|
|
| 59 |
"prompt": "<USER>Who was Johannes Vermeer?</USER><THINK>",
|
| 60 |
"expected_tool": "search",
|
| 61 |
"expected_arg": "Johannes Vermeer",
|
| 62 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Johannes Vermeer\")</CALL>",
|
| 63 |
"parsed_call": "search(query=\"Johannes Vermeer\")",
|
| 64 |
"route_ok": true,
|
| 65 |
"exact_copy_ok": true
|
|
|
|
| 70 |
"prompt": "<USER>Who is Ada Yonath?</USER><THINK>",
|
| 71 |
"expected_tool": "search",
|
| 72 |
"expected_arg": "Ada Yonath",
|
| 73 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Ada Yonath\")</CALL>",
|
| 74 |
"parsed_call": "search(query=\"Ada Yonath\")",
|
| 75 |
"route_ok": true,
|
| 76 |
"exact_copy_ok": true
|
|
|
|
| 81 |
"prompt": "<USER>Who was Srinivasa Ramanujan?</USER><THINK>",
|
| 82 |
"expected_tool": "search",
|
| 83 |
"expected_arg": "Srinivasa Ramanujan",
|
| 84 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Srinivasa Ramanujan\")</CALL>",
|
| 85 |
"parsed_call": "search(query=\"Srinivasa Ramanujan\")",
|
| 86 |
"route_ok": true,
|
| 87 |
"exact_copy_ok": true
|
|
|
|
| 114 |
"prompt": "<USER>Weather on Antananarivo</USER><THINK>",
|
| 115 |
"expected_tool": "get_weather",
|
| 116 |
"expected_arg": "Antananarivo",
|
| 117 |
+
"raw_response": "The user requests current weather information. I will query the get_weather tool.</THINK><CALL>get_weather(city=\"Antananarivo\")</CALL>",
|
| 118 |
"parsed_call": "get_weather(city=\"Antananarivo\")",
|
| 119 |
"route_ok": true,
|
| 120 |
"exact_copy_ok": true
|
|
|
|
| 202 |
"prompt": "<USER>Explain bose einstein condensate</USER><THINK>",
|
| 203 |
"expected_tool": "search",
|
| 204 |
"expected_arg": "bose einstein condensate",
|
| 205 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"bose einstein condensate\")</CALL>",
|
| 206 |
"parsed_call": "search(query=\"bose einstein condensate\")",
|
| 207 |
"route_ok": true,
|
| 208 |
"exact_copy_ok": true
|
|
|
|
| 235 |
"prompt": "<USER>Define baklava</USER><THINK>",
|
| 236 |
"expected_tool": "search",
|
| 237 |
"expected_arg": "baklava",
|
| 238 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"baklava\")</CALL>",
|
| 239 |
"parsed_call": "search(query=\"baklava\")",
|
| 240 |
"route_ok": true,
|
| 241 |
"exact_copy_ok": true
|
|
|
|
| 246 |
"prompt": "<USER>What is shakshuka?</USER><THINK>",
|
| 247 |
"expected_tool": "search",
|
| 248 |
"expected_arg": "shakshuka",
|
| 249 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"shakshuka\")</CALL>",
|
| 250 |
"parsed_call": "search(query=\"shakshuka\")",
|
| 251 |
"route_ok": true,
|
| 252 |
"exact_copy_ok": true
|
|
|
|
| 301 |
"prompt": "<USER>Define Zyntral Krelvok</USER><THINK>",
|
| 302 |
"expected_tool": "search",
|
| 303 |
"expected_arg": "Zyntral Krelvok",
|
| 304 |
+
"raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Zyntral Krelvok\")</CALL>",
|
| 305 |
"parsed_call": "search(query=\"Zyntral Krelvok\")",
|
| 306 |
"route_ok": true,
|
| 307 |
"exact_copy_ok": true
|
interactive.py
DELETED
|
@@ -1,329 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""
|
| 3 |
-
Interactive CLI Runner for ScaledGPT (~18.95M Parameter Recurrent 2D-RoPE Transformer).
|
| 4 |
-
Supports live interactive inference across:
|
| 5 |
-
1. High-digit column scratchpad arithmetic (up to 50 digits).
|
| 6 |
-
2. Expanded tool execution (Single-tool, Chaining, Nested JSON schemas, Error handling).
|
| 7 |
-
3. Multi-turn interactive dialogues.
|
| 8 |
-
"""
|
| 9 |
-
|
| 10 |
-
import os
|
| 11 |
-
import sys
|
| 12 |
-
import re
|
| 13 |
-
import json
|
| 14 |
-
import time
|
| 15 |
-
import argparse
|
| 16 |
-
from typing import Tuple, Optional, Dict, Any
|
| 17 |
-
|
| 18 |
-
import torch
|
| 19 |
-
|
| 20 |
-
ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
| 21 |
-
if ROOT_DIR not in sys.path:
|
| 22 |
-
sys.path.insert(0, ROOT_DIR)
|
| 23 |
-
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
|
| 24 |
-
if THIS_DIR not in sys.path:
|
| 25 |
-
sys.path.insert(0, THIS_DIR)
|
| 26 |
-
|
| 27 |
-
if hasattr(sys.stdout, "reconfigure"):
|
| 28 |
-
sys.stdout.reconfigure(line_buffering=True)
|
| 29 |
-
|
| 30 |
-
try:
|
| 31 |
-
from champion_model.scaled_t4.model import ScaledGPT, ScaledGPTConfig
|
| 32 |
-
from champion_model.scaled_t4.tokenizer import ScaledTokenizer
|
| 33 |
-
from champion_model.scaled_t4.dataset import extract_scratchpad_answer
|
| 34 |
-
except ImportError:
|
| 35 |
-
from model import ScaledGPT, ScaledGPTConfig
|
| 36 |
-
from tokenizer import ScaledTokenizer
|
| 37 |
-
from dataset import extract_scratchpad_answer
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
def mock_execute_tool(call_text: str) -> str:
|
| 41 |
-
"""Executes or mocks tool calls parsed from <CALL>...</CALL>."""
|
| 42 |
-
# 1. calculator(expr="...")
|
| 43 |
-
calc_m = re.match(r'calculator\(expr="([^"]+)"\)', call_text)
|
| 44 |
-
if calc_m:
|
| 45 |
-
expr = calc_m.group(1)
|
| 46 |
-
try:
|
| 47 |
-
allowed = re.match(r"^[\d\+\-\*\/\(\)\s\.]+$", expr)
|
| 48 |
-
if allowed:
|
| 49 |
-
val = eval(expr)
|
| 50 |
-
return json.dumps({"result": val})
|
| 51 |
-
except ZeroDivisionError:
|
| 52 |
-
return json.dumps({"error": "ZeroDivisionError: division by zero"})
|
| 53 |
-
except Exception as e:
|
| 54 |
-
return json.dumps({"error": str(e)})
|
| 55 |
-
|
| 56 |
-
# 2. get_weather(city="...")
|
| 57 |
-
w_m = re.match(r'get_weather\(city="([^"]+)"\)', call_text)
|
| 58 |
-
if w_m:
|
| 59 |
-
city = w_m.group(1)
|
| 60 |
-
if city.lower() in ["atlantis", "narnia"]:
|
| 61 |
-
return json.dumps({"status": "error", "code": 404, "message": f"City '{city}' not found"})
|
| 62 |
-
return json.dumps({"temp": 22, "condition": "Sunny", "city": city})
|
| 63 |
-
|
| 64 |
-
# 3. get_time(timezone="...")
|
| 65 |
-
t_m = re.match(r'get_time\(timezone="([^"]+)"\)', call_text)
|
| 66 |
-
if t_m:
|
| 67 |
-
tz = t_m.group(1)
|
| 68 |
-
return json.dumps({"time": "14:30", "timezone": tz})
|
| 69 |
-
|
| 70 |
-
# 4. unit_convert
|
| 71 |
-
u_m = re.match(r'unit_convert\(value=([\d\.]+),\s*from_unit="([^"]+)",\s*to_unit="([^"]+)"\)', call_text)
|
| 72 |
-
if u_m:
|
| 73 |
-
val = float(u_m.group(1))
|
| 74 |
-
f_u, t_u = u_m.group(2), u_m.group(3)
|
| 75 |
-
factors = {
|
| 76 |
-
("km", "miles"): 0.621371,
|
| 77 |
-
("miles", "km"): 1.60934,
|
| 78 |
-
("kg", "lbs"): 2.20462,
|
| 79 |
-
("lbs", "kg"): 0.453592,
|
| 80 |
-
}
|
| 81 |
-
factor = factors.get((f_u, t_u), 1.0)
|
| 82 |
-
return json.dumps({"value": round(val * factor, 2), "unit": t_u})
|
| 83 |
-
|
| 84 |
-
# 5. database_query
|
| 85 |
-
if call_text.startswith("database_query("):
|
| 86 |
-
return json.dumps({"status": "success", "count": 3, "records": [{"id": 101, "status": "active"}]})
|
| 87 |
-
|
| 88 |
-
# 6. create_event
|
| 89 |
-
if call_text.startswith("create_event("):
|
| 90 |
-
return json.dumps({"event_id": "evt_98214", "status": "confirmed"})
|
| 91 |
-
|
| 92 |
-
# 7. file_manager
|
| 93 |
-
if call_text.startswith("file_manager("):
|
| 94 |
-
return json.dumps({"status": "ok", "bytes_written": 64})
|
| 95 |
-
|
| 96 |
-
# 8. fetch_api
|
| 97 |
-
if call_text.startswith("fetch_api("):
|
| 98 |
-
return json.dumps({"status": 200, "data": {"status": "healthy"}})
|
| 99 |
-
|
| 100 |
-
# 9. search(query="...")
|
| 101 |
-
s_m = re.match(r'search\(query="([^"]+)"\)', call_text)
|
| 102 |
-
if s_m:
|
| 103 |
-
query = s_m.group(1).strip()
|
| 104 |
-
q_lower = query.lower()
|
| 105 |
-
|
| 106 |
-
# Offline dictionary lookup for instant, deterministic testing
|
| 107 |
-
kb = {
|
| 108 |
-
"tomato": "the edible berry of the plant Solanum lycopersicum, commonly prepared as a culinary vegetable.",
|
| 109 |
-
"tomatoes": "the edible berry of the plant Solanum lycopersicum, commonly prepared as a culinary vegetable.",
|
| 110 |
-
"apple": "a round, edible fruit produced by an apple tree (Malus domestica), cultivated worldwide.",
|
| 111 |
-
"apples": "a round, edible fruit produced by an apple tree (Malus domestica), cultivated worldwide.",
|
| 112 |
-
"potato": "a starchy, tuberous root crop of the nightshade family Solanaceae.",
|
| 113 |
-
"potatoes": "a starchy, tuberous root crop of the nightshade family Solanaceae.",
|
| 114 |
-
"coffee": "a brewed drink prepared from roasted coffee beans of flowering Coffea species.",
|
| 115 |
-
"banana": "an elongated, edible fruit produced by several kinds of large herbaceous plants in the genus Musa.",
|
| 116 |
-
"bananas": "an elongated, edible fruit produced by several kinds of large herbaceous plants in the genus Musa.",
|
| 117 |
-
"gravity": "a fundamental interaction causing mutual attraction between all things with mass or energy.",
|
| 118 |
-
"photosynthesis": "a process used by plants to convert light energy into chemical energy stored in glucose.",
|
| 119 |
-
"atom": "the basic unit of a chemical element, consisting of a dense central nucleus surrounded by electrons.",
|
| 120 |
-
"python creator": "Guido van Rossum, a Dutch programmer.",
|
| 121 |
-
"speed of light": "approximately 299,792 kilometers per second in a vacuum.",
|
| 122 |
-
"mount everest": "Earth's highest mountain above sea level, located in the Himalayas.",
|
| 123 |
-
}
|
| 124 |
-
|
| 125 |
-
for k, v in kb.items():
|
| 126 |
-
if k in q_lower or q_lower in k:
|
| 127 |
-
return json.dumps({"snippet": v})
|
| 128 |
-
|
| 129 |
-
# Try live Wikipedia REST API if network is available
|
| 130 |
-
try:
|
| 131 |
-
import urllib.request
|
| 132 |
-
import urllib.parse
|
| 133 |
-
clean_q = urllib.parse.quote(query.replace(" ", "_"))
|
| 134 |
-
url = f"https://en.wikipedia.org/api/rest_v1/page/summary/{clean_q}"
|
| 135 |
-
req = urllib.request.Request(url, headers={"User-Agent": "ScaledGPT-Search/1.0"})
|
| 136 |
-
with urllib.request.urlopen(req, timeout=2.5) as resp:
|
| 137 |
-
data = json.loads(resp.read().decode("utf-8"))
|
| 138 |
-
if "extract" in data and data["extract"]:
|
| 139 |
-
return json.dumps({"snippet": data["extract"][:220]})
|
| 140 |
-
except Exception:
|
| 141 |
-
pass
|
| 142 |
-
|
| 143 |
-
return json.dumps({"snippet": f"{query.capitalize()} is an entity and topic in verified knowledge sources."})
|
| 144 |
-
|
| 145 |
-
# Fallback
|
| 146 |
-
return json.dumps({"status": "executed", "call": call_text})
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
def print_human_readable_output(text: str, is_math: bool = False):
|
| 150 |
-
"""Formats internal model output into clean, human-readable terminal blocks."""
|
| 151 |
-
print("\n" + "-" * 80)
|
| 152 |
-
if is_math:
|
| 153 |
-
m = re.search(r"(?:<BOS>)?(.*?)=(?:<THINK>)?(.*?)(?:</THINK>=)(.*?)(?:<EOS>|$)", text)
|
| 154 |
-
if m:
|
| 155 |
-
problem = m.group(1).strip()
|
| 156 |
-
scratch = m.group(2).strip()
|
| 157 |
-
ans = m.group(3).strip()
|
| 158 |
-
print(f"Problem: {problem} = ?")
|
| 159 |
-
print(f"Scratchpad: {scratch}")
|
| 160 |
-
print(f"Answer: {ans}")
|
| 161 |
-
try:
|
| 162 |
-
if "+" in problem:
|
| 163 |
-
parts = problem.split("+")
|
| 164 |
-
if len(parts) == 2 and parts[0].isdigit() and parts[1].isdigit():
|
| 165 |
-
expected = int(parts[0]) + int(parts[1])
|
| 166 |
-
status = "[PASS]" if str(expected) == ans else "[FAIL]"
|
| 167 |
-
print(f"Validation: {status} (Expected: {expected})")
|
| 168 |
-
except Exception:
|
| 169 |
-
pass
|
| 170 |
-
else:
|
| 171 |
-
print(f"Result: {text}")
|
| 172 |
-
else:
|
| 173 |
-
think_m = re.search(r"<THINK>(.*?)</THINK>", text)
|
| 174 |
-
call_m = re.search(r"<CALL>(.*?)</CALL>", text)
|
| 175 |
-
result_m = re.search(r"<RESULT>(.*?)</RESULT>", text)
|
| 176 |
-
answer_m = re.search(r"<ANSWER>(.*?)</ANSWER>", text)
|
| 177 |
-
|
| 178 |
-
if think_m:
|
| 179 |
-
print(f"Reasoning: {think_m.group(1).strip()}")
|
| 180 |
-
if call_m:
|
| 181 |
-
print(f"Tool Call: {call_m.group(1).strip()}")
|
| 182 |
-
if result_m:
|
| 183 |
-
print(f"Tool Result: {result_m.group(1).strip()}")
|
| 184 |
-
if answer_m:
|
| 185 |
-
print(f"Response: {answer_m.group(1).strip()}")
|
| 186 |
-
if not any([think_m, call_m, result_m, answer_m]):
|
| 187 |
-
print(f"Response: {text}")
|
| 188 |
-
|
| 189 |
-
print("-" * 80)
|
| 190 |
-
print(f"[RAW TOKENS] {text}\n", flush=True)
|
| 191 |
-
|
| 192 |
-
|
| 193 |
-
def run_math_prompt(model: ScaledGPT, tokenizer: ScaledTokenizer, prompt: str, device: torch.device) -> str:
|
| 194 |
-
"""Processes a multi-digit arithmetic expression."""
|
| 195 |
-
clean_prompt = prompt.strip()
|
| 196 |
-
if not clean_prompt.endswith("="):
|
| 197 |
-
clean_prompt += "="
|
| 198 |
-
full_prompt = clean_prompt + "<THINK>"
|
| 199 |
-
|
| 200 |
-
tokens = tokenizer.encode(full_prompt, add_bos=True, add_eos=False)
|
| 201 |
-
x = torch.tensor([tokens], dtype=torch.long, device=device)
|
| 202 |
-
|
| 203 |
-
with torch.no_grad():
|
| 204 |
-
out = model.generate(x, max_new_tokens=700, temperature=0.0, stop_tokens=[tokenizer.eos_id])
|
| 205 |
-
|
| 206 |
-
gen_text = tokenizer.decode(out[0].tolist(), skip_special=False)
|
| 207 |
-
print_human_readable_output(gen_text, is_math=True)
|
| 208 |
-
return gen_text
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
def run_dialogue_prompt(model: ScaledGPT, tokenizer: ScaledTokenizer, user_input: str, device: torch.device) -> str:
|
| 212 |
-
"""Processes a user dialogue turn with live tool execution."""
|
| 213 |
-
prompt = f"<USER>{user_input}</USER><THINK>"
|
| 214 |
-
tokens = tokenizer.encode(prompt, add_bos=True, add_eos=False)
|
| 215 |
-
x = torch.tensor([tokens], dtype=torch.long, device=device)
|
| 216 |
-
|
| 217 |
-
with torch.no_grad():
|
| 218 |
-
stop_ids = [tokenizer.call_e_id, tokenizer.answer_e_id, tokenizer.eos_id]
|
| 219 |
-
out = model.generate(x, max_new_tokens=256, temperature=0.0, stop_tokens=stop_ids)
|
| 220 |
-
|
| 221 |
-
gen_text = tokenizer.decode(out[0].tolist(), skip_special=False)
|
| 222 |
-
|
| 223 |
-
if "<CALL>" in gen_text and "</CALL>" in gen_text:
|
| 224 |
-
call_part = gen_text.split("<CALL>")[1].split("</CALL>")[0]
|
| 225 |
-
result_str = mock_execute_tool(call_part)
|
| 226 |
-
|
| 227 |
-
# Continue generation with the tool result (append tokens directly to avoid double BOS)
|
| 228 |
-
res_suffix = f"<RESULT>{result_str}</RESULT>"
|
| 229 |
-
res_tokens = tokenizer.encode(res_suffix, add_bos=False, add_eos=False)
|
| 230 |
-
c_x = torch.cat([out, torch.tensor([res_tokens], dtype=torch.long, device=device)], dim=1)
|
| 231 |
-
|
| 232 |
-
with torch.no_grad():
|
| 233 |
-
out2 = model.generate(c_x, max_new_tokens=256, temperature=0.0, stop_tokens=[tokenizer.answer_e_id, tokenizer.eos_id])
|
| 234 |
-
|
| 235 |
-
full_dialogue = tokenizer.decode(out2[0].tolist(), skip_special=False)
|
| 236 |
-
else:
|
| 237 |
-
full_dialogue = gen_text
|
| 238 |
-
|
| 239 |
-
print_human_readable_output(full_dialogue, is_math=False)
|
| 240 |
-
return full_dialogue
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
def main():
|
| 244 |
-
parser = argparse.ArgumentParser(description="ScaledGPT Interactive CLI")
|
| 245 |
-
parser.add_argument("--ckpt", type=str, default="checkpoints/cbt/cbt_18m_2d_rope_R3_k45_sft.pt", help="Path to checkpoint")
|
| 246 |
-
parser.add_argument("--prompt", type=str, default=None, help="Execute a single prompt and exit")
|
| 247 |
-
parser.add_argument("--threads", type=int, default=8, help="Number of CPU threads for inference")
|
| 248 |
-
args = parser.parse_args()
|
| 249 |
-
|
| 250 |
-
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
|
| 251 |
-
if device.type == "cpu":
|
| 252 |
-
torch.set_num_threads(args.threads)
|
| 253 |
-
tokenizer = ScaledTokenizer()
|
| 254 |
-
config = ScaledGPTConfig(vocab_size=tokenizer.vocab_size)
|
| 255 |
-
model = ScaledGPT(config, tokenizer=tokenizer).to(device)
|
| 256 |
-
|
| 257 |
-
# Search candidate paths for the trained checkpoint
|
| 258 |
-
candidates = [
|
| 259 |
-
args.ckpt,
|
| 260 |
-
os.path.join(ROOT_DIR, args.ckpt),
|
| 261 |
-
os.path.join(ROOT_DIR, "checkpoints/cbt/cbt_18m_2d_rope_R3_k45_sft.pt"),
|
| 262 |
-
os.path.join(ROOT_DIR, "champion_final.pt"),
|
| 263 |
-
os.path.join(ROOT_DIR, "checkpoints/cbt/cbt_18m_2d_rope_R3_k32_base.pt"),
|
| 264 |
-
os.path.join(ROOT_DIR, "checkpoints/cbt/cbt_18m_2d_rope_R3_k32_pretrain.pt"),
|
| 265 |
-
]
|
| 266 |
-
resolved_path = None
|
| 267 |
-
for p in candidates:
|
| 268 |
-
if os.path.exists(p) and not os.path.isdir(p):
|
| 269 |
-
resolved_path = p
|
| 270 |
-
break
|
| 271 |
-
|
| 272 |
-
if resolved_path is None:
|
| 273 |
-
try:
|
| 274 |
-
from huggingface_hub import hf_hub_download
|
| 275 |
-
print("[INFO] Local weights not found. Automatically pulling champion weights from Hugging Face (Neithanvarvas/CBT)...")
|
| 276 |
-
target_dir = os.path.join(ROOT_DIR, "checkpoints", "cbt")
|
| 277 |
-
resolved_path = hf_hub_download(repo_id="Neithanvarvas/CBT", filename="champion_final.pt", local_dir=target_dir)
|
| 278 |
-
local_path = os.path.join(ROOT_DIR, "checkpoints/cbt/cbt_18m_2d_rope_R3_k45_sft.pt")
|
| 279 |
-
os.replace(resolved_path, local_path)
|
| 280 |
-
resolved_path = local_path
|
| 281 |
-
print(f"[PASS] Successfully pulled weights to: {resolved_path}")
|
| 282 |
-
except Exception as e:
|
| 283 |
-
print(f"[WARN] Automatic Hugging Face pull unsuccessful ({e}).")
|
| 284 |
-
|
| 285 |
-
if resolved_path is not None:
|
| 286 |
-
ckpt = torch.load(resolved_path, map_location=device, weights_only=False)
|
| 287 |
-
model.load_state_dict(ckpt["model_state_dict"])
|
| 288 |
-
step_info = f" (Step {ckpt.get('step')})" if 'step' in ckpt else ""
|
| 289 |
-
print(f"[INFO] Successfully loaded trained checkpoint{step_info} from: {resolved_path}")
|
| 290 |
-
else:
|
| 291 |
-
print(f"[WARN] Running with randomly initialized weights.")
|
| 292 |
-
|
| 293 |
-
model.eval()
|
| 294 |
-
|
| 295 |
-
if args.prompt:
|
| 296 |
-
user_input = args.prompt.strip()
|
| 297 |
-
is_math = bool(re.match(r"^[\d\s\+\-\=]+$", user_input) and ("+" in user_input or "-" in user_input))
|
| 298 |
-
if is_math:
|
| 299 |
-
run_math_prompt(model, tokenizer, user_input, device)
|
| 300 |
-
else:
|
| 301 |
-
run_dialogue_prompt(model, tokenizer, user_input, device)
|
| 302 |
-
return
|
| 303 |
-
|
| 304 |
-
print("\n" + "=" * 80)
|
| 305 |
-
print("CHARACTER BASED TRANSFORMER (CBT, ~18.95M) INTERACTIVE TERMINAL")
|
| 306 |
-
print("Type an arithmetic equation (e.g. '123456789+987654321=') or ask a question.")
|
| 307 |
-
print("Type 'exit' or 'quit' to terminate.")
|
| 308 |
-
print("=" * 80 + "\n")
|
| 309 |
-
|
| 310 |
-
while True:
|
| 311 |
-
try:
|
| 312 |
-
user_input = input("User >> ").strip()
|
| 313 |
-
except (EOFError, KeyboardInterrupt):
|
| 314 |
-
print("\n[INFO] Exiting interactive CLI.")
|
| 315 |
-
break
|
| 316 |
-
|
| 317 |
-
if not user_input or user_input.lower() in ["exit", "quit"]:
|
| 318 |
-
break
|
| 319 |
-
|
| 320 |
-
# Check if input is arithmetic (contains only digits and + or -)
|
| 321 |
-
is_math = bool(re.match(r"^[\d\s\+\-\=]+$", user_input) and ("+" in user_input or "-" in user_input))
|
| 322 |
-
if is_math:
|
| 323 |
-
run_math_prompt(model, tokenizer, user_input, device)
|
| 324 |
-
else:
|
| 325 |
-
run_dialogue_prompt(model, tokenizer, user_input, device)
|
| 326 |
-
|
| 327 |
-
|
| 328 |
-
if __name__ == "__main__":
|
| 329 |
-
main()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
champion_final.pt → model.safetensors
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:44ffcb10b8b4bc31fa7ca7d8a8848c8ee0cbe8c4f2fff9ced9898787b7ce997d
|
| 3 |
+
size 76010552
|
RESEARCH_PAPER.md → paper/CBT-18.95M.md
RENAMED
|
@@ -4,21 +4,38 @@
|
|
| 4 |
|
| 5 |
## Abstract
|
| 6 |
|
| 7 |
-
We set out to train a small autoregressive language model that adds multi-digit integers correctly at lengths it has not seen. A standard 18.91M-parameter decoder with one-dimensional rotary positions (1D-RoPE), trained on 1-12-digit sums with a column scratchpad, solves short sums but scores 0/30 at every unseen length we tested (14-35 digits), with either a character or a 50k-token BPE vocabulary; three alternative architectures (looped, hierarchical, hybrid) do the same
|
| 8 |
|
| 9 |
## 1. Introduction
|
| 10 |
|
| 11 |
Small Transformers trained from scratch learn multi-digit addition within their training lengths but fail beyond them [1, 2, 3, 4, 5]. Addition is a convenient test of whether a model has learned an algorithm rather than a lookup over seen lengths: the procedure is a small finite-state machine applied once per column, and the answer is exactly checkable at any length. This paper reports a sequence of experiments that began with that question and ended with a compact character-level model. We present them in the order we ran them.
|
| 12 |
|
| 13 |
-
1. **A standard language model.** We trained an 18.91M-parameter decoder with 1D-RoPE [6] on 1-12-digit sums with an explicit column scratchpad [7, 8], using a character vocabulary and, separately, a 50k BPE vocabulary. It solved 4- and 8-digit sums, scored at most 14/30 at 12 digits, and scored 0/30 at every unseen length (Section
|
| 14 |
-
2. **Other architectures.** Looped, hierarchical
|
| 15 |
-
3. **Two-dimensional coordinates.** Assigning each token a coordinate $(p, \tau)$, place value and step within the column, and rotating queries and keys by both (Decoupled 2D-RoPE) made the models extrapolate. The extrapolation limit tracks the coordinate range seen in training, which suggests the model learns a fixed relation between positions of equal place value (Section
|
| 16 |
-
4. **From digits to characters.** Character-level models must compose words from characters, which is a known weakness [12, 13, 14]. We asked whether the same relational layout helps a character model with words, and built the best small model we could around it: CBT-18.95M (Sections
|
| 17 |
-
5. **A 1D control.** To find out which results the layout is responsible for, we trained a twin with standard 1D-RoPE on identical data. It does not extrapolate, but it matches the 2D model on tool use (Section
|
| 18 |
|
| 19 |
-
**
|
| 20 |
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
**Addition as a state machine.** For operands with digits $a_i, b_i$ (column $i=0$ is the units digit) the sum is produced by
|
| 24 |
|
|
@@ -28,9 +45,9 @@ with $c_{-1}=0$. The same 200-entry transition table applies at every column, so
|
|
| 28 |
|
| 29 |
**Scratchpad format.** Character models are trained on `A+B=<THINK>` followed by one block `(c+a+b)=c's` per column, from the units column up, then `</THINK>=S`. For example, `47+85=<THINK>(0+7+5)=12(1+4+8)=13</THINK>=132`. BPE models use the same steps in words (`Column 0: 7+5=12, digit 2, carry 1. ...`). The loss is masked on the prompt `A+B=`.
|
| 30 |
|
| 31 |
-
**Benchmark.** For each length $d$ we draw 30 problems with two $d$-digit operands from a stratified generator: 70% uniform random digits, 15% dense carries (all digits 5-9), and 15% long carry cascades. Decoding is greedy. A problem counts as correct only if the answer after `</THINK>=` matches the true sum exactly. We report $k/n$ with 95% Wilson intervals. Lengths inside the training range are labelled *seen*. Baseline traces up to 35 digits fit in their 512-token context, so truncation does not explain the failures below, with
|
| 32 |
|
| 33 |
-
##
|
| 34 |
|
| 35 |
Table 1 shows the 18.9M-parameter baselines ($L=6$, $d_{\text{model}}=512$, 8 heads, 1D-RoPE, 400M training tokens, 1-12-digit operands). All three solve short sums: the character model is 30/30 at 4 and 8 digits. None of them reaches 12 digits reliably, which is the longest length in training: the character model scores 0/30 there and the BPE word-chain model 14/30. Past 12 digits every baseline scores 0/30 at every length we tested, up to 35 digits. So the failure is not confined to unseen lengths; it begins at the edge of the training range, where the model has seen few examples, and is complete beyond it. For the philosophy-mixed BPE model, 42 of its 60 working traces at 30 and 35 digits exceeded the 512-token window, so those two zeros partly reflect truncation; its zeros at 14-25 digits do not.
|
| 36 |
|
|
@@ -45,26 +62,22 @@ Table 1 shows the 18.9M-parameter baselines ($L=6$, $d_{\text{model}}=512$, 8 he
|
|
| 45 |
|
| 46 |
**How they fail.** The wrong answers come from reading the wrong operand digit, not from arithmetic slips. For the 20-digit problem `64012498003866726193+44789680393999591374`, the character model's first column is `(0+9+4)=13`: it read the tens digit of the first operand (9) instead of its units digit (3). It then wrote only nine columns for a 20-column sum and answered `834569453`. At 12 digits (`320623293553+963365697522`) the first column is correct, the second column repeats it (`(0+3+2)=05(0+3+2)=05`, where the tens digits are 5 and 2), and the working stops one column early. With a flat index $t$, the distance between a scratchpad column and the operand digits it must read grows with the number of digits. At unseen lengths the model has to attend at relative offsets it never saw in training, and it reads the wrong digit. Adding 50% philosophy text to the BPE model lowered its short-sum accuracy (21/30 at 4 digits, 18/30 at 8) and did not change the result beyond 12 digits.
|
| 47 |
|
| 48 |
-
##
|
| 49 |
|
| 50 |
-
We trained
|
| 51 |
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
**Table 2.** Alternative architectures do not extrapolate. Same protocol and training data as Table 1; correct answers out of 30. Parameters are non-embedding. The last row is the 2D model of Section 5, shown for comparison.
|
| 55 |
|
| 56 |
| Model | Params | 4 | 8 | 12 | 14 | 16 | 18 | 20 | 25 | 30 | 35 |
|
| 57 |
| :--- | :---: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
|
| 58 |
| Looped (2 blocks x 6) | 6.3M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 59 |
| Hierarchical (fast 4, slow 3) | 7.3M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 60 |
| Hybrid gated recurrence + attention | 18.9M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 61 |
-
| Harmonic register machine (H-NRM) | 18.9M | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 62 |
-
| Abacus-looped (ALSM) | 12.7M | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 63 |
| **Decoupled 2D-RoPE ($k=32$)** | 1.57M | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 28 | 19 |
|
| 64 |
|
| 65 |
-
**Randomized 1D positions.** We also trained four 100k-parameter character models ($L=2$, $d_{\text{model}}=64$) on the same 6,000 1-15-digit sums with a 1024-token context for 3,000 steps: standard 1D-RoPE, 1D-RoPE with a random offset $\Delta \sim \mathcal{U}\{0..512\}$, randomized sorted positions [11], and Decoupled 2D-RoPE with $k=32$ (Section
|
| 66 |
|
| 67 |
-
##
|
| 68 |
|
| 69 |
**Coordinates.** Each token $x_t$ receives a coordinate $(p_t, \tau_t)$ from a deterministic parser applied inside the forward pass, identically in training and inference (Table 3, Part A). An operand digit gets $p$ equal to its place value (0 for units) and $\tau=0$. Inside the scratchpad, each `(` starts a new column, so $p$ advances by one and $\tau$ counts the tokens since the `(` (capped at 9). After `</THINK>=`, the $i$-th answer digit gets $p=\max(0, C-1-i)$, where $C$ is the number of scratchpad columns the model emitted. The model never predicts coordinates, but it controls column advancement by emitting `(`.
|
| 70 |
|
|
@@ -84,7 +97,7 @@ $$p_t \leftarrow p_t + \Delta \cdot \mathbb{I}(p_t > 0),\qquad \Delta \sim \math
|
|
| 84 |
|
| 85 |
so that place coordinates up to $p_{\max} = N_{\text{train}} + k$ appear in training while the prompt start stays at $p=0$. This is a coordinate-space version of randomized positions [11].
|
| 86 |
|
| 87 |
-
**Table 3.** Coordinates assigned by the parser. *Part A, arithmetic*: prompt `47+85=`, scratchpad `<THINK>(0+7+5)=12(1+4+8)=13</THINK>`, answer `=132`. Digits of the same place value share $p$ in the prompt and scratchpad, whatever the operand length. In the answer, $p$ counts down from the last scratchpad column, so when the sum has a carry-out digit (as here, `132`) the answer coordinates are one place lower than the true place values; the model is exact on such sums (Section
|
| 88 |
|
| 89 |
Part A (arithmetic; the middle of the second scratchpad column is omitted, so the 1D index jumps from 17 to 28):
|
| 90 |
|
|
@@ -101,7 +114,7 @@ Part B (dialogue):
|
|
| 101 |
| Segment $p_t$ | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 4 | 4 | 4 |
|
| 102 |
| Character $\tau_t$ | 0 | 1 | 2 | 3 | 4 | 0 | 1 | 2 | 3 | 0 | 1 | 2 | 3 | 4 | 5 | 0 | 1 | 2 | 0 | 1 | 2 |
|
| 103 |
|
| 104 |
-
**Result 1: a small 2D model extrapolates.** We trained a 1.57M-parameter (non-embedding) model with Decoupled 2D-RoPE and shift masking $k=32$ (embedding width 256, 4 heads, a two-level fast/slow recurrent core) on the same data as the baselines: 400M tokens of 1-12-digit sums with the same scratchpad. Table 2 (last row) shows it solves every sum we tested from 4 to 25 digits (240/240 over 8 lengths, including 12 digits, its longest training length), more than twice the training length, and scores 28/30 at 30 digits and 19/30 at 35. The 18.9M-parameter baselines with the same data and 12x more parameters score 0/30 at 14 digits. The accuracy begins to fall at 30 digits, below $p_{\max} = 12 + 32 = 44$, so the horizon is not sharp; we have no explanation for the gap beyond noting that the 100k models in Result 3 and CBT-18.95M in Section
|
| 105 |
|
| 106 |
**Sample efficiency.** The same architecture trained for 250-600 steps on only $N$ sums (Table 4) fits the training range with $N=1000$ (29/30 at 4 digits, 30/30 at 8, 28/30 at 12) but extrapolates only briefly (22/30 at 14 digits, 14/30 at 16, 1/30 at 25). With $N=100$ and $N=10$ it did not fit the task (2/30 and 0/30 at 4 digits). So the extrapolation we see with full training is not free: it needs the model to fit the training range first, and 1000 examples are enough only for a short margin.
|
| 107 |
|
|
@@ -137,13 +150,13 @@ Part B (dialogue):
|
|
| 137 |
|
| 138 |
**Interpretation.** With a flat index, "read the operand digit of the current column" is a different relative offset for every column and every problem length, so a model trained on short problems learns offsets that do not exist at longer lengths. With place coordinates the same instruction becomes one relation, attend where $\Delta p = 0$, identical for every column and length. We think the 2D models extrapolate because they learn this single fixed relation, in effect memorizing a linear relation in coordinate space rather than a set of length-specific distances. The relation only holds for coordinate values the model has seen, which is why the limit follows $p_{\max}$ rather than the training length. This account is consistent with the shift sweep and the place-only ablation; we have not tested it directly, for example by probing attention patterns at unseen lengths.
|
| 139 |
|
| 140 |
-
##
|
| 141 |
|
| 142 |
Subword vocabularies give a language model ready-made units that carry meaning. A character-level model has no such units: every word must be assembled from characters, sequences are several times longer, and prior work finds character models harder to train and weaker or slower than subword models at equal scale [12, 13, 14]. Character and byte models remain attractive because they have no tokenization artifacts and a tiny vocabulary [15, 16, 17, 18]. For arithmetic in particular, BPE merges digits of different place values into one token, which breaks the column alignment above.
|
| 143 |
|
| 144 |
-
The addition results suggested a question for character models. If a fixed relation in coordinate space lets a model align digits regardless of length, can the same kind of coordinate help a character model treat a word as a unit? We extended the parser to dialogue (Table 3, Part B): $p$ indexes the segment (`<USER>`, `<THINK>`, `<CALL>`, `<RESULT>`, `<ANSWER>`) and $\tau$ counts characters within it. Copying an entity from the user message into a tool call then becomes a relation between two segments, from $p=0$ to $p=2$, at matching character offsets within the entity, analogous to $\Delta p=0$ in addition. The hypothesis we test is that a compact character model built on this layout can do both exact arithmetic and usable word-level tasks such as tool routing and entity copying. Section
|
| 145 |
|
| 146 |
-
##
|
| 147 |
|
| 148 |
**Architecture.** CBT-18.95M is a decoder with $L=6$ blocks, $d_{\text{model}}=512$, 8 heads ($d_k=64$), pre-LayerNorm, no biases, tied input and output embeddings, and a 1024-token context. The six blocks are applied $R=3$ times with shared weights (18 effective layers) using a residual update scaled by $1/\sqrt{R}$. We kept recurrence as a parameter-efficient way to add depth, not because addition needs it: it gave no benefit in the 100k ablation, and we did not test $R=1$ at this scale. The model has 18,944,512 parameters, 18,887,680 of them outside the embedding table, or about 36 MB in FP16.
|
| 149 |
|
|
@@ -157,15 +170,47 @@ The addition results suggested a question for character models. If a fixed relat
|
|
| 157 |
4. *Intermediate fine-tuning* (12,000 steps, final loss 0.0034, $k=45$): 45% multi-tool dialogues, 35% search copy and 20% arithmetic replay with 1-25-digit operands. This is the *intermediate model*.
|
| 158 |
5. *Final fine-tuning* (23,927 steps, stopped by a 6-hour budget, $k=45$): a decontaminated online stream of 35% search copy, 20% arithmetic replay (1-25 digits) and the rest multi-tool and chat dialogues, with loss weight 2 on tool-call arguments. Its phrasings were broadened after the intermediate model's failures were seen. Held-out tool exactness at the end of training was 63/64 and held-out math 100%. **This is CBT-18.95M.**
|
| 159 |
|
| 160 |
-
Because of the replay, sums of up to 25 digits are seen in training and $p_{\max} = 25 + 45 = 70$. All test prompts and entities were checked against the fine-tuning corpus. Section
|
| 161 |
|
| 162 |
-
##
|
| 163 |
|
| 164 |
All results in this section were obtained on CPU from the checkpoints, with greedy decoding. Tool calls are free-running and scored by exact match of the emitted call. Arithmetic is scored by teacher-forced greedy agreement: a sum counts as correct only if the greedy argmax equals the gold scratchpad and answer at every position after the prompt, which is the condition for free-running greedy decoding to reproduce the gold sequence. Each length has 30 random sums (70% uniform digits, 15% dense carries, 15% cascades such as $99\ldots9+b$).
|
| 165 |
|
| 166 |
**Arithmetic.** Table 6 shows that CBT-18.95M is exact on all 150 sums at 4-16 digits, on all 60 at 20 and 25 digits, and on all 210 at 26-55 digits (Wilson 95% CI 98.2-100%), which are longer than any sum in training. Accuracy falls at 60 digits (25/30) and 70 digits (19/30), still inside $p_{\max}=70$ (254/270 pooled over 26-70 digits, CI 90.6-96.3%). Localizing the first wrong token of the 16 failures at 55-70 digits shows that all are uniform-digit sums with an error inside the scratchpad, at columns 59-70 (token positions 705-840). There is no error in columns 1-58 on any problem, and every cascade (17/17) and dense-carry sum (9/9) at 55-70 digits is exact, so the failures are local errors in a column's carry or digit, not the digit drift of the 1D baselines. At 80 and 100 digits the full trace would be about 1,150 and 1,430 tokens, longer than the 1,024-token context. For reference, the intermediate model scored free-running on a different random sample was exact on 240/240 sums of 16-50 digits, 14/20 at 60 and 6/15 at 70 digits.
|
| 167 |
|
| 168 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
|
| 170 |
| Digits | Status | 2D-RoPE | 1D-RoPE |
|
| 171 |
| :--- | :--- | :-: | :-: |
|
|
@@ -179,7 +224,7 @@ All results in this section were obtained on CPU from the checkpoints, with gree
|
|
| 179 |
|
| 180 |
**Tool use.** The phrasing suite has 26 prompts with paraphrases and invented entities over weather, time, unit conversion, search, calculator, conversational reply and four nested-schema tools (database query, HTTP fetch, calendar event, file manager), whose schemas occur in the training data. The out-of-distribution (OOD) suite has 18 prompts whose entities do not occur in the fine-tuning corpus (audited: 0 overlaps). The unseen-copy suite has 30 prompts in five groups (historical figures, cities, science terms, concepts, invented words), none of whose entities appear in any training stage. Table 7 reports routing (right tool) and exact match (right tool and argument string).
|
| 181 |
|
| 182 |
-
**Table 7.** Tool calls across fine-tuning stages. Right tool / exact call. The base is the pre-SFT checkpoint, Intermediate is the model after the first fine-tuning round, 2D is CBT-18.95M and 1D its twin. The base was scored only on unseen copy. The final fine-tuning data was broadened against the phrasing and OOD suites (Section
|
| 183 |
|
| 184 |
| Suite | Base | Interm. | 2D | 1D |
|
| 185 |
| :--- | :-: | :-: | :-: | :-: |
|
|
@@ -189,33 +234,30 @@ All results in this section were obtained on CPU from the checkpoints, with gree
|
|
| 189 |
|
| 190 |
CBT-18.95M routes and copies correctly on all 74 prompts (Wilson 95% lower bounds 87%, 82% and 89% for the three suites). The unseen-copy set is the cleanest evidence, since none of its entities occur in any training corpus: all five groups are 6/6, and the exact-copy rate is above 50% (one-sided binomial $p=9.3\times10^{-10}$). The phrasing and OOD suites are less clean because the final fine-tuning data was broadened after the intermediate model's failures on them were seen.
|
| 191 |
|
| 192 |
-
**Effect of the fine-tuning stages on copying.** Copying is the weak part of a character model, and the earlier stages show it. On the unseen-copy suite the pre-SFT base copies 4/30 entities exactly and the intermediate model 17/30 (routing roughly constant, 28 and 29 of 30). The base's errors are mostly fragments (`Belo` for Belo Horizonte, `Cai` for Chiang Mai, `quarks` for quarks and leptons); the intermediate model's are mostly one- or two-character slips (`Barch Spinoza`, `Ada Nonath`, `Chiang Maix`) and, on the OOD and phrasing suites, corrupted spellings (`Auccra` for Accra, `Rerevan` for Yerevan, `zggdrasil`) and truncated names (`Santo` for Santo Domingo). The intermediate model also routed all four nested-schema phrasings wrongly (three of them to `search`). CBT-18.95M gets every one of these cases exact, including the city, time-zone and multi-word cases that the intermediate model corrupted. This is the word-formation weakness of character models discussed in Section
|
| 193 |
|
| 194 |
-
##
|
| 195 |
|
| 196 |
-
The experiments so far leave open which of CBT-18.95M's results the coordinate layout is responsible for. We trained a twin that differs in position encoding only: standard RoPE over the whole head dimension at positions $0,\dots,T-1$, with the same architecture (18,944,512 parameters), vocabulary, intermediate recipe (12,000 steps) and final recipe (23,927 steps, step-matched). The final-stage corpus is byte-identical after decompression (SHA256 of the text matches). A uniform shift of positions is an exact no-op under RoPE, so the twin has no shift augmentation, and the comparison is between the full 2D scheme and plain 1D-RoPE. The lineages also differ before the intermediate stage: the 2D model's pretraining stopped at step 45,059 while the twin completed 50,000 steps, and the 2D model also received the search-copy and all-tools stages (11,000 steps, neither with arithmetic) that the twin skipped. Both models were scored with the harness of Section
|
| 197 |
|
| 198 |
-
**Arithmetic.** The twin drops from 29/30 at 25 digits to 0/30 at 26, one digit past the training maximum, and scores 0 on every unseen length (Table 6, 0/270 pooled, CI 0.0-1.4%). Localizing the first wrong token of the twin's 150 failures at 26, 30, 40, 55 and 70 digits shows that all of them lie inside the scratchpad. For the 124 uniform and dense sums the first error is in column 1 (93), 2 (29) or 3 (2): the twin misreads an operand digit at the start of the scratchpad. For the 26 carry cascades it is in columns 22-27 and always writes `0` for an operand digit `9`, as if the operand of nines ended near the 25-digit training maximum. This contrasts with the 2D model, whose first error is never before column 59. The pattern is consistent with the Proposition in Section
|
| 199 |
|
| 200 |
**Tool use.** Table 7 shows no such gap. The twin routes and copies as well as the 2D model on all three suites (73/74 against 74/74 exact calls); its one failure, the HTTP-fetch phrasing, picks the right tool and endpoint and then breaks the `params` dictionary so that the call never closes.
|
| 201 |
|
| 202 |
-
This changes how Section
|
| 203 |
-
|
| 204 |
-
## 10. Discussion and Limitations
|
| 205 |
|
| 206 |
-
|
| 207 |
|
| 208 |
**Limitations.**
|
| 209 |
|
| 210 |
- The coordinates are assigned by a rule-based parser, not learned, and the parser assumes the `A+B=` format. The arithmetic result is conditional on correct coordinates and covers only addition with a fixed scratchpad.
|
| 211 |
-
- The 1D control (Section
|
| 212 |
- The final fine-tuning data were broadened after the intermediate model's failures on the phrasing and OOD suites were seen, so CBT-18.95M's scores on those suites partly reflect targeted coverage. The unseen-copy set, whose entities are absent from the corpus, is the cleaner evidence. The twin shares this data, so the 2D/1D comparison is unaffected. The ceiling on the three suites (100% observed, lower Wilson bounds of 87%, 82% and 89%) also limits how small a difference they can detect.
|
| 213 |
-
- Two of the alternative architectures (H-NRM and ALSM) did not fit the task, so Table 2 cannot show that those designs fail to extrapolate, only that they did not train here.
|
| 214 |
- Every configuration is a single run, and several test sets are small (14 cases for the factor ablation, 26 and 18 for tools, 30 per length for arithmetic).
|
| 215 |
- Extrapolation is bounded by $p_{\max}$, and CBT-18.95M already degrades below it (60 and 70 digits).
|
| 216 |
- Generation is character by character with three recurrent passes, so per-token compute is about that of a 57M-parameter dense model; the 36 MB figure covers weights only.
|
| 217 |
|
| 218 |
-
##
|
| 219 |
|
| 220 |
We asked a small autoregressive model to add integers longer than any it had seen. Standard 1D-RoPE Transformers with a character or BPE vocabulary solved short sums and then scored 0/30 at every length past 12 digits, and three recurrent or hierarchical variants did the same. Giving each token a two-dimensional coordinate for place value and step let a 1.57M-parameter model add 14-25-digit sums without error (150/150) after training on 1-12 digits, and controlled 100k-parameter runs show that its limit follows the range of coordinates seen in training. We read this as the model learning one fixed relation between digits of equal place value, which is a hypothesis we have not tested directly.
|
| 221 |
|
|
|
|
| 4 |
|
| 5 |
## Abstract
|
| 6 |
|
| 7 |
+
We set out to train a small autoregressive language model that adds multi-digit integers correctly at lengths it has not seen. A standard 18.91M-parameter decoder with one-dimensional rotary positions (1D-RoPE), trained on 1-12-digit sums with a column scratchpad, solves short sums but scores 0/30 at every unseen length we tested (14-35 digits), with either a character or a 50k-token BPE vocabulary; three alternative architectures (looped, hierarchical, hybrid) do the same. Giving each token a two-dimensional coordinate, the decimal place $p$ and the step $\tau$ within a column, through Decoupled 2D-RoPE changes the outcome: a 1.57M-parameter model trained on the same 1-12-digit range is exact on 150/150 sums of 14-25 digits and on 47/60 sums of 30-35 digits, and in controlled 100k-parameter runs the extrapolation limit moves with the coordinate range seen in training ($p_{\max} = N_{\text{train}} + k$). Our interpretation is that the coordinates turn place-value alignment into one fixed relation ($\Delta p = 0$) that does not change with length, so the model learns it once. Character-level models have no meaning-bearing subword embeddings and must assemble every word from characters, which prior work finds hard. We therefore asked whether the same relational layout, with $p$ indexing the dialogue segment and $\tau$ the character within it, could support a compact character-level model that also handles words. The resulting Character-Based Transformer (CBT-18.95M) is exact on 210/210 random sums of 26-55 digits, all longer than any sum in training (254/270 up to 70 digits), and exact on all 74 prompts of three tool-use suites (26/26 phrasing, 18/18 out-of-distribution, and 30/30 copies of entities absent from the training corpus, against 4/30 for the pre-fine-tuning base). Its final fine-tuning data was broadened after earlier failures were seen, so the unseen-entity set is the cleaner evidence. To attribute these results we trained a 1D-RoPE twin with the same architecture, data and step count. It scores 0/270 on every unseen length from 26 to 70 digits, but matches the 2D model on tool use (73/74 exact calls against 74/74). The 2D coordinates are therefore responsible for length generalization, while the tool-use and copying results are not attributable to them.
|
| 8 |
|
| 9 |
## 1. Introduction
|
| 10 |
|
| 11 |
Small Transformers trained from scratch learn multi-digit addition within their training lengths but fail beyond them [1, 2, 3, 4, 5]. Addition is a convenient test of whether a model has learned an algorithm rather than a lookup over seen lengths: the procedure is a small finite-state machine applied once per column, and the answer is exactly checkable at any length. This paper reports a sequence of experiments that began with that question and ended with a compact character-level model. We present them in the order we ran them.
|
| 12 |
|
| 13 |
+
1. **A standard language model.** We trained an 18.91M-parameter decoder with 1D-RoPE [6] on 1-12-digit sums with an explicit column scratchpad [7, 8], using a character vocabulary and, separately, a 50k BPE vocabulary. It solved 4- and 8-digit sums, scored at most 14/30 at 12 digits, and scored 0/30 at every unseen length (Section 4).
|
| 14 |
+
2. **Other architectures.** Looped, hierarchical and hybrid recurrent variants [9, 10], trained the same way, either scored 0/30 on every length from 12 digits up (three models), and randomized 1D positions [11] did not help a 100k model (Section 5).
|
| 15 |
+
3. **Two-dimensional coordinates.** Assigning each token a coordinate $(p, \tau)$, place value and step within the column, and rotating queries and keys by both (Decoupled 2D-RoPE) made the models extrapolate. The extrapolation limit tracks the coordinate range seen in training, which suggests the model learns a fixed relation between positions of equal place value (Section 6).
|
| 16 |
+
4. **From digits to characters.** Character-level models must compose words from characters, which is a known weakness [12, 13, 14]. We asked whether the same relational layout helps a character model with words, and built the best small model we could around it: CBT-18.95M (Sections 7-9).
|
| 17 |
+
5. **A 1D control.** To find out which results the layout is responsible for, we trained a twin with standard 1D-RoPE on identical data. It does not extrapolate, but it matches the 2D model on tool use (Section 10).
|
| 18 |
|
| 19 |
+
**Our contributions** are:
|
| 20 |
|
| 21 |
+
- A re-evaluation of six 1D-position checkpoints (standard, BPE and alternative architectures) on a common random-sum benchmark. None of them solves any unseen length.
|
| 22 |
+
- Evidence from controlled ablations that 2D place-value coordinates are the factor that enables extrapolation, and that the extrapolation limit is set by the trained coordinate range (Section 6).
|
| 23 |
+
- CBT-18.95M, an 18.9M-parameter character model that combines exact addition at lengths up to 55 digits (trained on at most 25) with tool calling and entity copying (Sections 8-9).
|
| 24 |
+
- A matched 1D-RoPE twin of CBT-18.95M, which shows that the 2D coordinates account for the arithmetic extrapolation (254/270 against 0/270 on unseen lengths) and not for the tool-use results (Section 10).
|
| 25 |
+
|
| 26 |
+
All results are single training runs; test sets and their sizes are given with every number. Figures (including the headline Figure 1) are in the PDF version of the paper.
|
| 27 |
+
|
| 28 |
+
## 2. Related Work
|
| 29 |
+
|
| 30 |
+
**Length generalization in arithmetic.** Transformers trained on short sums fail on longer ones [1, 2, 3], and how well they extrapolate depends strongly on the position encoding [4]. Scratchpads of intermediate steps help within the training range [7, 8].
|
| 31 |
+
|
| 32 |
+
**Digit-aligned positions.** Abacus embeddings [5] add to each digit's token embedding a learned embedding indexed by its position within its number, so that digits of equal place value share an index, and they report addition far beyond the training length. Index hints and positional formats [20, 19, 21] insert or reformat tokens to align digits, and randomized positions [11] train on position values drawn from a wider range. Our coordinates enter differently: they drive a rotary rotation inside attention, so only coordinate differences matter ($\Delta p = 0$ for aligned digits), a second coordinate carries the step within a column, and the same mechanism extends to dialogue segments. We did not compare against Abacus embeddings or index hints under our protocol, so we make no claim that our scheme is better.
|
| 33 |
+
|
| 34 |
+
**Depth.** Looped and universal Transformers [9, 10, 22] address length generalization through iterated depth. CBT-18.95M uses weight-tied recurrence, but our ablation found no addition benefit from it.
|
| 35 |
+
|
| 36 |
+
**Character-level models.** Character and byte models avoid tokenization artifacts but must compose every word from characters and tend to be harder to train than subword models [12, 13, 14, 15, 16, 17, 18].
|
| 37 |
+
|
| 38 |
+
## 3. Task and Evaluation Protocol
|
| 39 |
|
| 40 |
**Addition as a state machine.** For operands with digits $a_i, b_i$ (column $i=0$ is the units digit) the sum is produced by
|
| 41 |
|
|
|
|
| 45 |
|
| 46 |
**Scratchpad format.** Character models are trained on `A+B=<THINK>` followed by one block `(c+a+b)=c's` per column, from the units column up, then `</THINK>=S`. For example, `47+85=<THINK>(0+7+5)=12(1+4+8)=13</THINK>=132`. BPE models use the same steps in words (`Column 0: 7+5=12, digit 2, carry 1. ...`). The loss is masked on the prompt `A+B=`.
|
| 47 |
|
| 48 |
+
**Benchmark.** For each length $d$ we draw 30 problems with two $d$-digit operands from a stratified generator: 70% uniform random digits, 15% dense carries (all digits 5-9), and 15% long carry cascades. Decoding is greedy. A problem counts as correct only if the answer after `</THINK>=` matches the true sum exactly. We report $k/n$ with 95% Wilson intervals. Lengths inside the training range are labelled *seen*. Baseline traces up to 35 digits fit in their 512-token context, so truncation does not explain the failures below, with one exception noted where it occurs: the philosophy-mixed BPE model (Section 4).
|
| 49 |
|
| 50 |
+
## 4. Step 1: A Standard Language Model Does Not Extrapolate
|
| 51 |
|
| 52 |
Table 1 shows the 18.9M-parameter baselines ($L=6$, $d_{\text{model}}=512$, 8 heads, 1D-RoPE, 400M training tokens, 1-12-digit operands). All three solve short sums: the character model is 30/30 at 4 and 8 digits. None of them reaches 12 digits reliably, which is the longest length in training: the character model scores 0/30 there and the BPE word-chain model 14/30. Past 12 digits every baseline scores 0/30 at every length we tested, up to 35 digits. So the failure is not confined to unseen lengths; it begins at the edge of the training range, where the model has seen few examples, and is complete beyond it. For the philosophy-mixed BPE model, 42 of its 60 working traces at 30 and 35 digits exceeded the 512-token window, so those two zeros partly reflect truncation; its zeros at 14-25 digits do not.
|
| 53 |
|
|
|
|
| 62 |
|
| 63 |
**How they fail.** The wrong answers come from reading the wrong operand digit, not from arithmetic slips. For the 20-digit problem `64012498003866726193+44789680393999591374`, the character model's first column is `(0+9+4)=13`: it read the tens digit of the first operand (9) instead of its units digit (3). It then wrote only nine columns for a 20-column sum and answered `834569453`. At 12 digits (`320623293553+963365697522`) the first column is correct, the second column repeats it (`(0+3+2)=05(0+3+2)=05`, where the tens digits are 5 and 2), and the working stops one column early. With a flat index $t$, the distance between a scratchpad column and the operand digits it must read grows with the number of digits. At unseen lengths the model has to attend at relative offsets it never saw in training, and it reads the wrong digit. Adding 50% philosophy text to the BPE model lowered its short-sum accuracy (21/30 at 4 digits, 18/30 at 8) and did not change the result beyond 12 digits.
|
| 64 |
|
| 65 |
+
## 5. Step 2: Other Architectures Do Not Fix It
|
| 66 |
|
| 67 |
+
We trained three other architectures on the same data and budget (Table 2): a looped Transformer that reuses two blocks six times [9, 10], a hierarchical fast/slow recurrent model, and a hybrid of gated linear recurrence and attention. The looped, hierarchical and hybrid models behave like the plain Transformer: 30/30 at 4 and 8 digits, 0/30 at 12 and at every longer length. Recurrence and hierarchy do not make a flat index generalize.
|
| 68 |
|
| 69 |
+
**Table 2.** Alternative architectures do not extrapolate. Same protocol and training data as Table 1; correct answers out of 30. Parameters are non-embedding. The last row is the 2D model of Section 6, shown for comparison.
|
|
|
|
|
|
|
| 70 |
|
| 71 |
| Model | Params | 4 | 8 | 12 | 14 | 16 | 18 | 20 | 25 | 30 | 35 |
|
| 72 |
| :--- | :---: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
|
| 73 |
| Looped (2 blocks x 6) | 6.3M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 74 |
| Hierarchical (fast 4, slow 3) | 7.3M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 75 |
| Hybrid gated recurrence + attention | 18.9M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
|
|
|
|
|
|
| 76 |
| **Decoupled 2D-RoPE ($k=32$)** | 1.57M | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 28 | 19 |
|
| 77 |
|
| 78 |
+
**Randomized 1D positions.** We also trained four 100k-parameter character models ($L=2$, $d_{\text{model}}=64$) on the same 6,000 1-15-digit sums with a 1024-token context for 3,000 steps: standard 1D-RoPE, 1D-RoPE with a random offset $\Delta \sim \mathcal{U}\{0..512\}$, randomized sorted positions [11], and Decoupled 2D-RoPE with $k=32$ (Section 6). Evaluated on 200 random problems per length, all three 1D variants scored 0/600 even on seen 8-15-digit sums (final losses 0.098, 0.096 and 0.232), while the 2D model reached 417/600 on seen lengths and 977/1,200 (81.4%) on unseen 16-40 digits (identical on re-evaluation). Retraining 1D-RoPE for 6,000 steps with a tuned learning rate reached 56/200 at 8 digits and 0/200 from 12 digits on (Table 5). In this setting the 1D models do not learn the task at all, so this comparison concerns trainability; the length-generalization failure of 1D models that do fit the task is the one shown in Table 1.
|
| 79 |
|
| 80 |
+
## 6. Step 3: Two-Dimensional Coordinates
|
| 81 |
|
| 82 |
**Coordinates.** Each token $x_t$ receives a coordinate $(p_t, \tau_t)$ from a deterministic parser applied inside the forward pass, identically in training and inference (Table 3, Part A). An operand digit gets $p$ equal to its place value (0 for units) and $\tau=0$. Inside the scratchpad, each `(` starts a new column, so $p$ advances by one and $\tau$ counts the tokens since the `(` (capped at 9). After `</THINK>=`, the $i$-th answer digit gets $p=\max(0, C-1-i)$, where $C$ is the number of scratchpad columns the model emitted. The model never predicts coordinates, but it controls column advancement by emitting `(`.
|
| 83 |
|
|
|
|
| 97 |
|
| 98 |
so that place coordinates up to $p_{\max} = N_{\text{train}} + k$ appear in training while the prompt start stays at $p=0$. This is a coordinate-space version of randomized positions [11].
|
| 99 |
|
| 100 |
+
**Table 3.** Coordinates assigned by the parser. *Part A, arithmetic*: prompt `47+85=`, scratchpad `<THINK>(0+7+5)=12(1+4+8)=13</THINK>`, answer `=132`. Digits of the same place value share $p$ in the prompt and scratchpad, whatever the operand length. In the answer, $p$ counts down from the last scratchpad column, so when the sum has a carry-out digit (as here, `132`) the answer coordinates are one place lower than the true place values; the model is exact on such sums (Section 9). *Part B, dialogue*: $p$ is the index of the most recent segment tag and $\tau$ counts characters since that tag.
|
| 101 |
|
| 102 |
Part A (arithmetic; the middle of the second scratchpad column is omitted, so the 1D index jumps from 17 to 28):
|
| 103 |
|
|
|
|
| 114 |
| Segment $p_t$ | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 4 | 4 | 4 |
|
| 115 |
| Character $\tau_t$ | 0 | 1 | 2 | 3 | 4 | 0 | 1 | 2 | 3 | 0 | 1 | 2 | 3 | 4 | 5 | 0 | 1 | 2 | 0 | 1 | 2 |
|
| 116 |
|
| 117 |
+
**Result 1: a small 2D model extrapolates.** We trained a 1.57M-parameter (non-embedding) model with Decoupled 2D-RoPE and shift masking $k=32$ (embedding width 256, 4 heads, a two-level fast/slow recurrent core) on the same data as the baselines: 400M tokens of 1-12-digit sums with the same scratchpad. Table 2 (last row) shows it solves every sum we tested from 4 to 25 digits (240/240 over 8 lengths, including 12 digits, its longest training length), more than twice the training length, and scores 28/30 at 30 digits and 19/30 at 35. The 18.9M-parameter baselines with the same data and 12x more parameters score 0/30 at 14 digits. The accuracy begins to fall at 30 digits, below $p_{\max} = 12 + 32 = 44$, so the horizon is not sharp; we have no explanation for the gap beyond noting that the 100k models in Result 3 and CBT-18.95M in Section 9 also degrade somewhat before $p_{\max}$.
|
| 118 |
|
| 119 |
**Sample efficiency.** The same architecture trained for 250-600 steps on only $N$ sums (Table 4) fits the training range with $N=1000$ (29/30 at 4 digits, 30/30 at 8, 28/30 at 12) but extrapolates only briefly (22/30 at 14 digits, 14/30 at 16, 1/30 at 25). With $N=100$ and $N=10$ it did not fit the task (2/30 and 0/30 at 4 digits). So the extrapolation we see with full training is not free: it needs the model to fit the training range first, and 1000 examples are enough only for a short margin.
|
| 120 |
|
|
|
|
| 150 |
|
| 151 |
**Interpretation.** With a flat index, "read the operand digit of the current column" is a different relative offset for every column and every problem length, so a model trained on short problems learns offsets that do not exist at longer lengths. With place coordinates the same instruction becomes one relation, attend where $\Delta p = 0$, identical for every column and length. We think the 2D models extrapolate because they learn this single fixed relation, in effect memorizing a linear relation in coordinate space rather than a set of length-specific distances. The relation only holds for coordinate values the model has seen, which is why the limit follows $p_{\max}$ rather than the training length. This account is consistent with the shift sweep and the place-only ablation; we have not tested it directly, for example by probing attention patterns at unseen lengths.
|
| 152 |
|
| 153 |
+
## 7. Step 4: From Digits to Characters
|
| 154 |
|
| 155 |
Subword vocabularies give a language model ready-made units that carry meaning. A character-level model has no such units: every word must be assembled from characters, sequences are several times longer, and prior work finds character models harder to train and weaker or slower than subword models at equal scale [12, 13, 14]. Character and byte models remain attractive because they have no tokenization artifacts and a tiny vocabulary [15, 16, 17, 18]. For arithmetic in particular, BPE merges digits of different place values into one token, which breaks the column alignment above.
|
| 156 |
|
| 157 |
+
The addition results suggested a question for character models. If a fixed relation in coordinate space lets a model align digits regardless of length, can the same kind of coordinate help a character model treat a word as a unit? We extended the parser to dialogue (Table 3, Part B): $p$ indexes the segment (`<USER>`, `<THINK>`, `<CALL>`, `<RESULT>`, `<ANSWER>`) and $\tau$ counts characters within it. Copying an entity from the user message into a tool call then becomes a relation between two segments, from $p=0$ to $p=2$, at matching character offsets within the entity, analogous to $\Delta p=0$ in addition. The hypothesis we test is that a compact character model built on this layout can do both exact arithmetic and usable word-level tasks such as tool routing and entity copying. Section 10 tests whether the layout is responsible for the language results by training a 1D twin on the same data.
|
| 158 |
|
| 159 |
+
## 8. CBT-18.95M
|
| 160 |
|
| 161 |
**Architecture.** CBT-18.95M is a decoder with $L=6$ blocks, $d_{\text{model}}=512$, 8 heads ($d_k=64$), pre-LayerNorm, no biases, tied input and output embeddings, and a 1024-token context. The six blocks are applied $R=3$ times with shared weights (18 effective layers) using a residual update scaled by $1/\sqrt{R}$. We kept recurrence as a parameter-efficient way to add depth, not because addition needs it: it gave no benefit in the 100k ablation, and we did not test $R=1$ at this scale. The model has 18,944,512 parameters, 18,887,680 of them outside the embedding table, or about 36 MB in FP16.
|
| 162 |
|
|
|
|
| 170 |
4. *Intermediate fine-tuning* (12,000 steps, final loss 0.0034, $k=45$): 45% multi-tool dialogues, 35% search copy and 20% arithmetic replay with 1-25-digit operands. This is the *intermediate model*.
|
| 171 |
5. *Final fine-tuning* (23,927 steps, stopped by a 6-hour budget, $k=45$): a decontaminated online stream of 35% search copy, 20% arithmetic replay (1-25 digits) and the rest multi-tool and chat dialogues, with loss weight 2 on tool-call arguments. Its phrasings were broadened after the intermediate model's failures were seen. Held-out tool exactness at the end of training was 63/64 and held-out math 100%. **This is CBT-18.95M.**
|
| 172 |
|
| 173 |
+
Because of the replay, sums of up to 25 digits are seen in training and $p_{\max} = 25 + 45 = 70$. All test prompts and entities were checked against the fine-tuning corpus. Section 9 evaluates CBT-18.95M and, where noted, the earlier stages; Section 10 compares it with a 1D-RoPE twin.
|
| 174 |
|
| 175 |
+
## 9. Results of CBT-18.95M
|
| 176 |
|
| 177 |
All results in this section were obtained on CPU from the checkpoints, with greedy decoding. Tool calls are free-running and scored by exact match of the emitted call. Arithmetic is scored by teacher-forced greedy agreement: a sum counts as correct only if the greedy argmax equals the gold scratchpad and answer at every position after the prompt, which is the condition for free-running greedy decoding to reproduce the gold sequence. Each length has 30 random sums (70% uniform digits, 15% dense carries, 15% cascades such as $99\ldots9+b$).
|
| 178 |
|
| 179 |
**Arithmetic.** Table 6 shows that CBT-18.95M is exact on all 150 sums at 4-16 digits, on all 60 at 20 and 25 digits, and on all 210 at 26-55 digits (Wilson 95% CI 98.2-100%), which are longer than any sum in training. Accuracy falls at 60 digits (25/30) and 70 digits (19/30), still inside $p_{\max}=70$ (254/270 pooled over 26-70 digits, CI 90.6-96.3%). Localizing the first wrong token of the 16 failures at 55-70 digits shows that all are uniform-digit sums with an error inside the scratchpad, at columns 59-70 (token positions 705-840). There is no error in columns 1-58 on any problem, and every cascade (17/17) and dense-carry sum (9/9) at 55-70 digits is exact, so the failures are local errors in a column's carry or digit, not the digit drift of the 1D baselines. At 80 and 100 digits the full trace would be about 1,150 and 1,430 tokens, longer than the 1,024-token context. For reference, the intermediate model scored free-running on a different random sample was exact on 240/240 sums of 16-50 digits, 14/20 at 60 and 6/15 at 70 digits.
|
| 180 |
|
| 181 |
+
Table 6b varies the two operand lengths independently, with 10 random sums per cell (rows: digits of the second operand b, columns: digits of the first operand a; entries are correct sums out of 10). The 2D model is exact on all 250 cells inside the trained range (both operands up to 25 digits) and on 644 of the 750 sums that have at least one longer operand; the twin gets 234/250 inside the range and 0/750 outside it. The 2D model is perfect for operands up to 50 digits in both positions, and its errors are concentrated where an operand has 60 or 70 digits (195/250 when both exceed 25 digits, with the weakest row and column at 70). So unequal operand lengths cause no extra difficulty beyond the length of the longer operand.
|
| 182 |
+
|
| 183 |
+
**Table 6b.** Addition accuracy by operand lengths, CBT-18.95M (2D-RoPE), correct out of 10 per cell. Operand lengths seen in training: at most 25 digits.
|
| 184 |
+
|
| 185 |
+
| b \ a | 5 | 10 | 15 | 20 | 25 | 30 | 40 | 50 | 60 | 70 |
|
| 186 |
+
| :-- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
|
| 187 |
+
| **5** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 9 | 9 |
|
| 188 |
+
| **10** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 7 | 7 |
|
| 189 |
+
| **15** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 8 | 7 |
|
| 190 |
+
| **20** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 9 | 6 |
|
| 191 |
+
| **25** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 2 |
|
| 192 |
+
| **30** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 9 | 3 |
|
| 193 |
+
| **40** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 4 |
|
| 194 |
+
| **50** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 8 | 5 |
|
| 195 |
+
| **60** | 9 | 10 | 9 | 9 | 9 | 9 | 9 | 9 | 10 | 6 |
|
| 196 |
+
| **70** | 8 | 6 | 6 | 6 | 3 | 5 | 4 | 4 | 4 | 6 |
|
| 197 |
+
|
| 198 |
+
**Table 6c.** The same grid for the 1D-RoPE twin.
|
| 199 |
+
|
| 200 |
+
| b \ a | 5 | 10 | 15 | 20 | 25 | 30 | 40 | 50 | 60 | 70 |
|
| 201 |
+
| :-- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
|
| 202 |
+
| **5** | 10 | 10 | 10 | 10 | 9 | 0 | 0 | 0 | 0 | 0 |
|
| 203 |
+
| **10** | 10 | 10 | 10 | 10 | 7 | 0 | 0 | 0 | 0 | 0 |
|
| 204 |
+
| **15** | 9 | 10 | 10 | 9 | 7 | 0 | 0 | 0 | 0 | 0 |
|
| 205 |
+
| **20** | 10 | 10 | 9 | 10 | 6 | 0 | 0 | 0 | 0 | 0 |
|
| 206 |
+
| **25** | 10 | 10 | 9 | 9 | 10 | 0 | 0 | 0 | 0 | 0 |
|
| 207 |
+
| **30** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 208 |
+
| **40** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 209 |
+
| **50** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 210 |
+
| **60** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 211 |
+
| **70** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 212 |
+
|
| 213 |
+
**Table 6.** Addition of CBT-18.95M (2D-RoPE) and its 1D-RoPE twin. Correct sums out of 30 per length. Training covers 1-25 digits; the twin is described in Section 10.
|
| 214 |
|
| 215 |
| Digits | Status | 2D-RoPE | 1D-RoPE |
|
| 216 |
| :--- | :--- | :-: | :-: |
|
|
|
|
| 224 |
|
| 225 |
**Tool use.** The phrasing suite has 26 prompts with paraphrases and invented entities over weather, time, unit conversion, search, calculator, conversational reply and four nested-schema tools (database query, HTTP fetch, calendar event, file manager), whose schemas occur in the training data. The out-of-distribution (OOD) suite has 18 prompts whose entities do not occur in the fine-tuning corpus (audited: 0 overlaps). The unseen-copy suite has 30 prompts in five groups (historical figures, cities, science terms, concepts, invented words), none of whose entities appear in any training stage. Table 7 reports routing (right tool) and exact match (right tool and argument string).
|
| 226 |
|
| 227 |
+
**Table 7.** Tool calls across fine-tuning stages. Right tool / exact call. The base is the pre-SFT checkpoint, Intermediate is the model after the first fine-tuning round, 2D is CBT-18.95M and 1D its twin. The base was scored only on unseen copy. The final fine-tuning data was broadened against the phrasing and OOD suites (Section 8).
|
| 228 |
|
| 229 |
| Suite | Base | Interm. | 2D | 1D |
|
| 230 |
| :--- | :-: | :-: | :-: | :-: |
|
|
|
|
| 234 |
|
| 235 |
CBT-18.95M routes and copies correctly on all 74 prompts (Wilson 95% lower bounds 87%, 82% and 89% for the three suites). The unseen-copy set is the cleanest evidence, since none of its entities occur in any training corpus: all five groups are 6/6, and the exact-copy rate is above 50% (one-sided binomial $p=9.3\times10^{-10}$). The phrasing and OOD suites are less clean because the final fine-tuning data was broadened after the intermediate model's failures on them were seen.
|
| 236 |
|
| 237 |
+
**Effect of the fine-tuning stages on copying.** Copying is the weak part of a character model, and the earlier stages show it. On the unseen-copy suite the pre-SFT base copies 4/30 entities exactly and the intermediate model 17/30 (routing roughly constant, 28 and 29 of 30). The base's errors are mostly fragments (`Belo` for Belo Horizonte, `Cai` for Chiang Mai, `quarks` for quarks and leptons); the intermediate model's are mostly one- or two-character slips (`Barch Spinoza`, `Ada Nonath`, `Chiang Maix`) and, on the OOD and phrasing suites, corrupted spellings (`Auccra` for Accra, `Rerevan` for Yerevan, `zggdrasil`) and truncated names (`Santo` for Santo Domingo). The intermediate model also routed all four nested-schema phrasings wrongly (three of them to `search`). CBT-18.95M gets every one of these cases exact, including the city, time-zone and multi-word cases that the intermediate model corrupted. This is the word-formation weakness of character models discussed in Section 7, and the result shows that more targeted training data removes it at this scale.
|
| 238 |
|
| 239 |
+
## 10. Control: A 1D-RoPE Twin
|
| 240 |
|
| 241 |
+
The experiments so far leave open which of CBT-18.95M's results the coordinate layout is responsible for. We trained a twin that differs in position encoding only: standard RoPE over the whole head dimension at positions $0,\dots,T-1$, with the same architecture (18,944,512 parameters), vocabulary, intermediate recipe (12,000 steps) and final recipe (23,927 steps, step-matched). The final-stage corpus is byte-identical after decompression (SHA256 of the text matches). A uniform shift of positions is an exact no-op under RoPE, so the twin has no shift augmentation, and the comparison is between the full 2D scheme and plain 1D-RoPE. The lineages also differ before the intermediate stage: the 2D model's pretraining stopped at step 45,059 while the twin completed 50,000 steps, and the 2D model also received the search-copy and all-tools stages (11,000 steps, neither with arithmetic) that the twin skipped. Both models were scored with the harness of Section 9.
|
| 242 |
|
| 243 |
+
**Arithmetic.** The twin drops from 29/30 at 25 digits to 0/30 at 26, one digit past the training maximum, and scores 0 on every unseen length (Table 6, 0/270 pooled, CI 0.0-1.4%). Localizing the first wrong token of the twin's 150 failures at 26, 30, 40, 55 and 70 digits shows that all of them lie inside the scratchpad. For the 124 uniform and dense sums the first error is in column 1 (93), 2 (29) or 3 (2): the twin misreads an operand digit at the start of the scratchpad. For the 26 carry cascades it is in columns 22-27 and always writes `0` for an operand digit `9`, as if the operand of nines ended near the 25-digit training maximum. This contrasts with the 2D model, whose first error is never before column 59. The pattern is consistent with the Proposition in Section 6: under 1D positions the distance between a scratchpad column and the operand digit it must read grows with the operand length and leaves the trained range, while under 2D coordinates it does not. We did not test that explanation directly, for example with attention maps, and we did not find a single consistent wrong-place offset in the misread digits.
|
| 244 |
|
| 245 |
**Tool use.** Table 7 shows no such gap. The twin routes and copies as well as the 2D model on all three suites (73/74 against 74/74 exact calls); its one failure, the HTTP-fetch phrasing, picks the right tool and endpoint and then breaks the `params` dictionary so that the call never closes.
|
| 246 |
|
| 247 |
+
This changes how Section 7 should be read. The final fine-tuning data, not the layout, produced the gain in tool use and copying: the intermediate model rose from 17/30 to 30/30 exact copies on the unseen set (entities absent from its corpus), and a 1D model trained on the same data reached the same score. The hypothesis that segment and character-offset coordinates help a character model assemble words is therefore not supported at this scale and sample size; a character model with plain 1D-RoPE copies unseen entities just as well once the data cover the failure modes. What the layout does contribute is arithmetic length generalization. The other direction also holds: a character-level model with 1D positions can be a competent tool user and still be unable to add a 26-digit sum.
|
|
|
|
|
|
|
| 248 |
|
| 249 |
+
## 11. Discussion and Limitations
|
| 250 |
|
| 251 |
**Limitations.**
|
| 252 |
|
| 253 |
- The coordinates are assigned by a rule-based parser, not learned, and the parser assumes the `A+B=` format. The arithmetic result is conditional on correct coordinates and covers only addition with a fixed scratchpad.
|
| 254 |
+
- The 1D control (Section 10) compares the full 2D scheme, coordinates and shift augmentation together, with plain 1D-RoPE, so it does not show which component is necessary. The two lineages also differ before the intermediate stage (the twin had 50,000 pretraining steps against 45,059 and skipped two tool fine-tuning stages with no arithmetic), each model is a single run, and the mechanism is inferred from where the twin fails, not tested directly.
|
| 255 |
- The final fine-tuning data were broadened after the intermediate model's failures on the phrasing and OOD suites were seen, so CBT-18.95M's scores on those suites partly reflect targeted coverage. The unseen-copy set, whose entities are absent from the corpus, is the cleaner evidence. The twin shares this data, so the 2D/1D comparison is unaffected. The ceiling on the three suites (100% observed, lower Wilson bounds of 87%, 82% and 89%) also limits how small a difference they can detect.
|
|
|
|
| 256 |
- Every configuration is a single run, and several test sets are small (14 cases for the factor ablation, 26 and 18 for tools, 30 per length for arithmetic).
|
| 257 |
- Extrapolation is bounded by $p_{\max}$, and CBT-18.95M already degrades below it (60 and 70 digits).
|
| 258 |
- Generation is character by character with three recurrent passes, so per-token compute is about that of a 57M-parameter dense model; the 36 MB figure covers weights only.
|
| 259 |
|
| 260 |
+
## 12. Conclusion
|
| 261 |
|
| 262 |
We asked a small autoregressive model to add integers longer than any it had seen. Standard 1D-RoPE Transformers with a character or BPE vocabulary solved short sums and then scored 0/30 at every length past 12 digits, and three recurrent or hierarchical variants did the same. Giving each token a two-dimensional coordinate for place value and step let a 1.57M-parameter model add 14-25-digit sums without error (150/150) after training on 1-12 digits, and controlled 100k-parameter runs show that its limit follows the range of coordinates seen in training. We read this as the model learning one fixed relation between digits of equal place value, which is a hypothesis we have not tested directly.
|
| 263 |
|
RESEARCH_PAPER.pdf → paper/CBT-18.95M.pdf
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6b0c1a95c219f2fd636cc748f8d63809049da42845d69ce70f9cddf059361186
|
| 3 |
+
size 389324
|
random_extrapolation_16_to_100d.json
DELETED
|
@@ -1,655 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"per_length": {
|
| 3 |
-
"16": {
|
| 4 |
-
"digits": 16,
|
| 5 |
-
"n_samples": 30,
|
| 6 |
-
"exact_match": 30,
|
| 7 |
-
"accuracy_pct": 100.0,
|
| 8 |
-
"wilson_95_ci": [
|
| 9 |
-
88.6,
|
| 10 |
-
100.0
|
| 11 |
-
],
|
| 12 |
-
"think_exact": 30,
|
| 13 |
-
"think_acc_pct": 100.0,
|
| 14 |
-
"think_wilson_95_ci": [
|
| 15 |
-
88.6,
|
| 16 |
-
100.0
|
| 17 |
-
],
|
| 18 |
-
"lparen_errors": 0,
|
| 19 |
-
"off_by_one_readout_errors": 0,
|
| 20 |
-
"seq_tokens": 226,
|
| 21 |
-
"elapsed_sec": 20.63,
|
| 22 |
-
"tok_per_sec": 328.6
|
| 23 |
-
},
|
| 24 |
-
"20": {
|
| 25 |
-
"digits": 20,
|
| 26 |
-
"n_samples": 30,
|
| 27 |
-
"exact_match": 30,
|
| 28 |
-
"accuracy_pct": 100.0,
|
| 29 |
-
"wilson_95_ci": [
|
| 30 |
-
88.6,
|
| 31 |
-
100.0
|
| 32 |
-
],
|
| 33 |
-
"think_exact": 30,
|
| 34 |
-
"think_acc_pct": 100.0,
|
| 35 |
-
"think_wilson_95_ci": [
|
| 36 |
-
88.6,
|
| 37 |
-
100.0
|
| 38 |
-
],
|
| 39 |
-
"lparen_errors": 0,
|
| 40 |
-
"off_by_one_readout_errors": 0,
|
| 41 |
-
"seq_tokens": 278,
|
| 42 |
-
"elapsed_sec": 21.19,
|
| 43 |
-
"tok_per_sec": 393.5
|
| 44 |
-
},
|
| 45 |
-
"25": {
|
| 46 |
-
"digits": 25,
|
| 47 |
-
"n_samples": 30,
|
| 48 |
-
"exact_match": 30,
|
| 49 |
-
"accuracy_pct": 100.0,
|
| 50 |
-
"wilson_95_ci": [
|
| 51 |
-
88.6,
|
| 52 |
-
100.0
|
| 53 |
-
],
|
| 54 |
-
"think_exact": 30,
|
| 55 |
-
"think_acc_pct": 100.0,
|
| 56 |
-
"think_wilson_95_ci": [
|
| 57 |
-
88.6,
|
| 58 |
-
100.0
|
| 59 |
-
],
|
| 60 |
-
"lparen_errors": 0,
|
| 61 |
-
"off_by_one_readout_errors": 0,
|
| 62 |
-
"seq_tokens": 343,
|
| 63 |
-
"elapsed_sec": 28.65,
|
| 64 |
-
"tok_per_sec": 359.2
|
| 65 |
-
},
|
| 66 |
-
"30": {
|
| 67 |
-
"digits": 30,
|
| 68 |
-
"n_samples": 30,
|
| 69 |
-
"exact_match": 30,
|
| 70 |
-
"accuracy_pct": 100.0,
|
| 71 |
-
"wilson_95_ci": [
|
| 72 |
-
88.6,
|
| 73 |
-
100.0
|
| 74 |
-
],
|
| 75 |
-
"think_exact": 30,
|
| 76 |
-
"think_acc_pct": 100.0,
|
| 77 |
-
"think_wilson_95_ci": [
|
| 78 |
-
88.6,
|
| 79 |
-
100.0
|
| 80 |
-
],
|
| 81 |
-
"lparen_errors": 0,
|
| 82 |
-
"off_by_one_readout_errors": 0,
|
| 83 |
-
"seq_tokens": 408,
|
| 84 |
-
"elapsed_sec": 46.16,
|
| 85 |
-
"tok_per_sec": 265.2
|
| 86 |
-
},
|
| 87 |
-
"35": {
|
| 88 |
-
"digits": 35,
|
| 89 |
-
"n_samples": 30,
|
| 90 |
-
"exact_match": 30,
|
| 91 |
-
"accuracy_pct": 100.0,
|
| 92 |
-
"wilson_95_ci": [
|
| 93 |
-
88.6,
|
| 94 |
-
100.0
|
| 95 |
-
],
|
| 96 |
-
"think_exact": 30,
|
| 97 |
-
"think_acc_pct": 100.0,
|
| 98 |
-
"think_wilson_95_ci": [
|
| 99 |
-
88.6,
|
| 100 |
-
100.0
|
| 101 |
-
],
|
| 102 |
-
"lparen_errors": 0,
|
| 103 |
-
"off_by_one_readout_errors": 0,
|
| 104 |
-
"seq_tokens": 473,
|
| 105 |
-
"elapsed_sec": 55.95,
|
| 106 |
-
"tok_per_sec": 253.6
|
| 107 |
-
},
|
| 108 |
-
"40": {
|
| 109 |
-
"digits": 40,
|
| 110 |
-
"n_samples": 30,
|
| 111 |
-
"exact_match": 30,
|
| 112 |
-
"accuracy_pct": 100.0,
|
| 113 |
-
"wilson_95_ci": [
|
| 114 |
-
88.6,
|
| 115 |
-
100.0
|
| 116 |
-
],
|
| 117 |
-
"think_exact": 30,
|
| 118 |
-
"think_acc_pct": 100.0,
|
| 119 |
-
"think_wilson_95_ci": [
|
| 120 |
-
88.6,
|
| 121 |
-
100.0
|
| 122 |
-
],
|
| 123 |
-
"lparen_errors": 0,
|
| 124 |
-
"off_by_one_readout_errors": 0,
|
| 125 |
-
"seq_tokens": 538,
|
| 126 |
-
"elapsed_sec": 61.79,
|
| 127 |
-
"tok_per_sec": 261.2
|
| 128 |
-
},
|
| 129 |
-
"45": {
|
| 130 |
-
"digits": 45,
|
| 131 |
-
"n_samples": 30,
|
| 132 |
-
"exact_match": 28,
|
| 133 |
-
"accuracy_pct": 93.3,
|
| 134 |
-
"wilson_95_ci": [
|
| 135 |
-
78.7,
|
| 136 |
-
98.2
|
| 137 |
-
],
|
| 138 |
-
"think_exact": 29,
|
| 139 |
-
"think_acc_pct": 96.7,
|
| 140 |
-
"think_wilson_95_ci": [
|
| 141 |
-
83.3,
|
| 142 |
-
99.4
|
| 143 |
-
],
|
| 144 |
-
"lparen_errors": 0,
|
| 145 |
-
"off_by_one_readout_errors": 0,
|
| 146 |
-
"seq_tokens": 603,
|
| 147 |
-
"elapsed_sec": 63.52,
|
| 148 |
-
"tok_per_sec": 284.8
|
| 149 |
-
},
|
| 150 |
-
"50": {
|
| 151 |
-
"digits": 50,
|
| 152 |
-
"n_samples": 30,
|
| 153 |
-
"exact_match": 29,
|
| 154 |
-
"accuracy_pct": 96.7,
|
| 155 |
-
"wilson_95_ci": [
|
| 156 |
-
83.3,
|
| 157 |
-
99.4
|
| 158 |
-
],
|
| 159 |
-
"think_exact": 29,
|
| 160 |
-
"think_acc_pct": 96.7,
|
| 161 |
-
"think_wilson_95_ci": [
|
| 162 |
-
83.3,
|
| 163 |
-
99.4
|
| 164 |
-
],
|
| 165 |
-
"lparen_errors": 0,
|
| 166 |
-
"off_by_one_readout_errors": 0,
|
| 167 |
-
"seq_tokens": 668,
|
| 168 |
-
"elapsed_sec": 85.37,
|
| 169 |
-
"tok_per_sec": 234.7
|
| 170 |
-
},
|
| 171 |
-
"55": {
|
| 172 |
-
"digits": 55,
|
| 173 |
-
"n_samples": 20,
|
| 174 |
-
"exact_match": 18,
|
| 175 |
-
"accuracy_pct": 90.0,
|
| 176 |
-
"wilson_95_ci": [
|
| 177 |
-
69.9,
|
| 178 |
-
97.2
|
| 179 |
-
],
|
| 180 |
-
"think_exact": 19,
|
| 181 |
-
"think_acc_pct": 95.0,
|
| 182 |
-
"think_wilson_95_ci": [
|
| 183 |
-
76.4,
|
| 184 |
-
99.1
|
| 185 |
-
],
|
| 186 |
-
"lparen_errors": 0,
|
| 187 |
-
"off_by_one_readout_errors": 0,
|
| 188 |
-
"seq_tokens": 799,
|
| 189 |
-
"elapsed_sec": 86.78,
|
| 190 |
-
"tok_per_sec": 184.2
|
| 191 |
-
},
|
| 192 |
-
"60": {
|
| 193 |
-
"digits": 60,
|
| 194 |
-
"n_samples": 20,
|
| 195 |
-
"exact_match": 9,
|
| 196 |
-
"accuracy_pct": 45.0,
|
| 197 |
-
"wilson_95_ci": [
|
| 198 |
-
25.8,
|
| 199 |
-
65.8
|
| 200 |
-
],
|
| 201 |
-
"think_exact": 9,
|
| 202 |
-
"think_acc_pct": 45.0,
|
| 203 |
-
"think_wilson_95_ci": [
|
| 204 |
-
25.8,
|
| 205 |
-
65.8
|
| 206 |
-
],
|
| 207 |
-
"lparen_errors": 3,
|
| 208 |
-
"off_by_one_readout_errors": 0,
|
| 209 |
-
"seq_tokens": 869,
|
| 210 |
-
"elapsed_sec": 81.06,
|
| 211 |
-
"tok_per_sec": 214.4
|
| 212 |
-
},
|
| 213 |
-
"70": {
|
| 214 |
-
"digits": 70,
|
| 215 |
-
"n_samples": 15,
|
| 216 |
-
"exact_match": 1,
|
| 217 |
-
"accuracy_pct": 6.7,
|
| 218 |
-
"wilson_95_ci": [
|
| 219 |
-
1.2,
|
| 220 |
-
29.8
|
| 221 |
-
],
|
| 222 |
-
"think_exact": 1,
|
| 223 |
-
"think_acc_pct": 6.7,
|
| 224 |
-
"think_wilson_95_ci": [
|
| 225 |
-
1.2,
|
| 226 |
-
29.8
|
| 227 |
-
],
|
| 228 |
-
"lparen_errors": 0,
|
| 229 |
-
"off_by_one_readout_errors": 0,
|
| 230 |
-
"seq_tokens": 928,
|
| 231 |
-
"elapsed_sec": 68.66,
|
| 232 |
-
"tok_per_sec": 202.7
|
| 233 |
-
},
|
| 234 |
-
"80": {
|
| 235 |
-
"digits": 80,
|
| 236 |
-
"n_samples": 10,
|
| 237 |
-
"exact_match": 0,
|
| 238 |
-
"accuracy_pct": 0.0,
|
| 239 |
-
"wilson_95_ci": [
|
| 240 |
-
0.0,
|
| 241 |
-
27.8
|
| 242 |
-
],
|
| 243 |
-
"think_exact": 0,
|
| 244 |
-
"think_acc_pct": 0.0,
|
| 245 |
-
"think_wilson_95_ci": [
|
| 246 |
-
0.0,
|
| 247 |
-
27.8
|
| 248 |
-
],
|
| 249 |
-
"lparen_errors": 4,
|
| 250 |
-
"off_by_one_readout_errors": 0,
|
| 251 |
-
"seq_tokens": 1149,
|
| 252 |
-
"elapsed_sec": 72.05,
|
| 253 |
-
"tok_per_sec": 159.5
|
| 254 |
-
},
|
| 255 |
-
"100": {
|
| 256 |
-
"digits": 100,
|
| 257 |
-
"n_samples": 10,
|
| 258 |
-
"exact_match": 0,
|
| 259 |
-
"accuracy_pct": 0.0,
|
| 260 |
-
"wilson_95_ci": [
|
| 261 |
-
0.0,
|
| 262 |
-
27.8
|
| 263 |
-
],
|
| 264 |
-
"think_exact": 0,
|
| 265 |
-
"think_acc_pct": 0.0,
|
| 266 |
-
"think_wilson_95_ci": [
|
| 267 |
-
0.0,
|
| 268 |
-
27.8
|
| 269 |
-
],
|
| 270 |
-
"lparen_errors": 0,
|
| 271 |
-
"off_by_one_readout_errors": 0,
|
| 272 |
-
"seq_tokens": 1429,
|
| 273 |
-
"elapsed_sec": 101.11,
|
| 274 |
-
"tok_per_sec": 141.3
|
| 275 |
-
}
|
| 276 |
-
},
|
| 277 |
-
"failures": [
|
| 278 |
-
{
|
| 279 |
-
"digits": 45,
|
| 280 |
-
"kind": "uniform_random",
|
| 281 |
-
"expected_len": 46,
|
| 282 |
-
"pred_len": 46,
|
| 283 |
-
"think_ok": false,
|
| 284 |
-
"expected_head": "10465044933514809662",
|
| 285 |
-
"pred_head": "10465044933515809662"
|
| 286 |
-
},
|
| 287 |
-
{
|
| 288 |
-
"digits": 45,
|
| 289 |
-
"kind": "dense_carry",
|
| 290 |
-
"expected_len": 46,
|
| 291 |
-
"pred_len": 46,
|
| 292 |
-
"think_ok": true,
|
| 293 |
-
"expected_head": "19673285845443761366",
|
| 294 |
-
"pred_head": "14673285845443761366"
|
| 295 |
-
},
|
| 296 |
-
{
|
| 297 |
-
"digits": 50,
|
| 298 |
-
"kind": "uniform_random",
|
| 299 |
-
"expected_len": 50,
|
| 300 |
-
"pred_len": 50,
|
| 301 |
-
"think_ok": false,
|
| 302 |
-
"expected_head": "68905429896446161053",
|
| 303 |
-
"pred_head": "68905429896446161053"
|
| 304 |
-
},
|
| 305 |
-
{
|
| 306 |
-
"digits": 55,
|
| 307 |
-
"kind": "cascade_mix",
|
| 308 |
-
"expected_len": 55,
|
| 309 |
-
"pred_len": 133,
|
| 310 |
-
"think_ok": true,
|
| 311 |
-
"expected_head": "99999899800000895006",
|
| 312 |
-
"pred_head": "99999899800000895006"
|
| 313 |
-
},
|
| 314 |
-
{
|
| 315 |
-
"digits": 55,
|
| 316 |
-
"kind": "cascade_mix",
|
| 317 |
-
"expected_len": 56,
|
| 318 |
-
"pred_len": 56,
|
| 319 |
-
"think_ok": false,
|
| 320 |
-
"expected_head": "18300979999929997971",
|
| 321 |
-
"pred_head": "18290979999929997971"
|
| 322 |
-
},
|
| 323 |
-
{
|
| 324 |
-
"digits": 60,
|
| 325 |
-
"kind": "uniform_random",
|
| 326 |
-
"expected_len": 61,
|
| 327 |
-
"pred_len": 61,
|
| 328 |
-
"think_ok": false,
|
| 329 |
-
"expected_head": "17794705984524592751",
|
| 330 |
-
"pred_head": "17784705984524592751"
|
| 331 |
-
},
|
| 332 |
-
{
|
| 333 |
-
"digits": 60,
|
| 334 |
-
"kind": "uniform_random",
|
| 335 |
-
"expected_len": 61,
|
| 336 |
-
"pred_len": 62,
|
| 337 |
-
"think_ok": false,
|
| 338 |
-
"expected_head": "10358362896129198913",
|
| 339 |
-
"pred_head": "10035836289612919891"
|
| 340 |
-
},
|
| 341 |
-
{
|
| 342 |
-
"digits": 60,
|
| 343 |
-
"kind": "uniform_random",
|
| 344 |
-
"expected_len": 60,
|
| 345 |
-
"pred_len": 60,
|
| 346 |
-
"think_ok": false,
|
| 347 |
-
"expected_head": "99455736002861930179",
|
| 348 |
-
"pred_head": "99445736002861930179"
|
| 349 |
-
},
|
| 350 |
-
{
|
| 351 |
-
"digits": 60,
|
| 352 |
-
"kind": "uniform_random",
|
| 353 |
-
"expected_len": 61,
|
| 354 |
-
"pred_len": 61,
|
| 355 |
-
"think_ok": false,
|
| 356 |
-
"expected_head": "13771859705436256056",
|
| 357 |
-
"pred_head": "14771859705436256056"
|
| 358 |
-
},
|
| 359 |
-
{
|
| 360 |
-
"digits": 60,
|
| 361 |
-
"kind": "uniform_random",
|
| 362 |
-
"expected_len": 61,
|
| 363 |
-
"pred_len": 61,
|
| 364 |
-
"think_ok": false,
|
| 365 |
-
"expected_head": "11927411935032822174",
|
| 366 |
-
"pred_head": "41927411935032822174"
|
| 367 |
-
},
|
| 368 |
-
{
|
| 369 |
-
"digits": 60,
|
| 370 |
-
"kind": "uniform_random",
|
| 371 |
-
"expected_len": 61,
|
| 372 |
-
"pred_len": 61,
|
| 373 |
-
"think_ok": false,
|
| 374 |
-
"expected_head": "11836982981890419337",
|
| 375 |
-
"pred_head": "11736982981890419337"
|
| 376 |
-
},
|
| 377 |
-
{
|
| 378 |
-
"digits": 60,
|
| 379 |
-
"kind": "uniform_random",
|
| 380 |
-
"expected_len": 61,
|
| 381 |
-
"pred_len": 62,
|
| 382 |
-
"think_ok": false,
|
| 383 |
-
"expected_head": "13495686920877645844",
|
| 384 |
-
"pred_head": "13595686920877645844"
|
| 385 |
-
},
|
| 386 |
-
{
|
| 387 |
-
"digits": 60,
|
| 388 |
-
"kind": "uniform_random",
|
| 389 |
-
"expected_len": 60,
|
| 390 |
-
"pred_len": 60,
|
| 391 |
-
"think_ok": false,
|
| 392 |
-
"expected_head": "33303324997839267821",
|
| 393 |
-
"pred_head": "35303324997839267821"
|
| 394 |
-
},
|
| 395 |
-
{
|
| 396 |
-
"digits": 60,
|
| 397 |
-
"kind": "dense_carry",
|
| 398 |
-
"expected_len": 61,
|
| 399 |
-
"pred_len": 62,
|
| 400 |
-
"think_ok": false,
|
| 401 |
-
"expected_head": "17833947643427329535",
|
| 402 |
-
"pred_head": "31783394764342732953"
|
| 403 |
-
},
|
| 404 |
-
{
|
| 405 |
-
"digits": 60,
|
| 406 |
-
"kind": "cascade_mix",
|
| 407 |
-
"expected_len": 60,
|
| 408 |
-
"pred_len": 133,
|
| 409 |
-
"think_ok": false,
|
| 410 |
-
"expected_head": "99999990999993999900",
|
| 411 |
-
"pred_head": "49999999099999399990"
|
| 412 |
-
},
|
| 413 |
-
{
|
| 414 |
-
"digits": 60,
|
| 415 |
-
"kind": "cascade_mix",
|
| 416 |
-
"expected_len": 61,
|
| 417 |
-
"pred_len": 62,
|
| 418 |
-
"think_ok": false,
|
| 419 |
-
"expected_head": "15999869832993500000",
|
| 420 |
-
"pred_head": "16999869832993500000"
|
| 421 |
-
},
|
| 422 |
-
{
|
| 423 |
-
"digits": 70,
|
| 424 |
-
"kind": "uniform_random",
|
| 425 |
-
"expected_len": 71,
|
| 426 |
-
"pred_len": 71,
|
| 427 |
-
"think_ok": false,
|
| 428 |
-
"expected_head": "13709577792207576758",
|
| 429 |
-
"pred_head": "13610577792207576758"
|
| 430 |
-
},
|
| 431 |
-
{
|
| 432 |
-
"digits": 70,
|
| 433 |
-
"kind": "uniform_random",
|
| 434 |
-
"expected_len": 71,
|
| 435 |
-
"pred_len": 71,
|
| 436 |
-
"think_ok": false,
|
| 437 |
-
"expected_head": "18254804996856400492",
|
| 438 |
-
"pred_head": "18274805096859400492"
|
| 439 |
-
},
|
| 440 |
-
{
|
| 441 |
-
"digits": 70,
|
| 442 |
-
"kind": "uniform_random",
|
| 443 |
-
"expected_len": 71,
|
| 444 |
-
"pred_len": 71,
|
| 445 |
-
"think_ok": false,
|
| 446 |
-
"expected_head": "15468076189969829446",
|
| 447 |
-
"pred_head": "56459076190969829446"
|
| 448 |
-
},
|
| 449 |
-
{
|
| 450 |
-
"digits": 70,
|
| 451 |
-
"kind": "uniform_random",
|
| 452 |
-
"expected_len": 71,
|
| 453 |
-
"pred_len": 71,
|
| 454 |
-
"think_ok": false,
|
| 455 |
-
"expected_head": "17132113958771129774",
|
| 456 |
-
"pred_head": "17292113958771129774"
|
| 457 |
-
},
|
| 458 |
-
{
|
| 459 |
-
"digits": 70,
|
| 460 |
-
"kind": "uniform_random",
|
| 461 |
-
"expected_len": 70,
|
| 462 |
-
"pred_len": 71,
|
| 463 |
-
"think_ok": false,
|
| 464 |
-
"expected_head": "81569945829520864708",
|
| 465 |
-
"pred_head": "81669945829420864708"
|
| 466 |
-
},
|
| 467 |
-
{
|
| 468 |
-
"digits": 70,
|
| 469 |
-
"kind": "uniform_random",
|
| 470 |
-
"expected_len": 71,
|
| 471 |
-
"pred_len": 71,
|
| 472 |
-
"think_ok": false,
|
| 473 |
-
"expected_head": "14659845102561298448",
|
| 474 |
-
"pred_head": "13657849102361298448"
|
| 475 |
-
},
|
| 476 |
-
{
|
| 477 |
-
"digits": 70,
|
| 478 |
-
"kind": "uniform_random",
|
| 479 |
-
"expected_len": 71,
|
| 480 |
-
"pred_len": 71,
|
| 481 |
-
"think_ok": false,
|
| 482 |
-
"expected_head": "12532903951038538802",
|
| 483 |
-
"pred_head": "11332904551038538802"
|
| 484 |
-
},
|
| 485 |
-
{
|
| 486 |
-
"digits": 70,
|
| 487 |
-
"kind": "uniform_random",
|
| 488 |
-
"expected_len": 70,
|
| 489 |
-
"pred_len": 71,
|
| 490 |
-
"think_ok": false,
|
| 491 |
-
"expected_head": "91885390650615687834",
|
| 492 |
-
"pred_head": "91885390980615687834"
|
| 493 |
-
},
|
| 494 |
-
{
|
| 495 |
-
"digits": 70,
|
| 496 |
-
"kind": "uniform_random",
|
| 497 |
-
"expected_len": 71,
|
| 498 |
-
"pred_len": 71,
|
| 499 |
-
"think_ok": false,
|
| 500 |
-
"expected_head": "17102804133024228749",
|
| 501 |
-
"pred_head": "17102903132924228749"
|
| 502 |
-
},
|
| 503 |
-
{
|
| 504 |
-
"digits": 70,
|
| 505 |
-
"kind": "uniform_random",
|
| 506 |
-
"expected_len": 71,
|
| 507 |
-
"pred_len": 71,
|
| 508 |
-
"think_ok": false,
|
| 509 |
-
"expected_head": "14193174229527824431",
|
| 510 |
-
"pred_head": "13273174229827824431"
|
| 511 |
-
},
|
| 512 |
-
{
|
| 513 |
-
"digits": 70,
|
| 514 |
-
"kind": "dense_carry",
|
| 515 |
-
"expected_len": 71,
|
| 516 |
-
"pred_len": 71,
|
| 517 |
-
"think_ok": false,
|
| 518 |
-
"expected_head": "15754532624644617966",
|
| 519 |
-
"pred_head": "95754532624644617966"
|
| 520 |
-
},
|
| 521 |
-
{
|
| 522 |
-
"digits": 70,
|
| 523 |
-
"kind": "dense_carry",
|
| 524 |
-
"expected_len": 71,
|
| 525 |
-
"pred_len": 71,
|
| 526 |
-
"think_ok": false,
|
| 527 |
-
"expected_head": "15424597674457653251",
|
| 528 |
-
"pred_head": "15624597684457653251"
|
| 529 |
-
},
|
| 530 |
-
{
|
| 531 |
-
"digits": 70,
|
| 532 |
-
"kind": "cascade_mix",
|
| 533 |
-
"expected_len": 71,
|
| 534 |
-
"pred_len": 71,
|
| 535 |
-
"think_ok": false,
|
| 536 |
-
"expected_head": "14007494074599980006",
|
| 537 |
-
"pred_head": "13907494074599980006"
|
| 538 |
-
},
|
| 539 |
-
{
|
| 540 |
-
"digits": 70,
|
| 541 |
-
"kind": "cascade_mix",
|
| 542 |
-
"expected_len": 71,
|
| 543 |
-
"pred_len": 71,
|
| 544 |
-
"think_ok": false,
|
| 545 |
-
"expected_head": "17503600000090000000",
|
| 546 |
-
"pred_head": "17503600000080000000"
|
| 547 |
-
},
|
| 548 |
-
{
|
| 549 |
-
"digits": 80,
|
| 550 |
-
"kind": "uniform_random",
|
| 551 |
-
"expected_len": 81,
|
| 552 |
-
"pred_len": 7,
|
| 553 |
-
"think_ok": false,
|
| 554 |
-
"expected_head": "12847570704495513468",
|
| 555 |
-
"pred_head": "NO_STOP"
|
| 556 |
-
},
|
| 557 |
-
{
|
| 558 |
-
"digits": 80,
|
| 559 |
-
"kind": "uniform_random",
|
| 560 |
-
"expected_len": 81,
|
| 561 |
-
"pred_len": 84,
|
| 562 |
-
"think_ok": false,
|
| 563 |
-
"expected_head": "10585471026702913113",
|
| 564 |
-
"pred_head": "50910585470037702913"
|
| 565 |
-
},
|
| 566 |
-
{
|
| 567 |
-
"digits": 80,
|
| 568 |
-
"kind": "uniform_random",
|
| 569 |
-
"expected_len": 81,
|
| 570 |
-
"pred_len": 81,
|
| 571 |
-
"think_ok": false,
|
| 572 |
-
"expected_head": "12278363141943820719",
|
| 573 |
-
"pred_head": "41279373131933820717"
|
| 574 |
-
},
|
| 575 |
-
{
|
| 576 |
-
"digits": 80,
|
| 577 |
-
"kind": "uniform_random",
|
| 578 |
-
"expected_len": 81,
|
| 579 |
-
"pred_len": 82,
|
| 580 |
-
"think_ok": false,
|
| 581 |
-
"expected_head": "11724726771333240441",
|
| 582 |
-
"pred_head": "41182576676137324034"
|
| 583 |
-
},
|
| 584 |
-
{
|
| 585 |
-
"digits": 80,
|
| 586 |
-
"kind": "uniform_random",
|
| 587 |
-
"expected_len": 81,
|
| 588 |
-
"pred_len": 133,
|
| 589 |
-
"think_ok": false,
|
| 590 |
-
"expected_head": "14074311451742727401",
|
| 591 |
-
"pred_head": "40001406331265174282"
|
| 592 |
-
},
|
| 593 |
-
{
|
| 594 |
-
"digits": 80,
|
| 595 |
-
"kind": "uniform_random",
|
| 596 |
-
"expected_len": 81,
|
| 597 |
-
"pred_len": 7,
|
| 598 |
-
"think_ok": false,
|
| 599 |
-
"expected_head": "16952626939497179229",
|
| 600 |
-
"pred_head": "NO_STOP"
|
| 601 |
-
},
|
| 602 |
-
{
|
| 603 |
-
"digits": 80,
|
| 604 |
-
"kind": "uniform_random",
|
| 605 |
-
"expected_len": 80,
|
| 606 |
-
"pred_len": 183,
|
| 607 |
-
"think_ok": false,
|
| 608 |
-
"expected_head": "82549613862831800793",
|
| 609 |
-
"pred_head": "87848613863331800803"
|
| 610 |
-
},
|
| 611 |
-
{
|
| 612 |
-
"digits": 80,
|
| 613 |
-
"kind": "dense_carry",
|
| 614 |
-
"expected_len": 81,
|
| 615 |
-
"pred_len": 81,
|
| 616 |
-
"think_ok": false,
|
| 617 |
-
"expected_head": "13272527565535783883",
|
| 618 |
-
"pred_head": "73272527565635784883"
|
| 619 |
-
},
|
| 620 |
-
{
|
| 621 |
-
"digits": 80,
|
| 622 |
-
"kind": "dense_carry",
|
| 623 |
-
"expected_len": 81,
|
| 624 |
-
"pred_len": 86,
|
| 625 |
-
"think_ok": false,
|
| 626 |
-
"expected_head": "16668493333645678724",
|
| 627 |
-
"pred_head": "74810166685944336456"
|
| 628 |
-
},
|
| 629 |
-
{
|
| 630 |
-
"digits": 80,
|
| 631 |
-
"kind": "cascade_mix",
|
| 632 |
-
"expected_len": 81,
|
| 633 |
-
"pred_len": 173,
|
| 634 |
-
"think_ok": false,
|
| 635 |
-
"expected_head": "10002799999960006628",
|
| 636 |
-
"pred_head": "49002600009959906628"
|
| 637 |
-
}
|
| 638 |
-
],
|
| 639 |
-
"summary_16_50": {
|
| 640 |
-
"total_samples": 240,
|
| 641 |
-
"exact_match": 237,
|
| 642 |
-
"accuracy_pct": 98.75,
|
| 643 |
-
"wilson_95_ci": [
|
| 644 |
-
96.4,
|
| 645 |
-
99.6
|
| 646 |
-
],
|
| 647 |
-
"think_exact": 238,
|
| 648 |
-
"think_acc_pct": 99.17,
|
| 649 |
-
"think_wilson_95_ci": [
|
| 650 |
-
97.0,
|
| 651 |
-
99.8
|
| 652 |
-
]
|
| 653 |
-
},
|
| 654 |
-
"total_elapsed_sec": 792.96
|
| 655 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
random_extrapolation_champion_16_to_100d.json
DELETED
|
@@ -1,610 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"per_length": {
|
| 3 |
-
"16": {
|
| 4 |
-
"digits": 16,
|
| 5 |
-
"n_samples": 30,
|
| 6 |
-
"exact_match": 30,
|
| 7 |
-
"accuracy_pct": 100.0,
|
| 8 |
-
"wilson_95_ci": [
|
| 9 |
-
88.6,
|
| 10 |
-
100.0
|
| 11 |
-
],
|
| 12 |
-
"think_exact": 30,
|
| 13 |
-
"think_acc_pct": 100.0,
|
| 14 |
-
"think_wilson_95_ci": [
|
| 15 |
-
88.6,
|
| 16 |
-
100.0
|
| 17 |
-
],
|
| 18 |
-
"lparen_errors": 0,
|
| 19 |
-
"off_by_one_readout_errors": 0,
|
| 20 |
-
"seq_tokens": 226,
|
| 21 |
-
"elapsed_sec": 27.6,
|
| 22 |
-
"tok_per_sec": 245.6
|
| 23 |
-
},
|
| 24 |
-
"20": {
|
| 25 |
-
"digits": 20,
|
| 26 |
-
"n_samples": 30,
|
| 27 |
-
"exact_match": 30,
|
| 28 |
-
"accuracy_pct": 100.0,
|
| 29 |
-
"wilson_95_ci": [
|
| 30 |
-
88.6,
|
| 31 |
-
100.0
|
| 32 |
-
],
|
| 33 |
-
"think_exact": 30,
|
| 34 |
-
"think_acc_pct": 100.0,
|
| 35 |
-
"think_wilson_95_ci": [
|
| 36 |
-
88.6,
|
| 37 |
-
100.0
|
| 38 |
-
],
|
| 39 |
-
"lparen_errors": 0,
|
| 40 |
-
"off_by_one_readout_errors": 0,
|
| 41 |
-
"seq_tokens": 278,
|
| 42 |
-
"elapsed_sec": 38.68,
|
| 43 |
-
"tok_per_sec": 215.6
|
| 44 |
-
},
|
| 45 |
-
"25": {
|
| 46 |
-
"digits": 25,
|
| 47 |
-
"n_samples": 30,
|
| 48 |
-
"exact_match": 30,
|
| 49 |
-
"accuracy_pct": 100.0,
|
| 50 |
-
"wilson_95_ci": [
|
| 51 |
-
88.6,
|
| 52 |
-
100.0
|
| 53 |
-
],
|
| 54 |
-
"think_exact": 30,
|
| 55 |
-
"think_acc_pct": 100.0,
|
| 56 |
-
"think_wilson_95_ci": [
|
| 57 |
-
88.6,
|
| 58 |
-
100.0
|
| 59 |
-
],
|
| 60 |
-
"lparen_errors": 0,
|
| 61 |
-
"off_by_one_readout_errors": 0,
|
| 62 |
-
"seq_tokens": 343,
|
| 63 |
-
"elapsed_sec": 54.9,
|
| 64 |
-
"tok_per_sec": 187.4
|
| 65 |
-
},
|
| 66 |
-
"30": {
|
| 67 |
-
"digits": 30,
|
| 68 |
-
"n_samples": 30,
|
| 69 |
-
"exact_match": 30,
|
| 70 |
-
"accuracy_pct": 100.0,
|
| 71 |
-
"wilson_95_ci": [
|
| 72 |
-
88.6,
|
| 73 |
-
100.0
|
| 74 |
-
],
|
| 75 |
-
"think_exact": 30,
|
| 76 |
-
"think_acc_pct": 100.0,
|
| 77 |
-
"think_wilson_95_ci": [
|
| 78 |
-
88.6,
|
| 79 |
-
100.0
|
| 80 |
-
],
|
| 81 |
-
"lparen_errors": 0,
|
| 82 |
-
"off_by_one_readout_errors": 0,
|
| 83 |
-
"seq_tokens": 408,
|
| 84 |
-
"elapsed_sec": 79.13,
|
| 85 |
-
"tok_per_sec": 154.7
|
| 86 |
-
},
|
| 87 |
-
"35": {
|
| 88 |
-
"digits": 35,
|
| 89 |
-
"n_samples": 30,
|
| 90 |
-
"exact_match": 30,
|
| 91 |
-
"accuracy_pct": 100.0,
|
| 92 |
-
"wilson_95_ci": [
|
| 93 |
-
88.6,
|
| 94 |
-
100.0
|
| 95 |
-
],
|
| 96 |
-
"think_exact": 30,
|
| 97 |
-
"think_acc_pct": 100.0,
|
| 98 |
-
"think_wilson_95_ci": [
|
| 99 |
-
88.6,
|
| 100 |
-
100.0
|
| 101 |
-
],
|
| 102 |
-
"lparen_errors": 0,
|
| 103 |
-
"off_by_one_readout_errors": 0,
|
| 104 |
-
"seq_tokens": 473,
|
| 105 |
-
"elapsed_sec": 95.72,
|
| 106 |
-
"tok_per_sec": 148.3
|
| 107 |
-
},
|
| 108 |
-
"40": {
|
| 109 |
-
"digits": 40,
|
| 110 |
-
"n_samples": 30,
|
| 111 |
-
"exact_match": 30,
|
| 112 |
-
"accuracy_pct": 100.0,
|
| 113 |
-
"wilson_95_ci": [
|
| 114 |
-
88.6,
|
| 115 |
-
100.0
|
| 116 |
-
],
|
| 117 |
-
"think_exact": 30,
|
| 118 |
-
"think_acc_pct": 100.0,
|
| 119 |
-
"think_wilson_95_ci": [
|
| 120 |
-
88.6,
|
| 121 |
-
100.0
|
| 122 |
-
],
|
| 123 |
-
"lparen_errors": 0,
|
| 124 |
-
"off_by_one_readout_errors": 0,
|
| 125 |
-
"seq_tokens": 538,
|
| 126 |
-
"elapsed_sec": 155.8,
|
| 127 |
-
"tok_per_sec": 103.6
|
| 128 |
-
},
|
| 129 |
-
"45": {
|
| 130 |
-
"digits": 45,
|
| 131 |
-
"n_samples": 30,
|
| 132 |
-
"exact_match": 30,
|
| 133 |
-
"accuracy_pct": 100.0,
|
| 134 |
-
"wilson_95_ci": [
|
| 135 |
-
88.6,
|
| 136 |
-
100.0
|
| 137 |
-
],
|
| 138 |
-
"think_exact": 30,
|
| 139 |
-
"think_acc_pct": 100.0,
|
| 140 |
-
"think_wilson_95_ci": [
|
| 141 |
-
88.6,
|
| 142 |
-
100.0
|
| 143 |
-
],
|
| 144 |
-
"lparen_errors": 0,
|
| 145 |
-
"off_by_one_readout_errors": 0,
|
| 146 |
-
"seq_tokens": 603,
|
| 147 |
-
"elapsed_sec": 182.82,
|
| 148 |
-
"tok_per_sec": 99.0
|
| 149 |
-
},
|
| 150 |
-
"50": {
|
| 151 |
-
"digits": 50,
|
| 152 |
-
"n_samples": 30,
|
| 153 |
-
"exact_match": 30,
|
| 154 |
-
"accuracy_pct": 100.0,
|
| 155 |
-
"wilson_95_ci": [
|
| 156 |
-
88.6,
|
| 157 |
-
100.0
|
| 158 |
-
],
|
| 159 |
-
"think_exact": 30,
|
| 160 |
-
"think_acc_pct": 100.0,
|
| 161 |
-
"think_wilson_95_ci": [
|
| 162 |
-
88.6,
|
| 163 |
-
100.0
|
| 164 |
-
],
|
| 165 |
-
"lparen_errors": 0,
|
| 166 |
-
"off_by_one_readout_errors": 0,
|
| 167 |
-
"seq_tokens": 668,
|
| 168 |
-
"elapsed_sec": 166.06,
|
| 169 |
-
"tok_per_sec": 120.7
|
| 170 |
-
},
|
| 171 |
-
"55": {
|
| 172 |
-
"digits": 55,
|
| 173 |
-
"n_samples": 20,
|
| 174 |
-
"exact_match": 20,
|
| 175 |
-
"accuracy_pct": 100.0,
|
| 176 |
-
"wilson_95_ci": [
|
| 177 |
-
83.9,
|
| 178 |
-
100.0
|
| 179 |
-
],
|
| 180 |
-
"think_exact": 20,
|
| 181 |
-
"think_acc_pct": 100.0,
|
| 182 |
-
"think_wilson_95_ci": [
|
| 183 |
-
83.9,
|
| 184 |
-
100.0
|
| 185 |
-
],
|
| 186 |
-
"lparen_errors": 0,
|
| 187 |
-
"off_by_one_readout_errors": 0,
|
| 188 |
-
"seq_tokens": 733,
|
| 189 |
-
"elapsed_sec": 134.79,
|
| 190 |
-
"tok_per_sec": 108.8
|
| 191 |
-
},
|
| 192 |
-
"60": {
|
| 193 |
-
"digits": 60,
|
| 194 |
-
"n_samples": 20,
|
| 195 |
-
"exact_match": 14,
|
| 196 |
-
"accuracy_pct": 70.0,
|
| 197 |
-
"wilson_95_ci": [
|
| 198 |
-
48.1,
|
| 199 |
-
85.5
|
| 200 |
-
],
|
| 201 |
-
"think_exact": 14,
|
| 202 |
-
"think_acc_pct": 70.0,
|
| 203 |
-
"think_wilson_95_ci": [
|
| 204 |
-
48.1,
|
| 205 |
-
85.5
|
| 206 |
-
],
|
| 207 |
-
"lparen_errors": 1,
|
| 208 |
-
"off_by_one_readout_errors": 0,
|
| 209 |
-
"seq_tokens": 798,
|
| 210 |
-
"elapsed_sec": 150.02,
|
| 211 |
-
"tok_per_sec": 106.4
|
| 212 |
-
},
|
| 213 |
-
"70": {
|
| 214 |
-
"digits": 70,
|
| 215 |
-
"n_samples": 15,
|
| 216 |
-
"exact_match": 6,
|
| 217 |
-
"accuracy_pct": 40.0,
|
| 218 |
-
"wilson_95_ci": [
|
| 219 |
-
19.8,
|
| 220 |
-
64.3
|
| 221 |
-
],
|
| 222 |
-
"think_exact": 6,
|
| 223 |
-
"think_acc_pct": 40.0,
|
| 224 |
-
"think_wilson_95_ci": [
|
| 225 |
-
19.8,
|
| 226 |
-
64.3
|
| 227 |
-
],
|
| 228 |
-
"lparen_errors": 0,
|
| 229 |
-
"off_by_one_readout_errors": 0,
|
| 230 |
-
"seq_tokens": 928,
|
| 231 |
-
"elapsed_sec": 161.99,
|
| 232 |
-
"tok_per_sec": 85.9
|
| 233 |
-
},
|
| 234 |
-
"80": {
|
| 235 |
-
"digits": 80,
|
| 236 |
-
"n_samples": 10,
|
| 237 |
-
"exact_match": 0,
|
| 238 |
-
"accuracy_pct": 0.0,
|
| 239 |
-
"wilson_95_ci": [
|
| 240 |
-
0.0,
|
| 241 |
-
27.8
|
| 242 |
-
],
|
| 243 |
-
"think_exact": 0,
|
| 244 |
-
"think_acc_pct": 0.0,
|
| 245 |
-
"think_wilson_95_ci": [
|
| 246 |
-
0.0,
|
| 247 |
-
27.8
|
| 248 |
-
],
|
| 249 |
-
"lparen_errors": 5,
|
| 250 |
-
"off_by_one_readout_errors": 0,
|
| 251 |
-
"seq_tokens": 1149,
|
| 252 |
-
"elapsed_sec": 154.23,
|
| 253 |
-
"tok_per_sec": 74.5
|
| 254 |
-
},
|
| 255 |
-
"100": {
|
| 256 |
-
"digits": 100,
|
| 257 |
-
"n_samples": 10,
|
| 258 |
-
"exact_match": 0,
|
| 259 |
-
"accuracy_pct": 0.0,
|
| 260 |
-
"wilson_95_ci": [
|
| 261 |
-
0.0,
|
| 262 |
-
27.8
|
| 263 |
-
],
|
| 264 |
-
"think_exact": 0,
|
| 265 |
-
"think_acc_pct": 0.0,
|
| 266 |
-
"think_wilson_95_ci": [
|
| 267 |
-
0.0,
|
| 268 |
-
27.8
|
| 269 |
-
],
|
| 270 |
-
"lparen_errors": 0,
|
| 271 |
-
"off_by_one_readout_errors": 0,
|
| 272 |
-
"seq_tokens": 1429,
|
| 273 |
-
"elapsed_sec": 200.31,
|
| 274 |
-
"tok_per_sec": 71.3
|
| 275 |
-
}
|
| 276 |
-
},
|
| 277 |
-
"failures": [
|
| 278 |
-
{
|
| 279 |
-
"digits": 60,
|
| 280 |
-
"kind": "uniform_random",
|
| 281 |
-
"expected_len": 60,
|
| 282 |
-
"pred_len": 61,
|
| 283 |
-
"think_ok": false,
|
| 284 |
-
"expected_head": "99455736002861930179",
|
| 285 |
-
"pred_head": "10045573600286193017"
|
| 286 |
-
},
|
| 287 |
-
{
|
| 288 |
-
"digits": 60,
|
| 289 |
-
"kind": "uniform_random",
|
| 290 |
-
"expected_len": 61,
|
| 291 |
-
"pred_len": 61,
|
| 292 |
-
"think_ok": false,
|
| 293 |
-
"expected_head": "13771859705436256056",
|
| 294 |
-
"pred_head": "14771859705436256056"
|
| 295 |
-
},
|
| 296 |
-
{
|
| 297 |
-
"digits": 60,
|
| 298 |
-
"kind": "uniform_random",
|
| 299 |
-
"expected_len": 60,
|
| 300 |
-
"pred_len": 60,
|
| 301 |
-
"think_ok": false,
|
| 302 |
-
"expected_head": "93182197277112619803",
|
| 303 |
-
"pred_head": "93182207277112619803"
|
| 304 |
-
},
|
| 305 |
-
{
|
| 306 |
-
"digits": 60,
|
| 307 |
-
"kind": "uniform_random",
|
| 308 |
-
"expected_len": 61,
|
| 309 |
-
"pred_len": 61,
|
| 310 |
-
"think_ok": false,
|
| 311 |
-
"expected_head": "13495686920877645844",
|
| 312 |
-
"pred_head": "13595686920877645844"
|
| 313 |
-
},
|
| 314 |
-
{
|
| 315 |
-
"digits": 60,
|
| 316 |
-
"kind": "uniform_random",
|
| 317 |
-
"expected_len": 60,
|
| 318 |
-
"pred_len": 60,
|
| 319 |
-
"think_ok": false,
|
| 320 |
-
"expected_head": "33303324997839267821",
|
| 321 |
-
"pred_head": "35303324997839267821"
|
| 322 |
-
},
|
| 323 |
-
{
|
| 324 |
-
"digits": 60,
|
| 325 |
-
"kind": "cascade_mix",
|
| 326 |
-
"expected_len": 61,
|
| 327 |
-
"pred_len": 61,
|
| 328 |
-
"think_ok": false,
|
| 329 |
-
"expected_head": "15999869832993500000",
|
| 330 |
-
"pred_head": "16999869832993500000"
|
| 331 |
-
},
|
| 332 |
-
{
|
| 333 |
-
"digits": 70,
|
| 334 |
-
"kind": "uniform_random",
|
| 335 |
-
"expected_len": 71,
|
| 336 |
-
"pred_len": 71,
|
| 337 |
-
"think_ok": false,
|
| 338 |
-
"expected_head": "13709577792207576758",
|
| 339 |
-
"pred_head": "13710577792207576758"
|
| 340 |
-
},
|
| 341 |
-
{
|
| 342 |
-
"digits": 70,
|
| 343 |
-
"kind": "uniform_random",
|
| 344 |
-
"expected_len": 71,
|
| 345 |
-
"pred_len": 71,
|
| 346 |
-
"think_ok": false,
|
| 347 |
-
"expected_head": "18254804996856400492",
|
| 348 |
-
"pred_head": "18254805096856400492"
|
| 349 |
-
},
|
| 350 |
-
{
|
| 351 |
-
"digits": 70,
|
| 352 |
-
"kind": "uniform_random",
|
| 353 |
-
"expected_len": 71,
|
| 354 |
-
"pred_len": 71,
|
| 355 |
-
"think_ok": false,
|
| 356 |
-
"expected_head": "15468076189969829446",
|
| 357 |
-
"pred_head": "15468076190969829446"
|
| 358 |
-
},
|
| 359 |
-
{
|
| 360 |
-
"digits": 70,
|
| 361 |
-
"kind": "uniform_random",
|
| 362 |
-
"expected_len": 71,
|
| 363 |
-
"pred_len": 71,
|
| 364 |
-
"think_ok": false,
|
| 365 |
-
"expected_head": "17132113958771129774",
|
| 366 |
-
"pred_head": "17162113958771129774"
|
| 367 |
-
},
|
| 368 |
-
{
|
| 369 |
-
"digits": 70,
|
| 370 |
-
"kind": "uniform_random",
|
| 371 |
-
"expected_len": 70,
|
| 372 |
-
"pred_len": 70,
|
| 373 |
-
"think_ok": false,
|
| 374 |
-
"expected_head": "81569945829520864708",
|
| 375 |
-
"pred_head": "81669945829520864708"
|
| 376 |
-
},
|
| 377 |
-
{
|
| 378 |
-
"digits": 70,
|
| 379 |
-
"kind": "uniform_random",
|
| 380 |
-
"expected_len": 71,
|
| 381 |
-
"pred_len": 71,
|
| 382 |
-
"think_ok": false,
|
| 383 |
-
"expected_head": "14659845102561298448",
|
| 384 |
-
"pred_head": "14657845102561298448"
|
| 385 |
-
},
|
| 386 |
-
{
|
| 387 |
-
"digits": 70,
|
| 388 |
-
"kind": "uniform_random",
|
| 389 |
-
"expected_len": 71,
|
| 390 |
-
"pred_len": 71,
|
| 391 |
-
"think_ok": false,
|
| 392 |
-
"expected_head": "12532903951038538802",
|
| 393 |
-
"pred_head": "12532903851038538802"
|
| 394 |
-
},
|
| 395 |
-
{
|
| 396 |
-
"digits": 70,
|
| 397 |
-
"kind": "uniform_random",
|
| 398 |
-
"expected_len": 71,
|
| 399 |
-
"pred_len": 71,
|
| 400 |
-
"think_ok": false,
|
| 401 |
-
"expected_head": "14193174229527824431",
|
| 402 |
-
"pred_head": "14193174229827824431"
|
| 403 |
-
},
|
| 404 |
-
{
|
| 405 |
-
"digits": 70,
|
| 406 |
-
"kind": "cascade_mix",
|
| 407 |
-
"expected_len": 71,
|
| 408 |
-
"pred_len": 71,
|
| 409 |
-
"think_ok": false,
|
| 410 |
-
"expected_head": "17503600000090000000",
|
| 411 |
-
"pred_head": "17503600000080000000"
|
| 412 |
-
},
|
| 413 |
-
{
|
| 414 |
-
"digits": 80,
|
| 415 |
-
"kind": "uniform_random",
|
| 416 |
-
"expected_len": 81,
|
| 417 |
-
"pred_len": 87,
|
| 418 |
-
"think_ok": false,
|
| 419 |
-
"expected_head": "12847570704495513468",
|
| 420 |
-
"pred_head": "00000012847570704495"
|
| 421 |
-
},
|
| 422 |
-
{
|
| 423 |
-
"digits": 80,
|
| 424 |
-
"kind": "uniform_random",
|
| 425 |
-
"expected_len": 81,
|
| 426 |
-
"pred_len": 63,
|
| 427 |
-
"think_ok": false,
|
| 428 |
-
"expected_head": "10585471026702913113",
|
| 429 |
-
"pred_head": "00000000000105854710"
|
| 430 |
-
},
|
| 431 |
-
{
|
| 432 |
-
"digits": 80,
|
| 433 |
-
"kind": "uniform_random",
|
| 434 |
-
"expected_len": 81,
|
| 435 |
-
"pred_len": 7,
|
| 436 |
-
"think_ok": false,
|
| 437 |
-
"expected_head": "12278363141943820719",
|
| 438 |
-
"pred_head": "NO_STOP"
|
| 439 |
-
},
|
| 440 |
-
{
|
| 441 |
-
"digits": 80,
|
| 442 |
-
"kind": "uniform_random",
|
| 443 |
-
"expected_len": 81,
|
| 444 |
-
"pred_len": 88,
|
| 445 |
-
"think_ok": false,
|
| 446 |
-
"expected_head": "11724726771333240441",
|
| 447 |
-
"pred_head": "00000001172472676133"
|
| 448 |
-
},
|
| 449 |
-
{
|
| 450 |
-
"digits": 80,
|
| 451 |
-
"kind": "uniform_random",
|
| 452 |
-
"expected_len": 81,
|
| 453 |
-
"pred_len": 7,
|
| 454 |
-
"think_ok": false,
|
| 455 |
-
"expected_head": "14074311451742727401",
|
| 456 |
-
"pred_head": "NO_STOP"
|
| 457 |
-
},
|
| 458 |
-
{
|
| 459 |
-
"digits": 80,
|
| 460 |
-
"kind": "uniform_random",
|
| 461 |
-
"expected_len": 81,
|
| 462 |
-
"pred_len": 86,
|
| 463 |
-
"think_ok": false,
|
| 464 |
-
"expected_head": "16952626939497179229",
|
| 465 |
-
"pred_head": "00000169526269394861"
|
| 466 |
-
},
|
| 467 |
-
{
|
| 468 |
-
"digits": 80,
|
| 469 |
-
"kind": "uniform_random",
|
| 470 |
-
"expected_len": 80,
|
| 471 |
-
"pred_len": 86,
|
| 472 |
-
"think_ok": false,
|
| 473 |
-
"expected_head": "82549613862831800793",
|
| 474 |
-
"pred_head": "80000082849613863331"
|
| 475 |
-
},
|
| 476 |
-
{
|
| 477 |
-
"digits": 80,
|
| 478 |
-
"kind": "dense_carry",
|
| 479 |
-
"expected_len": 81,
|
| 480 |
-
"pred_len": 7,
|
| 481 |
-
"think_ok": false,
|
| 482 |
-
"expected_head": "13272527565535783883",
|
| 483 |
-
"pred_head": "NO_STOP"
|
| 484 |
-
},
|
| 485 |
-
{
|
| 486 |
-
"digits": 80,
|
| 487 |
-
"kind": "dense_carry",
|
| 488 |
-
"expected_len": 81,
|
| 489 |
-
"pred_len": 7,
|
| 490 |
-
"think_ok": false,
|
| 491 |
-
"expected_head": "16668493333645678724",
|
| 492 |
-
"pred_head": "NO_STOP"
|
| 493 |
-
},
|
| 494 |
-
{
|
| 495 |
-
"digits": 80,
|
| 496 |
-
"kind": "cascade_mix",
|
| 497 |
-
"expected_len": 81,
|
| 498 |
-
"pred_len": 7,
|
| 499 |
-
"think_ok": false,
|
| 500 |
-
"expected_head": "10002799999960006628",
|
| 501 |
-
"pred_head": "NO_STOP"
|
| 502 |
-
},
|
| 503 |
-
{
|
| 504 |
-
"digits": 100,
|
| 505 |
-
"kind": "uniform_random",
|
| 506 |
-
"expected_len": 101,
|
| 507 |
-
"pred_len": 7,
|
| 508 |
-
"think_ok": false,
|
| 509 |
-
"expected_head": "11288401813144915928",
|
| 510 |
-
"pred_head": "NO_STOP"
|
| 511 |
-
},
|
| 512 |
-
{
|
| 513 |
-
"digits": 100,
|
| 514 |
-
"kind": "uniform_random",
|
| 515 |
-
"expected_len": 101,
|
| 516 |
-
"pred_len": 7,
|
| 517 |
-
"think_ok": false,
|
| 518 |
-
"expected_head": "15225010932074486266",
|
| 519 |
-
"pred_head": "NO_STOP"
|
| 520 |
-
},
|
| 521 |
-
{
|
| 522 |
-
"digits": 100,
|
| 523 |
-
"kind": "uniform_random",
|
| 524 |
-
"expected_len": 101,
|
| 525 |
-
"pred_len": 7,
|
| 526 |
-
"think_ok": false,
|
| 527 |
-
"expected_head": "17203428879913620849",
|
| 528 |
-
"pred_head": "NO_STOP"
|
| 529 |
-
},
|
| 530 |
-
{
|
| 531 |
-
"digits": 100,
|
| 532 |
-
"kind": "uniform_random",
|
| 533 |
-
"expected_len": 101,
|
| 534 |
-
"pred_len": 7,
|
| 535 |
-
"think_ok": false,
|
| 536 |
-
"expected_head": "13949786348913299912",
|
| 537 |
-
"pred_head": "NO_STOP"
|
| 538 |
-
},
|
| 539 |
-
{
|
| 540 |
-
"digits": 100,
|
| 541 |
-
"kind": "uniform_random",
|
| 542 |
-
"expected_len": 101,
|
| 543 |
-
"pred_len": 7,
|
| 544 |
-
"think_ok": false,
|
| 545 |
-
"expected_head": "12975883843361249782",
|
| 546 |
-
"pred_head": "NO_STOP"
|
| 547 |
-
},
|
| 548 |
-
{
|
| 549 |
-
"digits": 100,
|
| 550 |
-
"kind": "uniform_random",
|
| 551 |
-
"expected_len": 100,
|
| 552 |
-
"pred_len": 7,
|
| 553 |
-
"think_ok": false,
|
| 554 |
-
"expected_head": "45147010533867848323",
|
| 555 |
-
"pred_head": "NO_STOP"
|
| 556 |
-
},
|
| 557 |
-
{
|
| 558 |
-
"digits": 100,
|
| 559 |
-
"kind": "uniform_random",
|
| 560 |
-
"expected_len": 101,
|
| 561 |
-
"pred_len": 7,
|
| 562 |
-
"think_ok": false,
|
| 563 |
-
"expected_head": "13735571010375678962",
|
| 564 |
-
"pred_head": "NO_STOP"
|
| 565 |
-
},
|
| 566 |
-
{
|
| 567 |
-
"digits": 100,
|
| 568 |
-
"kind": "dense_carry",
|
| 569 |
-
"expected_len": 101,
|
| 570 |
-
"pred_len": 7,
|
| 571 |
-
"think_ok": false,
|
| 572 |
-
"expected_head": "16553554674857513255",
|
| 573 |
-
"pred_head": "NO_STOP"
|
| 574 |
-
},
|
| 575 |
-
{
|
| 576 |
-
"digits": 100,
|
| 577 |
-
"kind": "dense_carry",
|
| 578 |
-
"expected_len": 101,
|
| 579 |
-
"pred_len": 7,
|
| 580 |
-
"think_ok": false,
|
| 581 |
-
"expected_head": "18253466387437646273",
|
| 582 |
-
"pred_head": "NO_STOP"
|
| 583 |
-
},
|
| 584 |
-
{
|
| 585 |
-
"digits": 100,
|
| 586 |
-
"kind": "cascade_mix",
|
| 587 |
-
"expected_len": 101,
|
| 588 |
-
"pred_len": 7,
|
| 589 |
-
"think_ok": false,
|
| 590 |
-
"expected_head": "17200006000000004996",
|
| 591 |
-
"pred_head": "NO_STOP"
|
| 592 |
-
}
|
| 593 |
-
],
|
| 594 |
-
"summary_16_50": {
|
| 595 |
-
"total_samples": 240,
|
| 596 |
-
"exact_match": 240,
|
| 597 |
-
"accuracy_pct": 100.0,
|
| 598 |
-
"wilson_95_ci": [
|
| 599 |
-
98.4,
|
| 600 |
-
100.0
|
| 601 |
-
],
|
| 602 |
-
"think_exact": 240,
|
| 603 |
-
"think_acc_pct": 100.0,
|
| 604 |
-
"think_wilson_95_ci": [
|
| 605 |
-
98.4,
|
| 606 |
-
100.0
|
| 607 |
-
]
|
| 608 |
-
},
|
| 609 |
-
"total_elapsed_sec": 1602.17
|
| 610 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
requirements.txt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
torch>=2.2.0
|
| 2 |
+
safetensors>=0.4.0
|
| 3 |
+
huggingface_hub>=0.20.0
|