Neithanvarvas commited on
Commit
dde517d
·
verified ·
1 Parent(s): 6d78b43

[UPDATE] Restructure release: safetensors weights, cbt package, chat demo, clean model card

Browse files
Files changed (30) hide show
  1. .gitattributes +1 -0
  2. README.md +118 -50
  3. cbt/__init__.py +97 -0
  4. model.py → cbt/model.py +1 -7
  5. tokenizer.py → cbt/tokenizer.py +0 -0
  6. cbt/tools.py +92 -0
  7. cbt_1d_twin_arithmetic_20_70d.json +0 -224
  8. champion_v3_arithmetic.json +0 -224
  9. champion_v3_arithmetic_4_16d.json +0 -108
  10. chat.py +62 -0
  11. config.json +7 -21
  12. dataset.py +0 -1468
  13. equalized_15d_1024_and_ruoss_sweep.json +0 -890
  14. cbt_2d_arithmetic_20_70d.json → evaluation/cbt_2d/addition_20-70_digits.json +0 -0
  15. cbt_2d_arithmetic_4_16d.json → evaluation/cbt_2d/addition_4-16_digits.json +0 -0
  16. cbt_2d_ood.json → evaluation/cbt_2d/tools_out_of_distribution.json +0 -0
  17. cbt_2d_phrasing.json → evaluation/cbt_2d/tools_rephrased.json +0 -0
  18. cbt_2d_unseen_copy.json → evaluation/cbt_2d/tools_unseen_entities.json +0 -0
  19. cbt1d_v5_arithmetic.json → evaluation/twin_1d/addition_20-70_digits.json +0 -0
  20. cbt_1d_twin_arithmetic_4_16d.json → evaluation/twin_1d/addition_4-16_digits.json +0 -0
  21. champion_v3_ood.json → evaluation/twin_1d/tools_out_of_distribution.json +1 -1
  22. champion_v3_phrasing.json → evaluation/twin_1d/tools_rephrased.json +7 -7
  23. champion_v3_unseen_ood.json → evaluation/twin_1d/tools_unseen_entities.json +10 -10
  24. interactive.py +0 -329
  25. champion_final.pt → model.safetensors +2 -2
  26. RESEARCH_PAPER.md → paper/CBT-18.95M.md +81 -39
  27. RESEARCH_PAPER.pdf → paper/CBT-18.95M.pdf +2 -2
  28. random_extrapolation_16_to_100d.json +0 -655
  29. random_extrapolation_champion_16_to_100d.json +0 -610
  30. requirements.txt +3 -0
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  RESEARCH_PAPER.pdf filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  RESEARCH_PAPER.pdf filter=lfs diff=lfs merge=lfs -text
37
+ paper/CBT-18.95M.pdf filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -2,93 +2,161 @@
2
  language:
3
  - en
4
  license: mit
 
 
5
  tags:
6
  - arithmetic
7
  - length-generalization
8
- - recurrent-transformers
 
9
  - 2d-rope
10
  - tool-use
11
- - function-calling
12
- - reasoning
13
  datasets:
14
  - synthetic
15
  metrics:
16
  - accuracy
17
  ---
18
 
19
- # 2D Embeddings for Length Generalization in Character-Based Transformers (CBT)
20
 
21
- - **GitHub Repository**: [https://github.com/NeithanT/CBT](https://github.com/NeithanT/CBT)
22
- - **Research Paper**: [RESEARCH_PAPER.md](https://github.com/NeithanT/CBT/blob/main/RESEARCH_PAPER.md)
23
 
24
- ## 1. Model Overview
25
 
26
- **Character-Based Transformer (CBT-18.95M)** is an 18.94M-parameter character-level Transformer (111-token vocabulary) whose tokens carry a rule-assigned two-dimensional coordinate $(p, \tau)$ used by **Decoupled 2D Rotary Position Embeddings (2D-RoPE)**, with weight-tied recurrent depth ($R = 3$). This release (`champion_final.pt`) is the model after supervised fine-tuning: pretraining, search-copy and all-tools fine-tuning, an intermediate 12,000-step round, then a final 23,927-step round on a decontaminated online stream (35% search copy, 20% arithmetic replay of 1-25 digits, the rest multi-tool and chat). Coordinate shift masking uses $k=45$, so $p_{\max} = 25 + 45 = 70$.
27
 
28
- Results (greedy decoding; Wilson 95% CIs are in the paper and in `results/`):
29
- - **Addition**, 30 random problems per length (70% uniform, 15% dense carry, 15% cascades), scored as teacher-forced greedy agreement with the gold scratchpad and answer: **150/150** at 4-16 digits, **60/60** at 20 and 25 digits (seen), **210/210** at 26-55 digits (unseen), **25/30** at 60 and **19/30** at 70 digits. Pooled over the unseen 26-70 digits: **254/270**.
30
- - **Tool use**: **26/26** exact calls on a 26-prompt phrasing suite, **18/18** on 18 out-of-distribution prompts, and **30/30** exact copies of entities absent from the training corpus (unseen-30). The final fine-tuning data was broadened after the intermediate model's failures on the first two suites were seen, so the unseen-30 set is the cleaner evidence.
31
- - **1D-RoPE control**: a twin with the same architecture, data and step count but standard 1D-RoPE scores **0/270** at 26-70 digits and **73/74** exact tool calls. The 2D coordinates account for the arithmetic extrapolation, not for the tool-use results.
32
- - **Baselines**: standard 1D-RoPE models trained on 1-12 digit sums score 0/30 at every length from 14 to 35 digits (character and BPE vocabularies; looped, hierarchical and hybrid variants too).
33
-
34
- All results are single training runs. Details, limitations and the failure analysis are in the paper.
35
 
36
  ---
37
 
38
- ## 2. Model Specifications
39
 
40
- | Parameter | Specification |
41
- | :--- | :--- |
42
- | **Total Parameters** | 18,944,512 |
43
- | **Non-Embedding Parameters** | 18,887,680 |
44
- | **Physical Layers (L)** | 6 |
45
- | **Recurrent Passes (R)** | 3 (effective depth 18) |
46
- | **Hidden Dimension** | 512 |
47
- | **Attention Heads** | 8 (head dimension 64) |
48
- | **Context** | 1024 tokens |
49
- | **Positional Embedding** | Decoupled 2D-RoPE (half of each head rotated by place $p$, half by step $\tau$; rule-assigned) |
50
- | **Coordinate Shift Masking** | $k = 45$ |
51
- | **Vocabulary** | 111 (14 control tokens + 97 characters) |
52
- | **Weight Tying** | input embedding = output projection |
53
 
54
- ---
 
 
 
 
 
 
 
 
 
 
55
 
56
- ## 3. Usage
57
 
58
- The file is a `torch.save` dictionary with the config and `model_state_dict`.
59
 
60
  ```python
61
- import torch
62
- from model import ScaledGPT
63
- from tokenizer import ScaledTokenizer
64
-
65
- ckpt = torch.load("champion_final.pt", map_location="cpu", weights_only=False)
66
- tokenizer = ScaledTokenizer()
67
- model = ScaledGPT(ckpt["config"], tokenizer=tokenizer)
68
- model.load_state_dict(ckpt["model_state_dict"])
69
- model.eval()
70
  ```
71
 
72
- Interactive CLI:
73
 
74
- ```bash
75
- python interactive.py --ckpt champion_final.pt
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
76
  ```
77
 
 
 
78
  ---
79
 
80
- ## 4. Files
81
 
82
- `champion_final.pt`, source files for stand-alone execution, `RESEARCH_PAPER.md`, and the evaluation results in JSON.
 
 
83
 
84
  ---
85
 
86
- ## 5. Citation
 
 
 
 
 
 
 
 
 
 
 
 
 
87
 
88
  ```bibtex
89
  @article{vargas2026cbt,
90
- title={2D Embeddings for Length Generalization in Character-Based Transformers (CBT)},
91
- author={Vargas Vargas, Neithan},
92
- year={2026}
93
  }
94
  ```
 
2
  language:
3
  - en
4
  license: mit
5
+ library_name: pytorch
6
+ pipeline_tag: text-generation
7
  tags:
8
  - arithmetic
9
  - length-generalization
10
+ - character-level
11
+ - rope
12
  - 2d-rope
13
  - tool-use
 
 
14
  datasets:
15
  - synthetic
16
  metrics:
17
  - accuracy
18
  ---
19
 
20
+ # CBT-18.95M
21
 
22
+ **A 19M-parameter character-level Transformer that adds numbers longer than any it was trained on.**
 
23
 
24
+ CBT (Character-Based Transformer) gives every token a two-dimensional position: *which column* it belongs to and *which step* inside that column. With that layout, a model trained on sums of up to 25 digits keeps adding correctly at 26, 40 and 55 digits. An identical twin with ordinary 1D positions gets every one of those wrong.
25
 
26
+ The same model also does simple tool calling (weather, search, calculator, unit conversion, ...) in a small chat format.
27
 
28
+ [Paper (PDF)](paper/CBT-18.95M.pdf) · [Paper (Markdown)](paper/CBT-18.95M.md) · [Code on GitHub](https://github.com/NeithanT/CBT)
 
 
 
 
 
 
29
 
30
  ---
31
 
32
+ ## Try it
33
 
34
+ ```bash
35
+ git clone https://huggingface.co/Neithanvarvas/CBT && cd CBT
36
+ pip install -r requirements.txt
37
+ python chat.py
38
+ ```
 
 
 
 
 
 
 
 
39
 
40
+ ```text
41
+ > 123456789+987654321
42
+ scratchpad : (0+9+1)=10(1+8+2)=11(1+7+3)=11(1+6+4)=11 ... (1+0+0)=01
43
+ answer : 1111111110 [PASS]
44
+
45
+ > What is the weather in Tokyo?
46
+ think : The user requests current weather information. I will query the get_weather tool.
47
+ call : get_weather(city="Tokyo")
48
+ result : {"temp": 22, "condition": "Sunny", "city": "Tokyo"}
49
+ answer : The weather in Tokyo is currently Sunny with a temperature of 22 degrees Celsius.
50
+ ```
51
 
52
+ Runs on a CPU; no GPU needed. Pass `--online` to let the search tool query Wikipedia, or `--raw` to see the token stream.
53
 
54
+ ### From Python
55
 
56
  ```python
57
+ from cbt import load, add, chat
58
+
59
+ model, tok = load() # uses this folder, or downloads from the Hub if run elsewhere
60
+
61
+ add(model, tok, 31415926535897932384626433832795, 27182818284590452353602874713527)["answer"]
62
+ # '58598744820488384738229308546322'
63
+
64
+ chat(model, tok, "Convert 10 km to miles")["answer"]
 
65
  ```
66
 
67
+ `chat` runs one user turn and executes at most one tool call. Tools are offline stand-ins in [`cbt/tools.py`](cbt/tools.py); pass `tool_executor=your_function` to connect real ones.
68
 
69
+ ---
70
+
71
+ ## Results
72
+
73
+ Greedy decoding, 30 random problems per length. Training covered sums of 1 to 25 digits.
74
+
75
+ ### Addition: CBT vs. its 1D-RoPE twin
76
+
77
+ The twin has the same architecture, data and number of training steps. Only the position encoding differs.
78
+
79
+ | Operand digits | Seen in training? | CBT (2D) | 1D twin |
80
+ | :--- | :---: | :---: | :---: |
81
+ | 4, 8, 12, 14, 16 | yes | 150 / 150 | 149 / 150 |
82
+ | 20, 25 | yes | 60 / 60 | 57 / 60 |
83
+ | 26 | no | **30 / 30** | 0 / 30 |
84
+ | 30, 35, 40, 45, 50, 55 | no | **180 / 180** | 0 / 180 |
85
+ | 60 | no | 25 / 30 | 0 / 30 |
86
+ | 70 | no | 19 / 30 | 0 / 30 |
87
+ | **All unseen (26-70)** | no | **254 / 270** | **0 / 270** |
88
+
89
+ ### Tool use: exact tool and arguments
90
+
91
+ | Test set | Prompts | CBT (2D) | 1D twin |
92
+ | :--- | :---: | :---: | :---: |
93
+ | Rephrased requests | 26 | 26 | 25 |
94
+ | Out-of-distribution entities | 18 | 18 | 18 |
95
+ | Entities never seen in training | 30 | 30 | 30 |
96
+
97
+ Both models do equally well here, so tool use comes from the training data, not from the 2D layout. The fine-tuning data was broadened after an earlier model failed the first two sets, so the unseen-entity set is the fairer test.
98
+
99
+ All numbers come from single training runs with small test sets. Raw per-problem results are in [`evaluation/`](evaluation/); limitations are discussed in the paper.
100
+
101
+ ---
102
+
103
+ ## How it works
104
+
105
+ Each token gets a coordinate (p, tau):
106
+
107
+ - **Arithmetic:** p is the decimal place (units, tens, ...) and tau is the step within that column of the scratchpad.
108
+ - **Chat:** p is the dialogue segment (user, think, call, result, answer) and tau is the character position inside it.
109
+
110
+ Half of each attention head is rotated by p and half by tau (Decoupled 2D-RoPE). Aligning digits of the same place becomes one fixed relation, "same p", which does not change as numbers get longer, so the model learns it once. During training, p is randomly shifted by up to 45, so the model has seen place values up to 70.
111
+
112
+ | | |
113
+ | :--- | :--- |
114
+ | Parameters | 18.94M (18.89M non-embedding) |
115
+ | Layers | 6, each applied 3 times with shared weights (effective depth 18) |
116
+ | Width / heads | 512 / 8 (head dim 64) |
117
+ | Context | 1024 characters |
118
+ | Vocabulary | 111: 97 ASCII characters (letters, digits, punctuation, space, tab, newline) + 14 control tags |
119
+ | Positions | Decoupled 2D-RoPE, coordinate shift k = 45 |
120
+ | Training | Pretraining, then several supervised fine-tuning rounds (23,927 steps in the last) |
121
+
122
+ ### Prompt formats
123
+
124
+ ```text
125
+ Addition: <BOS>123+456=<THINK>(0+3+6)=09(0+2+5)=07(0+1+4)=05</THINK>=579<EOS>
126
+ Chat: <BOS><USER>question</USER><THINK>...</THINK><CALL>tool(arg="...")</CALL><RESULT>{json}</RESULT><ANSWER>...</ANSWER>
127
  ```
128
 
129
+ Scratchpad steps read `(carry + digit_a + digit_b) = carry_out digit`, from the units column upward.
130
+
131
  ---
132
 
133
+ ## Limitations
134
 
135
+ - Addition of non-negative integers is the only arithmetic it was trained on. Accuracy drops beyond 55 digits, and the 1024-character context caps sums at roughly 75 digits.
136
+ - The chat ability is narrow: it was trained on synthetic tool-calling dialogues with a fixed set of tools. It is not a general assistant and has little world knowledge.
137
+ - Character-level, English-only, ASCII-only. Other characters become `<UNK>`.
138
 
139
  ---
140
 
141
+ ## Files
142
+
143
+ ```text
144
+ README.md this card
145
+ config.json model hyperparameters
146
+ model.safetensors weights
147
+ cbt/ model, tokenizer, tool stubs, load/add/chat helpers
148
+ chat.py terminal demo
149
+ requirements.txt
150
+ paper/ the paper (PDF and Markdown)
151
+ evaluation/ per-problem results behind the tables above
152
+ ```
153
+
154
+ ## Citation
155
 
156
  ```bibtex
157
  @article{vargas2026cbt,
158
+ title = {From Digits to Characters: Two-Dimensional Rotary Coordinates for Length Generalization in a Compact Character-Level Transformer},
159
+ author = {Vargas Vargas, Neithan},
160
+ year = {2026}
161
  }
162
  ```
cbt/__init__.py ADDED
@@ -0,0 +1,97 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """CBT-18.95M: character-level Transformer with Decoupled 2D-RoPE.
2
+
3
+ Quick start:
4
+ from cbt import load, add, chat
5
+ model, tok = load() # local folder or the Hugging Face repo
6
+ add(model, tok, 123456789, 987654321) # {'answer': '1111111110', 'correct': True, ...}
7
+ chat(model, tok, "What is the weather in Tokyo?")
8
+ """
9
+
10
+ import json
11
+ import os
12
+ from dataclasses import fields
13
+ from typing import Any, Callable, Dict, Optional, Tuple, Union
14
+
15
+ import torch
16
+
17
+ from .model import ScaledGPT as CBT, ScaledGPTConfig as CBTConfig
18
+ from .tokenizer import ScaledTokenizer as CBTTokenizer
19
+ from .tools import execute_tool
20
+
21
+ __all__ = ["CBT", "CBTConfig", "CBTTokenizer", "load", "add", "chat", "execute_tool"]
22
+
23
+ REPO_ID = "Neithanvarvas/CBT"
24
+ _PKG_DIR = os.path.dirname(os.path.abspath(__file__))
25
+
26
+
27
+ def _resolve(source: Optional[str], filename: str) -> str:
28
+ """Finds `filename` in a local folder (default: the repo root next to this package) or downloads it."""
29
+ folder = source or os.path.dirname(_PKG_DIR)
30
+ local = os.path.join(folder, filename)
31
+ if os.path.isfile(local):
32
+ return local
33
+ from huggingface_hub import hf_hub_download
34
+ return hf_hub_download(repo_id=source if source and not os.path.isdir(source) else REPO_ID, filename=filename)
35
+
36
+
37
+ def load(source: Optional[str] = None, device: Optional[Union[str, torch.device]] = None) -> Tuple[CBT, CBTTokenizer]:
38
+ """Loads model and tokenizer from a local folder or a Hugging Face repo id; returns (model, tokenizer) in eval mode."""
39
+ from safetensors.torch import load_file
40
+
41
+ with open(_resolve(source, "config.json"), encoding="utf-8") as f:
42
+ raw: Dict[str, Any] = json.load(f)
43
+ config = CBTConfig(**{k.name: raw[k.name] for k in fields(CBTConfig) if k.name in raw})
44
+
45
+ device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
46
+ tok = CBTTokenizer()
47
+ model = CBT(config, tokenizer=tok)
48
+ model.load_state_dict(load_file(_resolve(source, "model.safetensors")))
49
+ return model.to(device).eval(), tok
50
+
51
+
52
+ @torch.no_grad()
53
+ def _generate(model: CBT, ids: torch.Tensor, max_new_tokens: int, stop: list) -> torch.Tensor:
54
+ return model.generate(ids, max_new_tokens=max_new_tokens, temperature=0.0, stop_tokens=stop)
55
+
56
+
57
+ def _encode(model: CBT, tok: CBTTokenizer, text: str, bos: bool = True) -> torch.Tensor:
58
+ device = next(model.parameters()).device
59
+ return torch.tensor([tok.encode(text, add_bos=bos, add_eos=False)], dtype=torch.long, device=device)
60
+
61
+
62
+ def add(model: CBT, tok: CBTTokenizer, a: Union[int, str], b: Union[int, str]) -> Dict[str, Any]:
63
+ """Greedy-decodes a + b with the column scratchpad. Trained on 1-25 digit operands."""
64
+ a, b = str(a).strip(), str(b).strip()
65
+ if not (a.isdigit() and b.isdigit()):
66
+ raise ValueError("operands must be non-negative integers")
67
+ budget = 12 * (max(len(a), len(b)) + 2) + 8
68
+ out = _generate(model, _encode(model, tok, f"{a}+{b}=<THINK>"), budget, [tok.eos_id])
69
+ raw = tok.decode(out[0].tolist())
70
+ think, _, tail = raw.partition("<THINK>")[2].partition("</THINK>")
71
+ answer = tail.lstrip("=").replace("<EOS>", "").strip()
72
+ return {"answer": answer, "scratchpad": think, "correct": answer == str(int(a) + int(b)), "raw": raw}
73
+
74
+
75
+ def chat(
76
+ model: CBT,
77
+ tok: CBTTokenizer,
78
+ message: str,
79
+ tool_executor: Optional[Callable[[str], str]] = execute_tool,
80
+ max_new_tokens: int = 256,
81
+ ) -> Dict[str, Any]:
82
+ """Runs one user turn: <THINK> reasoning, optional <CALL> executed by `tool_executor`, then <ANSWER>."""
83
+ ids = _encode(model, tok, f"<USER>{message}</USER><THINK>")
84
+ out = _generate(model, ids, max_new_tokens, [tok.call_e_id, tok.answer_e_id, tok.eos_id])
85
+ raw = tok.decode(out[0].tolist())
86
+
87
+ if "</CALL>" in raw and tool_executor is not None:
88
+ call = raw.split("<CALL>")[-1].split("</CALL>")[0]
89
+ cont = torch.cat([out, _encode(model, tok, f"<RESULT>{tool_executor(call)}</RESULT>", bos=False)], dim=1)
90
+ out = _generate(model, cont, max_new_tokens, [tok.answer_e_id, tok.eos_id])
91
+ raw = tok.decode(out[0].tolist())
92
+
93
+ def span(tag: str) -> Optional[str]:
94
+ s, e = f"<{tag}>", f"</{tag}>"
95
+ return raw.split(s, 1)[1].split(e, 1)[0].strip() if s in raw and e in raw else None
96
+
97
+ return {"think": span("THINK"), "call": span("CALL"), "result": span("RESULT"), "answer": span("ANSWER"), "raw": raw}
model.py → cbt/model.py RENAMED
@@ -22,13 +22,7 @@ import torch
22
  import torch.nn as nn
23
  import torch.nn.functional as F
24
 
25
- try:
26
- from champion_model.scaled_t4.tokenizer import ScaledTokenizer
27
- except ImportError:
28
- try:
29
- from tokenizer import ScaledTokenizer
30
- except ImportError:
31
- from champion_model.unified_200k.tokenizer import UnifiedTokenizer as ScaledTokenizer
32
 
33
 
34
  @dataclass
 
22
  import torch.nn as nn
23
  import torch.nn.functional as F
24
 
25
+ from .tokenizer import ScaledTokenizer
 
 
 
 
 
 
26
 
27
 
28
  @dataclass
tokenizer.py → cbt/tokenizer.py RENAMED
File without changes
cbt/tools.py ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Offline stand-ins for the tools CBT was fine-tuned to call. Replace `execute_tool` with real backends as needed."""
2
+
3
+ import ast
4
+ import json
5
+ import operator
6
+ import re
7
+ from typing import Any, Dict
8
+
9
+ _OPS = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv,
10
+ ast.USub: operator.neg, ast.UAdd: operator.pos}
11
+
12
+ _KNOWLEDGE: Dict[str, str] = {
13
+ "tomato": "the edible berry of the plant Solanum lycopersicum, commonly prepared as a culinary vegetable.",
14
+ "apple": "a round, edible fruit produced by an apple tree (Malus domestica), cultivated worldwide.",
15
+ "potato": "a starchy, tuberous root crop of the nightshade family Solanaceae.",
16
+ "coffee": "a brewed drink prepared from roasted coffee beans of flowering Coffea species.",
17
+ "banana": "an elongated, edible fruit produced by several kinds of large herbaceous plants in the genus Musa.",
18
+ "gravity": "a fundamental interaction causing mutual attraction between all things with mass or energy.",
19
+ "photosynthesis": "a process used by plants to convert light energy into chemical energy stored in glucose.",
20
+ "atom": "the basic unit of a chemical element, consisting of a dense nucleus surrounded by electrons.",
21
+ "speed of light": "approximately 299,792 kilometers per second in a vacuum.",
22
+ "mount everest": "Earth's highest mountain above sea level, located in the Himalayas.",
23
+ }
24
+
25
+ _UNITS = {("km", "miles"): 0.621371, ("miles", "km"): 1.60934, ("kg", "lbs"): 2.20462, ("lbs", "kg"): 0.453592}
26
+
27
+
28
+ def _calc(node: ast.AST) -> Any:
29
+ if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
30
+ return node.value
31
+ if isinstance(node, ast.BinOp) and type(node.op) in _OPS:
32
+ return _OPS[type(node.op)](_calc(node.left), _calc(node.right))
33
+ if isinstance(node, ast.UnaryOp) and type(node.op) in _OPS:
34
+ return _OPS[type(node.op)](_calc(node.operand))
35
+ raise ValueError("unsupported expression")
36
+
37
+
38
+ def _search(query: str, online: bool) -> Dict[str, str]:
39
+ q = query.lower().rstrip("s")
40
+ for key, text in _KNOWLEDGE.items():
41
+ if key in q or q in key:
42
+ return {"snippet": text}
43
+ if online:
44
+ try:
45
+ import urllib.parse
46
+ import urllib.request
47
+ url = "https://en.wikipedia.org/api/rest_v1/page/summary/" + urllib.parse.quote(query.replace(" ", "_"))
48
+ req = urllib.request.Request(url, headers={"User-Agent": "CBT-demo/1.0"})
49
+ with urllib.request.urlopen(req, timeout=3) as resp:
50
+ extract = json.loads(resp.read().decode("utf-8")).get("extract")
51
+ if extract:
52
+ return {"snippet": extract[:220]}
53
+ except Exception:
54
+ pass
55
+ return {"snippet": f"{query} is a topic in verified knowledge sources."}
56
+
57
+
58
+ def execute_tool(call: str, online: bool = False) -> str:
59
+ """Returns a JSON string for a call such as `get_weather(city="Tokyo")`."""
60
+ name = call.split("(", 1)[0].strip()
61
+ args = dict(re.findall(r'(\w+)="([^"]*)"', call))
62
+
63
+ if name == "calculator":
64
+ try:
65
+ return json.dumps({"result": _calc(ast.parse(args.get("expr", ""), mode="eval").body)})
66
+ except ZeroDivisionError:
67
+ return json.dumps({"error": "ZeroDivisionError: division by zero"})
68
+ except Exception as e:
69
+ return json.dumps({"error": str(e)})
70
+ if name == "get_weather":
71
+ city = args.get("city", "")
72
+ if city.lower() in ("atlantis", "narnia"):
73
+ return json.dumps({"status": "error", "code": 404, "message": f"City '{city}' not found"})
74
+ return json.dumps({"temp": 22, "condition": "Sunny", "city": city})
75
+ if name == "get_time":
76
+ return json.dumps({"time": "14:30", "timezone": args.get("timezone", "UTC")})
77
+ if name == "unit_convert":
78
+ m = re.search(r"value=([\d.]+)", call)
79
+ value = float(m.group(1)) if m else 0.0
80
+ to = args.get("to_unit", "")
81
+ return json.dumps({"value": round(value * _UNITS.get((args.get("from_unit", ""), to), 1.0), 2), "unit": to})
82
+ if name == "search":
83
+ return json.dumps(_search(args.get("query", ""), online))
84
+ if name == "database_query":
85
+ return json.dumps({"status": "success", "count": 3, "records": [{"id": 101, "status": "active"}]})
86
+ if name == "create_event":
87
+ return json.dumps({"event_id": "evt_98214", "status": "confirmed"})
88
+ if name == "file_manager":
89
+ return json.dumps({"status": "ok", "bytes_written": 64})
90
+ if name == "fetch_api":
91
+ return json.dumps({"status": 200, "data": {"status": "healthy"}})
92
+ return json.dumps({"status": "executed", "call": call})
cbt_1d_twin_arithmetic_20_70d.json DELETED
@@ -1,224 +0,0 @@
1
- {
2
- "checkpoint": "champion_model/checkpoints/1d/cbt_18m_1d_rope_R3_k45_sftv5.zip",
3
- "max_train_operand_digits": 25,
4
- "arithmetic": {
5
- "per_length": {
6
- "20": {
7
- "digits": 20,
8
- "mode": "teacher_forced",
9
- "seen_in_training": true,
10
- "n": 30,
11
- "exact": 28,
12
- "exact_ci": [
13
- 78.7,
14
- 98.2
15
- ],
16
- "think_exact": 28,
17
- "think_ci": [
18
- 78.7,
19
- 98.2
20
- ],
21
- "gen_tokens": 0,
22
- "elapsed_sec": 9.7
23
- },
24
- "25": {
25
- "digits": 25,
26
- "mode": "teacher_forced",
27
- "seen_in_training": true,
28
- "n": 30,
29
- "exact": 29,
30
- "exact_ci": [
31
- 83.3,
32
- 99.4
33
- ],
34
- "think_exact": 29,
35
- "think_ci": [
36
- 83.3,
37
- 99.4
38
- ],
39
- "gen_tokens": 0,
40
- "elapsed_sec": 12.08
41
- },
42
- "26": {
43
- "digits": 26,
44
- "mode": "teacher_forced",
45
- "seen_in_training": false,
46
- "n": 30,
47
- "exact": 0,
48
- "exact_ci": [
49
- 0.0,
50
- 11.4
51
- ],
52
- "think_exact": 0,
53
- "think_ci": [
54
- 0.0,
55
- 11.4
56
- ],
57
- "gen_tokens": 0,
58
- "elapsed_sec": 12.42
59
- },
60
- "30": {
61
- "digits": 30,
62
- "mode": "teacher_forced",
63
- "seen_in_training": false,
64
- "n": 30,
65
- "exact": 0,
66
- "exact_ci": [
67
- 0.0,
68
- 11.4
69
- ],
70
- "think_exact": 0,
71
- "think_ci": [
72
- 0.0,
73
- 11.4
74
- ],
75
- "gen_tokens": 0,
76
- "elapsed_sec": 14.01
77
- },
78
- "35": {
79
- "digits": 35,
80
- "mode": "teacher_forced",
81
- "seen_in_training": false,
82
- "n": 30,
83
- "exact": 0,
84
- "exact_ci": [
85
- 0.0,
86
- 11.4
87
- ],
88
- "think_exact": 0,
89
- "think_ci": [
90
- 0.0,
91
- 11.4
92
- ],
93
- "gen_tokens": 0,
94
- "elapsed_sec": 17.04
95
- },
96
- "40": {
97
- "digits": 40,
98
- "mode": "teacher_forced",
99
- "seen_in_training": false,
100
- "n": 30,
101
- "exact": 0,
102
- "exact_ci": [
103
- 0.0,
104
- 11.4
105
- ],
106
- "think_exact": 0,
107
- "think_ci": [
108
- 0.0,
109
- 11.4
110
- ],
111
- "gen_tokens": 0,
112
- "elapsed_sec": 19.7
113
- },
114
- "45": {
115
- "digits": 45,
116
- "mode": "teacher_forced",
117
- "seen_in_training": false,
118
- "n": 30,
119
- "exact": 0,
120
- "exact_ci": [
121
- 0.0,
122
- 11.4
123
- ],
124
- "think_exact": 0,
125
- "think_ci": [
126
- 0.0,
127
- 11.4
128
- ],
129
- "gen_tokens": 0,
130
- "elapsed_sec": 22.33
131
- },
132
- "50": {
133
- "digits": 50,
134
- "mode": "teacher_forced",
135
- "seen_in_training": false,
136
- "n": 30,
137
- "exact": 0,
138
- "exact_ci": [
139
- 0.0,
140
- 11.4
141
- ],
142
- "think_exact": 0,
143
- "think_ci": [
144
- 0.0,
145
- 11.4
146
- ],
147
- "gen_tokens": 0,
148
- "elapsed_sec": 23.61
149
- },
150
- "55": {
151
- "digits": 55,
152
- "mode": "teacher_forced",
153
- "seen_in_training": false,
154
- "n": 30,
155
- "exact": 0,
156
- "exact_ci": [
157
- 0.0,
158
- 11.4
159
- ],
160
- "think_exact": 0,
161
- "think_ci": [
162
- 0.0,
163
- 11.4
164
- ],
165
- "gen_tokens": 0,
166
- "elapsed_sec": 26.12
167
- },
168
- "60": {
169
- "digits": 60,
170
- "mode": "teacher_forced",
171
- "seen_in_training": false,
172
- "n": 30,
173
- "exact": 0,
174
- "exact_ci": [
175
- 0.0,
176
- 11.4
177
- ],
178
- "think_exact": 0,
179
- "think_ci": [
180
- 0.0,
181
- 11.4
182
- ],
183
- "gen_tokens": 0,
184
- "elapsed_sec": 28.45
185
- },
186
- "70": {
187
- "digits": 70,
188
- "mode": "teacher_forced",
189
- "seen_in_training": false,
190
- "n": 30,
191
- "exact": 0,
192
- "exact_ci": [
193
- 0.0,
194
- 11.4
195
- ],
196
- "think_exact": 0,
197
- "think_ci": [
198
- 0.0,
199
- 11.4
200
- ],
201
- "gen_tokens": 0,
202
- "elapsed_sec": 33.21
203
- }
204
- },
205
- "summary": {
206
- "unseen": {
207
- "n": 270,
208
- "exact": 0,
209
- "exact_ci": [
210
- 0.0,
211
- 1.4
212
- ]
213
- },
214
- "seen": {
215
- "n": 60,
216
- "exact": 57,
217
- "exact_ci": [
218
- 86.3,
219
- 98.3
220
- ]
221
- }
222
- }
223
- }
224
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
champion_v3_arithmetic.json DELETED
@@ -1,224 +0,0 @@
1
- {
2
- "checkpoint": "champion_model/checkpoints/champion_v3_final.zip",
3
- "max_train_operand_digits": 25,
4
- "arithmetic": {
5
- "per_length": {
6
- "20": {
7
- "digits": 20,
8
- "mode": "teacher_forced",
9
- "seen_in_training": true,
10
- "n": 30,
11
- "exact": 30,
12
- "exact_ci": [
13
- 88.6,
14
- 100.0
15
- ],
16
- "think_exact": 30,
17
- "think_ci": [
18
- 88.6,
19
- 100.0
20
- ],
21
- "gen_tokens": 0,
22
- "elapsed_sec": 60.31
23
- },
24
- "25": {
25
- "digits": 25,
26
- "mode": "teacher_forced",
27
- "seen_in_training": true,
28
- "n": 30,
29
- "exact": 30,
30
- "exact_ci": [
31
- 88.6,
32
- 100.0
33
- ],
34
- "think_exact": 30,
35
- "think_ci": [
36
- 88.6,
37
- 100.0
38
- ],
39
- "gen_tokens": 0,
40
- "elapsed_sec": 59.46
41
- },
42
- "26": {
43
- "digits": 26,
44
- "mode": "teacher_forced",
45
- "seen_in_training": false,
46
- "n": 30,
47
- "exact": 30,
48
- "exact_ci": [
49
- 88.6,
50
- 100.0
51
- ],
52
- "think_exact": 30,
53
- "think_ci": [
54
- 88.6,
55
- 100.0
56
- ],
57
- "gen_tokens": 0,
58
- "elapsed_sec": 67.86
59
- },
60
- "30": {
61
- "digits": 30,
62
- "mode": "teacher_forced",
63
- "seen_in_training": false,
64
- "n": 30,
65
- "exact": 30,
66
- "exact_ci": [
67
- 88.6,
68
- 100.0
69
- ],
70
- "think_exact": 30,
71
- "think_ci": [
72
- 88.6,
73
- 100.0
74
- ],
75
- "gen_tokens": 0,
76
- "elapsed_sec": 63.91
77
- },
78
- "35": {
79
- "digits": 35,
80
- "mode": "teacher_forced",
81
- "seen_in_training": false,
82
- "n": 30,
83
- "exact": 30,
84
- "exact_ci": [
85
- 88.6,
86
- 100.0
87
- ],
88
- "think_exact": 30,
89
- "think_ci": [
90
- 88.6,
91
- 100.0
92
- ],
93
- "gen_tokens": 0,
94
- "elapsed_sec": 76.18
95
- },
96
- "40": {
97
- "digits": 40,
98
- "mode": "teacher_forced",
99
- "seen_in_training": false,
100
- "n": 30,
101
- "exact": 30,
102
- "exact_ci": [
103
- 88.6,
104
- 100.0
105
- ],
106
- "think_exact": 30,
107
- "think_ci": [
108
- 88.6,
109
- 100.0
110
- ],
111
- "gen_tokens": 0,
112
- "elapsed_sec": 84.75
113
- },
114
- "45": {
115
- "digits": 45,
116
- "mode": "teacher_forced",
117
- "seen_in_training": false,
118
- "n": 30,
119
- "exact": 30,
120
- "exact_ci": [
121
- 88.6,
122
- 100.0
123
- ],
124
- "think_exact": 30,
125
- "think_ci": [
126
- 88.6,
127
- 100.0
128
- ],
129
- "gen_tokens": 0,
130
- "elapsed_sec": 91.2
131
- },
132
- "50": {
133
- "digits": 50,
134
- "mode": "teacher_forced",
135
- "seen_in_training": false,
136
- "n": 30,
137
- "exact": 30,
138
- "exact_ci": [
139
- 88.6,
140
- 100.0
141
- ],
142
- "think_exact": 30,
143
- "think_ci": [
144
- 88.6,
145
- 100.0
146
- ],
147
- "gen_tokens": 0,
148
- "elapsed_sec": 115.78
149
- },
150
- "55": {
151
- "digits": 55,
152
- "mode": "teacher_forced",
153
- "seen_in_training": false,
154
- "n": 30,
155
- "exact": 30,
156
- "exact_ci": [
157
- 88.6,
158
- 100.0
159
- ],
160
- "think_exact": 30,
161
- "think_ci": [
162
- 88.6,
163
- 100.0
164
- ],
165
- "gen_tokens": 0,
166
- "elapsed_sec": 107.83
167
- },
168
- "60": {
169
- "digits": 60,
170
- "mode": "teacher_forced",
171
- "seen_in_training": false,
172
- "n": 30,
173
- "exact": 25,
174
- "exact_ci": [
175
- 66.4,
176
- 92.7
177
- ],
178
- "think_exact": 25,
179
- "think_ci": [
180
- 66.4,
181
- 92.7
182
- ],
183
- "gen_tokens": 0,
184
- "elapsed_sec": 120.42
185
- },
186
- "70": {
187
- "digits": 70,
188
- "mode": "teacher_forced",
189
- "seen_in_training": false,
190
- "n": 30,
191
- "exact": 19,
192
- "exact_ci": [
193
- 45.5,
194
- 78.1
195
- ],
196
- "think_exact": 19,
197
- "think_ci": [
198
- 45.5,
199
- 78.1
200
- ],
201
- "gen_tokens": 0,
202
- "elapsed_sec": 144.89
203
- }
204
- },
205
- "summary": {
206
- "unseen": {
207
- "n": 270,
208
- "exact": 254,
209
- "exact_ci": [
210
- 90.6,
211
- 96.3
212
- ]
213
- },
214
- "seen": {
215
- "n": 60,
216
- "exact": 60,
217
- "exact_ci": [
218
- 94.0,
219
- 100.0
220
- ]
221
- }
222
- }
223
- }
224
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
champion_v3_arithmetic_4_16d.json DELETED
@@ -1,108 +0,0 @@
1
- {
2
- "checkpoint": "checkpoints/cbt/cbt_18m_2d_rope_R3_k45_sftv5.pt",
3
- "max_train_operand_digits": 25,
4
- "arithmetic": {
5
- "per_length": {
6
- "4": {
7
- "digits": 4,
8
- "mode": "teacher_forced",
9
- "seen_in_training": true,
10
- "n": 30,
11
- "exact": 30,
12
- "exact_ci": [
13
- 88.6,
14
- 100.0
15
- ],
16
- "think_exact": 30,
17
- "think_ci": [
18
- 88.6,
19
- 100.0
20
- ],
21
- "gen_tokens": 0,
22
- "elapsed_sec": 3.29
23
- },
24
- "8": {
25
- "digits": 8,
26
- "mode": "teacher_forced",
27
- "seen_in_training": true,
28
- "n": 30,
29
- "exact": 30,
30
- "exact_ci": [
31
- 88.6,
32
- 100.0
33
- ],
34
- "think_exact": 30,
35
- "think_ci": [
36
- 88.6,
37
- 100.0
38
- ],
39
- "gen_tokens": 0,
40
- "elapsed_sec": 5.23
41
- },
42
- "12": {
43
- "digits": 12,
44
- "mode": "teacher_forced",
45
- "seen_in_training": true,
46
- "n": 30,
47
- "exact": 30,
48
- "exact_ci": [
49
- 88.6,
50
- 100.0
51
- ],
52
- "think_exact": 30,
53
- "think_ci": [
54
- 88.6,
55
- 100.0
56
- ],
57
- "gen_tokens": 0,
58
- "elapsed_sec": 6.68
59
- },
60
- "14": {
61
- "digits": 14,
62
- "mode": "teacher_forced",
63
- "seen_in_training": true,
64
- "n": 30,
65
- "exact": 30,
66
- "exact_ci": [
67
- 88.6,
68
- 100.0
69
- ],
70
- "think_exact": 30,
71
- "think_ci": [
72
- 88.6,
73
- 100.0
74
- ],
75
- "gen_tokens": 0,
76
- "elapsed_sec": 7.93
77
- },
78
- "16": {
79
- "digits": 16,
80
- "mode": "teacher_forced",
81
- "seen_in_training": true,
82
- "n": 30,
83
- "exact": 30,
84
- "exact_ci": [
85
- 88.6,
86
- 100.0
87
- ],
88
- "think_exact": 30,
89
- "think_ci": [
90
- 88.6,
91
- 100.0
92
- ],
93
- "gen_tokens": 0,
94
- "elapsed_sec": 8.82
95
- }
96
- },
97
- "summary": {
98
- "seen": {
99
- "n": 150,
100
- "exact": 150,
101
- "exact_ci": [
102
- 97.5,
103
- 100.0
104
- ]
105
- }
106
- }
107
- }
108
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
chat.py ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Terminal demo for CBT-18.95M. Sums like `123+456` go to the adder; anything else is a chat turn.
3
+
4
+ python chat.py # interactive
5
+ python chat.py "98765432109876+1234" # one prompt
6
+ python chat.py --online "Who was Ada Lovelace?"
7
+ """
8
+
9
+ import argparse
10
+ import functools
11
+ import re
12
+
13
+ from cbt import add, chat, execute_tool, load
14
+
15
+ SUM = re.compile(r"^\s*(\d+)\s*\+\s*(\d+)\s*=?\s*$")
16
+
17
+
18
+ def respond(model, tok, text: str, online: bool, show_raw: bool) -> None:
19
+ m = SUM.match(text)
20
+ if m:
21
+ r = add(model, tok, m.group(1), m.group(2))
22
+ print(f" scratchpad : {r['scratchpad']}")
23
+ print(f" answer : {r['answer']} [{'PASS' if r['correct'] else 'FAIL'}]")
24
+ else:
25
+ r = chat(model, tok, text, tool_executor=functools.partial(execute_tool, online=online))
26
+ for key in ("think", "call", "result", "answer"):
27
+ if r[key] is not None:
28
+ print(f" {key:<10} : {r[key]}")
29
+ if show_raw:
30
+ print(f" raw : {r['raw']}")
31
+ print()
32
+
33
+
34
+ def main() -> None:
35
+ p = argparse.ArgumentParser(description="CBT-18.95M terminal demo")
36
+ p.add_argument("prompt", nargs="?", help="single prompt; omit for interactive mode")
37
+ p.add_argument("--model", default=None, help="local folder or Hugging Face repo id (default: this folder)")
38
+ p.add_argument("--device", default=None, help="cpu or cuda (default: auto)")
39
+ p.add_argument("--online", action="store_true", help="let the search tool query Wikipedia")
40
+ p.add_argument("--raw", action="store_true", help="also print the raw token stream")
41
+ args = p.parse_args()
42
+
43
+ model, tok = load(args.model, args.device)
44
+ if args.prompt:
45
+ respond(model, tok, args.prompt, args.online, args.raw)
46
+ return
47
+
48
+ print("CBT-18.95M. Type a sum (e.g. 123456789+987654321) or a question. Ctrl-D to quit.\n")
49
+ while True:
50
+ try:
51
+ text = input("> ").strip()
52
+ except (EOFError, KeyboardInterrupt):
53
+ print()
54
+ break
55
+ if text.lower() in ("exit", "quit"):
56
+ break
57
+ if text:
58
+ respond(model, tok, text, args.online, args.raw)
59
+
60
+
61
+ if __name__ == "__main__":
62
+ main()
config.json CHANGED
@@ -1,6 +1,8 @@
1
  {
2
- "model_type": "scaled_t4",
3
- "vocab_size": 115,
 
 
4
  "block_size": 1024,
5
  "n_layer": 6,
6
  "n_head": 8,
@@ -8,24 +10,8 @@
8
  "n_recur": 3,
9
  "dropout": 0.0,
10
  "bias": false,
11
- "pos_emb_type": "rope_2d",
12
  "rope_theta": 10000.0,
13
  "max_coords": 4096,
14
- "k_rand_shift": 32,
15
- "effective_depth": 18,
16
- "head_dim": 64,
17
- "parameters": {
18
- "total": 18953216,
19
- "non_embedding": 18887680,
20
- "target_scale": "~10M to ~20M",
21
- "status": "[OK] Scaled 18.95M parameter architecture with 2D-RoPE and R=3 recurrence"
22
- },
23
- "t4_optimization": {
24
- "precision": "FP16 AMP (torch.amp.autocast)",
25
- "scaler": "torch.amp.GradScaler('cuda')",
26
- "sdpa": "Scaled Dot-Product Attention with O(1) activation memory",
27
- "distributed": "PyTorch DistributedDataParallel (DDP) across 2x T4 GPUs",
28
- "bfloat16_allowed": false,
29
- "tf32_allowed": false
30
- }
31
- }
 
1
  {
2
+ "model_type": "cbt",
3
+ "architecture": "CBT-18.95M",
4
+ "training_step": 23927,
5
+ "vocab_size": 111,
6
  "block_size": 1024,
7
  "n_layer": 6,
8
  "n_head": 8,
 
10
  "n_recur": 3,
11
  "dropout": 0.0,
12
  "bias": false,
 
13
  "rope_theta": 10000.0,
14
  "max_coords": 4096,
15
+ "k_rand_shift": 45,
16
+ "pos_emb_type": "rope_2d"
17
+ }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
dataset.py DELETED
@@ -1,1468 +0,0 @@
1
- """
2
- Joint Balanced Dataset & Batch Encoder for ScaledGPT (Dual T4 Optimization).
3
- Supports:
4
- 1. Multi-digit column scratchpad arithmetic (1 to 50+ digits curriculum).
5
- 2. Expanded tool repertoire:
6
- - Single-tool calling (calculator, weather, search, time, unit_convert)
7
- - Multi-tool sequential chaining
8
- - Nested JSON parameter schemas
9
- - Error handling and recovery traces
10
- - Multi-turn conversational dialogues
11
- - Direct conversational routing (no-tool)
12
- 3. Strict prompt and environment output loss masking.
13
- """
14
-
15
- import os
16
- import sys
17
- import json
18
- import random
19
- from typing import List, Dict, Tuple, Any, Optional
20
- import torch
21
-
22
- ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
23
- if ROOT_DIR not in sys.path:
24
- sys.path.insert(0, ROOT_DIR)
25
-
26
- try:
27
- from champion_model.scaled_t4.tokenizer import ScaledTokenizer
28
- except ImportError:
29
- from tokenizer import ScaledTokenizer
30
-
31
-
32
- def format_addition_char_scratchpad(a: int, b: int) -> str:
33
- """Formats addition into column-by-column carry scratchpad for character tokenizer."""
34
- s_a, s_b = str(a), str(b)
35
- len_a, len_b = len(s_a), len(s_b)
36
- max_len = max(len_a, len_b)
37
-
38
- carry = 0
39
- steps, digits = [], []
40
-
41
- for col in range(max_len):
42
- da = int(s_a[len_a - 1 - col]) if col < len_a else 0
43
- db = int(s_b[len_b - 1 - col]) if col < len_b else 0
44
- col_sum = carry + da + db
45
- next_carry = col_sum // 10
46
- res_digit = col_sum % 10
47
- steps.append(f"({carry}+{da}+{db})={next_carry}{res_digit}")
48
- digits.append(str(res_digit))
49
- carry = next_carry
50
-
51
- if carry > 0:
52
- steps.append(f"({carry}+0+0)=0{carry}")
53
- digits.append(str(carry))
54
-
55
- ans = "".join(reversed(digits))
56
- return f"{a}+{b}=<THINK>" + "".join(steps) + f"</THINK>={ans}"
57
-
58
-
59
- def extract_scratchpad_answer(text: str) -> Tuple[str, str]:
60
- """Extracts reasoning scratchpad and predicted answer from generated output."""
61
- reasoning = ""
62
- ans = ""
63
- if "<THINK>" in text and "</THINK>" in text:
64
- parts = text.split("</THINK>")
65
- reasoning = parts[0].replace("<THINK>", "")
66
- after = parts[1]
67
- if "=" in after:
68
- ans = after.split("=")[-1].replace("<EOS>", "").replace("<PAD>", "").strip()
69
- else:
70
- ans = after.replace("<EOS>", "").replace("<PAD>", "").strip()
71
- elif "=" in text:
72
- ans = text.split("=")[-1].replace("<EOS>", "").replace("<PAD>", "").strip()
73
- return reasoning, ans
74
-
75
- # Factual databases for synthetic generation
76
- CITIES = [
77
- "New York", "London", "Tokyo", "Paris", "Berlin", "Sydney", "Toronto", "Madrid",
78
- "Rome", "Seoul", "Singapore", "Zurich", "Honolulu", "Reykjavik", "Ouagadougou",
79
- "Vancouver", "Santiago", "Auckland", "Casablanca", "Vienna", "Stockholm", "Kyoto",
80
- "Dublin", "Oslo", "Helsinki", "Bangkok", "Cairo", "Lisbon", "Amsterdam", "Athens",
81
- "Buenos Aires", "Kuala Lumpur", "Rio de Janeiro", "San Francisco", "Hong Kong",
82
- "Los Angeles", "Mexico City", "Cape Town", "Sao Paulo",
83
- ]
84
- CONDITIONS = ["Sunny", "Rainy", "Cloudy", "Snowy", "Windy", "Foggy", "Clear"]
85
- TIMEZONES = ["UTC", "EST", "PST", "GMT", "JST", "CET", "AEST", "CST", "IST", "HST"]
86
-
87
- # Nonce word phonetic generators to force pure Attention Induction Copying
88
- NONCE_ONSETS = ["bl", "br", "ch", "cl", "cr", "dr", "fl", "fr", "gl", "gr", "pl", "pr", "sc", "sk", "sl", "sm", "sn", "sp", "st", "sw", "th", "tr", "qu", "v", "z", "k", "j"]
89
- NONCE_NUCLEI = ["a", "e", "i", "o", "u", "y", "ai", "ea", "ee", "oa", "oo", "ou"]
90
- NONCE_CODAS = ["b", "d", "f", "g", "k", "l", "m", "n", "p", "r", "s", "t", "x", "z", "ck", "nk", "mp", "sh", "th", "nt", "rt"]
91
-
92
-
93
- def generate_nonce_entity(rng: random.Random, allow_multi_word: bool = True) -> Tuple[str, str, str]:
94
- """Generates an arbitrary pseudo-word entity (singular, plural, type), supporting single and multi-word spans."""
95
- ent_type = rng.choice(["concept", "plant", "creature", "artifact", "substance", "person", "theory"])
96
-
97
- # 30% chance of multi-word compound nonce (2-3 words) to train multi-word pointer induction
98
- if allow_multi_word and rng.random() < 0.30:
99
- n_words = rng.choice([2, 2, 3])
100
- words = []
101
- for _ in range(n_words):
102
- n_syl = rng.choice([1, 2, 2, 3])
103
- w = "".join(rng.choice(NONCE_ONSETS) + rng.choice(NONCE_NUCLEI) + rng.choice(NONCE_CODAS) for _ in range(n_syl))
104
- words.append(w.capitalize())
105
- singular = " ".join(words)
106
- last_w = words[-1]
107
- plural_last = last_w + ("es" if last_w[-1] in "sxz" else "s")
108
- plural = " ".join(words[:-1] + [plural_last])
109
- return singular, plural, ent_type
110
-
111
- n_syl = rng.choice([1, 2, 2, 3, 4])
112
- parts = []
113
- for _ in range(n_syl):
114
- parts.append(rng.choice(NONCE_ONSETS) + rng.choice(NONCE_NUCLEI) + rng.choice(NONCE_CODAS))
115
- singular = "".join(parts)
116
-
117
- # 20% chance of complex morphological suffix to train long-token copy fidelity
118
- if rng.random() < 0.20:
119
- suffix = rng.choice(["ism", "ology", "ification", "ation", "ological", "ity"])
120
- singular = singular + suffix
121
-
122
- plural = singular + ("es" if singular[-1] in "sxz" else "s")
123
- return singular, plural, ent_type
124
- SEARCH_FACTS = [
125
- ("Python creator", "Guido van Rossum", "creator"),
126
- ("capital of Australia", "Canberra", "capital"),
127
- ("speed of light", "299792 km/s", "constant"),
128
- ("tallest mountain on Earth", "Mount Everest", "geography"),
129
- ("deepest ocean trench", "Mariana Trench", "geography"),
130
- ("inventor of the telephone", "Alexander Graham Bell", "inventor"),
131
- ("currency of Japan", "Yen", "currency"),
132
- ("largest planet in solar system", "Jupiter", "astronomy"),
133
- ("author of Hamlet", "William Shakespeare", "literature"),
134
- ("chemical symbol for Gold", "Au", "chemistry"),
135
- ("Tokyo population", "14000000", "demographics"),
136
- ("Paris land area", "105 sq km", "geography"),
137
- ("distance from Earth to Moon", "384400 km", "astronomy"),
138
- ("human body temperature", "37 Celsius", "biology"),
139
- ]
140
-
141
- # Open-domain entity database for Universal Search Delegation
142
- SEARCH_ENTITIES = [
143
- # Foods & Plants
144
- {"entity": "tomato", "plural": "tomatoes", "type": "plant", "snippet": "the edible berry of the plant Solanum lycopersicum, commonly prepared as a culinary vegetable."},
145
- {"entity": "apple", "plural": "apples", "type": "fruit", "snippet": "a round, edible fruit produced by an apple tree, cultivated worldwide."},
146
- {"entity": "potato", "plural": "potatoes", "type": "vegetable", "snippet": "a starchy, tuberous root crop of the nightshade family Solanaceae."},
147
- {"entity": "coffee", "plural": "coffee beans", "type": "plant", "snippet": "a brewed beverage prepared from roasted coffee beans of flowering Coffea species."},
148
- {"entity": "banana", "plural": "bananas", "type": "fruit", "snippet": "an elongated, edible fruit produced by large herbaceous plants in the genus Musa."},
149
- {"entity": "onion", "plural": "onions", "type": "vegetable", "snippet": "a pungent vegetable that is the most widely cultivated species of the genus Allium."},
150
- {"entity": "wheat", "plural": "wheat grains", "type": "crop", "snippet": "a cereal grain cultivated worldwide as a major dietary staple."},
151
- {"entity": "rice", "plural": "rice grains", "type": "crop", "snippet": "the seed of the grass species Oryza sativa, a primary staple food for billions of people."},
152
- {"entity": "carrot", "plural": "carrots", "type": "vegetable", "snippet": "a root vegetable, typically orange, rich in beta-carotene and dietary fiber."},
153
- {"entity": "garlic", "plural": "garlic cloves", "type": "plant", "snippet": "a species of bulbous flowering plant in the genus Allium, used for culinary seasoning."},
154
- # Animals
155
- {"entity": "dog", "plural": "dogs", "type": "animal", "snippet": "a domesticated descendant of the wolf, widely kept as a companion and working animal."},
156
- {"entity": "cat", "plural": "cats", "type": "animal", "snippet": "a small, carnivorous mammal known for its agility, flexible body, and companionship with humans."},
157
- {"entity": "dolphin", "plural": "dolphins", "type": "animal", "snippet": "a highly intelligent marine mammal belonging to the cetacean order, known for echolocation."},
158
- {"entity": "elephant", "plural": "elephants", "type": "animal", "snippet": "the largest existing land animal, characterized by a long trunk, tusks, and large ear flaps."},
159
- {"entity": "penguin", "plural": "penguins", "type": "animal", "snippet": "an aquatic, flightless bird living almost exclusively in the Southern Hemisphere."},
160
- {"entity": "lion", "plural": "lions", "type": "animal", "snippet": "a large cat of the genus Panthera, native to Africa and India, known for social prides."},
161
- {"entity": "tiger", "plural": "tigers", "type": "animal", "snippet": "the largest living cat species, identifiable by its reddish-orange coat with dark vertical stripes."},
162
- {"entity": "whale", "plural": "whales", "type": "animal", "snippet": "a widely distributed and diverse group of fully aquatic placental marine mammals."},
163
- # Science & Nature
164
- {"entity": "gravity", "plural": "gravitational forces", "type": "physics", "snippet": "a fundamental interaction which causes mutual attraction between all things with mass or energy."},
165
- {"entity": "photosynthesis", "plural": "photosynthetic processes", "type": "biology", "snippet": "a biological process used by plants to convert light energy into chemical energy stored in glucose."},
166
- {"entity": "atom", "plural": "atoms", "type": "physics", "snippet": "the basic unit of a chemical element, consisting of a dense central nucleus surrounded by electrons."},
167
- {"entity": "DNA", "plural": "DNA molecules", "type": "genetics", "snippet": "deoxyribonucleic acid, a polymer composed of polynucleotide chains carrying genetic instructions."},
168
- {"entity": "black hole", "plural": "black holes", "type": "astronomy", "snippet": "a region of spacetime where gravity is so strong that nothing, including light, can escape from it."},
169
- {"entity": "water", "plural": "water molecules", "type": "chemistry", "snippet": "a transparent, odorless chemical substance with chemical formula H2O, essential for life."},
170
- {"entity": "oxygen", "plural": "oxygen molecules", "type": "chemistry", "snippet": "a chemical element with symbol O and atomic number 8, essential for aerobic respiration."},
171
- {"entity": "electricity", "plural": "electric currents", "type": "physics", "snippet": "physical phenomena associated with the presence and motion of matter possessing electric charge."},
172
- # Geography & Landmarks
173
- {"entity": "Eiffel Tower", "plural": "monuments", "type": "landmark", "snippet": "a wrought-iron lattice tower on the Champ de Mars in Paris, France, named after Gustave Eiffel."},
174
- {"entity": "Great Wall of China", "plural": "ancient walls", "type": "history", "snippet": "a series of fortifications built across the historical northern borders of ancient Chinese states."},
175
- {"entity": "Mount Everest", "plural": "peaks", "type": "geography", "snippet": "Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas."},
176
- {"entity": "Amazon River", "plural": "rivers", "type": "geography", "snippet": "the largest river by discharge volume of water in the world, located in South America."},
177
- {"entity": "Sahara Desert", "plural": "deserts", "type": "geography", "snippet": "the largest hot desert in the world and the third largest desert overall, covering North Africa."},
178
- # Technology & Society
179
- {"entity": "computer", "plural": "computers", "type": "technology", "snippet": "an electronic device for storing and processing data, typically according to binary instructions."},
180
- {"entity": "internet", "plural": "computer networks", "type": "technology", "snippet": "a global system of interconnected computer networks that uses the Internet protocol suite (TCP/IP)."},
181
- {"entity": "algorithm", "plural": "algorithms", "type": "computer science", "snippet": "a finite sequence of rigorous instructions used to solve problems or perform a computation."},
182
- {"entity": "telescope", "plural": "telescopes", "type": "instrument", "snippet": "an optical instrument that makes distant objects appear magnified using lenses or curved mirrors."},
183
- {"entity": "airplane", "plural": "airplanes", "type": "vehicle", "snippet": "a powered, fixed-wing aircraft that is propelled forward by thrust from an engine."},
184
- # Historical Figures & Creators
185
- {"entity": "Guido van Rossum", "plural": "programmers", "type": "person", "snippet": "the Dutch programmer best known as the creator of the Python programming language."},
186
- {"entity": "Albert Einstein", "plural": "physicists", "type": "person", "snippet": "a theoretical physicist who developed the theory of relativity, a pillar of modern physics."},
187
- {"entity": "William Shakespeare", "plural": "playwrights", "type": "person", "snippet": "an English playwright, poet, and actor widely regarded as the greatest writer in the English language."},
188
- {"entity": "Alexander Graham Bell", "plural": "inventors", "type": "person", "snippet": "an inventor and scientist credited with patenting the first practical telephone."},
189
- {"entity": "Isaac Newton", "plural": "scientists", "type": "person", "snippet": "an English mathematician and physicist who formulated the laws of motion and universal gravitation."},
190
- {"entity": "Nikola Tesla", "plural": "engineers", "type": "person", "snippet": "a Serbian-American engineer known for his contributions to alternating current electricity."},
191
- {"entity": "Marie Curie", "plural": "scientists", "type": "person", "snippet": "a Polish-French physicist who pioneered radioactivity research and won two Nobel Prizes."},
192
- {"entity": "Alan Turing", "plural": "mathematicians", "type": "person", "snippet": "an English mathematician widely considered the father of modern computer science and AI."},
193
- {"entity": "Ada Lovelace", "plural": "mathematicians", "type": "person", "snippet": "an English mathematician celebrated for writing the first computer algorithm."},
194
- {"entity": "Aristotle", "plural": "philosophers", "type": "person", "snippet": "an ancient Greek philosopher whose work shaped centuries of Western logic and metaphysics."},
195
- {"entity": "Leonardo da Vinci", "plural": "polymaths", "type": "person", "snippet": "an Italian polymath active as a painter, draughtsman, engineer, and scientist."},
196
- # Additional Produce & Animals
197
- {"entity": "lemon", "plural": "lemons", "type": "fruit", "snippet": "a species of small evergreen tree native to Asia, known for its yellow ellipsoidal citrus fruit."},
198
- {"entity": "strawberry", "plural": "strawberries", "type": "fruit", "snippet": "a widely grown hybrid species of the genus Fragaria, cultivated for its sweet red fruit."},
199
- {"entity": "grape", "plural": "grapes", "type": "fruit", "snippet": "a botanical berry of the deciduous woody vines of the flowering plant genus Vitis."},
200
- {"entity": "spinach", "plural": "spinach leaves", "type": "vegetable", "snippet": "an edible flowering plant in the family Amaranthaceae, rich in iron and vitamins."},
201
- {"entity": "broccoli", "plural": "broccoli florets", "type": "vegetable", "snippet": "an edible green plant in the cabbage family whose large flower head is eaten as a vegetable."},
202
- {"entity": "avocado", "plural": "avocados", "type": "fruit", "snippet": "a medium-sized, evergreen tree native to the Americas, harvested for its buttery fruit."},
203
- {"entity": "wolf", "plural": "wolves", "type": "animal", "snippet": "a large canine native to Eurasia and North America, known for pack hunting behavior."},
204
- {"entity": "eagle", "plural": "eagles", "type": "animal", "snippet": "a large, powerfully built bird of prey belonging to the family Accipitridae."},
205
- {"entity": "shark", "plural": "sharks", "type": "animal", "snippet": "a group of elasmobranch fish characterized by a cartilaginous skeleton and multiple gill slits."},
206
- {"entity": "octopus", "plural": "octopuses", "type": "animal", "snippet": "a soft-bodied, eight-limbed mollusc of the order Octopoda, famous for high intelligence."},
207
- # Science & Computing
208
- {"entity": "quantum entanglement", "plural": "entangled states", "type": "physics", "snippet": "a physical phenomenon occurring when pairs of particles interact such that quantum states cannot be described independently."},
209
- {"entity": "thermodynamics", "plural": "thermodynamic laws", "type": "physics", "snippet": "a branch of physics that deals with heat, work, temperature, and statistical mechanics."},
210
- {"entity": "compiler", "plural": "compilers", "type": "computer science", "snippet": "a computer program that translates computer code written in one language into another."},
211
- {"entity": "operating system", "plural": "operating systems", "type": "computer science", "snippet": "system software that manages computer hardware, software resources, and common services."},
212
- {"entity": "neural network", "plural": "neural networks", "type": "computer science", "snippet": "a computational model inspired by the biological neural networks of animal brains."},
213
- # Alphanumeric, Scientific & Tech Entities
214
- {"entity": "CRISPR Cas9", "plural": "CRISPR Cas9 systems", "type": "genetics", "snippet": "a gene editing technology using an RNA-guided endonuclease enzyme."},
215
- {"entity": "Boeing 747", "plural": "Boeing 747 aircraft", "type": "aerospace", "snippet": "a wide-body commercial airliner and cargo transport aircraft."},
216
- {"entity": "Apollo 11", "plural": "Apollo missions", "type": "spaceflight", "snippet": "the American spaceflight that first landed humans on the Moon in 1969."},
217
- {"entity": "Web3", "plural": "decentralized protocols", "type": "technology", "snippet": "an idea for an iteration of the World Wide Web based on blockchain technology."},
218
- {"entity": "Formula 1", "plural": "Formula 1 races", "type": "motorsport", "snippet": "the highest class of international racing for open-wheel single-seater formula cars."},
219
- {"entity": "IPv6", "plural": "IPv6 addresses", "type": "networking", "snippet": "the most recent version of the Internet Protocol providing a vast numerical address space."},
220
- # Classical, Philosophical & Cultural Terms
221
- {"entity": "Marcus Aurelius", "plural": "Roman emperors", "type": "person", "snippet": "a Roman emperor and Stoic philosopher who ruled from 161 to 180 AD."},
222
- {"entity": "brutalist architecture", "plural": "brutalist structures", "type": "architecture", "snippet": "an architectural style characterized by minimalist constructions showcasing bare building materials."},
223
- {"entity": "existentialism", "plural": "existentialist doctrines", "type": "philosophy", "snippet": "a philosophical inquiry exploring human existence, freedom, and meaning."},
224
- {"entity": "Ragnarok", "plural": "mythological battles", "type": "mythology", "snippet": "in Norse mythology, a series of catastrophic events leading to the renewal of the world."},
225
- {"entity": "theremin", "plural": "theremins", "type": "instrument", "snippet": "an electronic musical instrument controlled without physical contact by metal antennas."},
226
- {"entity": "hyperinflation", "plural": "hyperinflationary cycles", "type": "economics", "snippet": "rapid, excessive, and out-of-control general price increases in an economy."},
227
- {"entity": "quantum physics", "plural": "quantum systems", "type": "physics", "snippet": "a fundamental branch of physics that studies nature at the scale of atoms and subatomic particles."},
228
- {"entity": "interstellar", "plural": "interstellar regions", "type": "astronomy", "snippet": "the matter and radiation that exists in the space between the star systems in a galaxy."},
229
- {"entity": "tiramisu", "plural": "tiramisus", "type": "cuisine", "snippet": "a coffee-flavoured Italian dessert made of ladyfingers dipped in coffee and layered with mascarpone."},
230
- {"entity": "neutron stars", "plural": "neutron stars", "type": "astronomy", "snippet": "the collapsed core of a massive supergiant star composed almost entirely of neutrons."},
231
- {"entity": "transformers in deep learning", "plural": "transformer architectures", "type": "computer science", "snippet": "a deep learning architecture based on multi-head self-attention mechanisms."},
232
- ]
233
- DEPARTMENTS = ["engineering", "marketing", "sales", "finance", "legal", "product"]
234
-
235
-
236
- # -----------------------------------------------------------------------------
237
- # 1. Search Delegation & Copy Induction Dataset Generator
238
- # -----------------------------------------------------------------------------
239
-
240
- def generate_search_delegation_dataset(
241
- n_samples: int = 15000,
242
- seed: int = 42,
243
- ) -> List[Dict[str, str]]:
244
- """
245
- Generates concentrated Search Delegation samples for Induction Head copy training.
246
- Curriculum:
247
- - 65% Nonce Words (1-3 syllables) forcing pure Attention Pointer Copying (memorization impossible).
248
- - 35% Real-World Entities (tomatoes, photosynthesis, etc.) establishing lexical alignment.
249
- - Symmetric punctuation curriculum (50% with '?', '.', 50% raw text without punctuation).
250
- - Diverse question framing: What are, What is a, Define, Tell me about, Search for, Who is, Explain.
251
- """
252
- rng = random.Random(seed)
253
- samples: List[Dict[str, str]] = []
254
-
255
- for _ in range(n_samples):
256
- is_nonce = (rng.random() < 0.65)
257
- has_punct = (rng.random() < 0.50)
258
-
259
- if is_nonce:
260
- ent, plural, ent_type = generate_nonce_entity(rng)
261
- snippet = f"a hypothetical {ent_type} characterized in verified theoretical studies."
262
- else:
263
- ent_item = rng.choice(SEARCH_ENTITIES)
264
- ent = ent_item["entity"]
265
- plural = ent_item.get("plural", ent + "s")
266
- snippet = ent_item["snippet"]
267
- ent_type = ent_item.get("type", "concept")
268
-
269
- # Choose query framing
270
- frame = rng.choice([
271
- "what_are_plural",
272
- "what_is_singular",
273
- "what_is_concept",
274
- "define",
275
- "tell_me_about",
276
- "search_for",
277
- "who_is",
278
- "explain",
279
- "lookup",
280
- ])
281
-
282
- if frame == "what_are_plural":
283
- q_mark = "?" if has_punct else ""
284
- user_prompt = f"What are {plural}{q_mark}"
285
- search_q = plural
286
- thought = rng.choice([
287
- "The user is asking a factual question. I will query the search tool.",
288
- "The user requests factual information. Querying search.",
289
- "The user is seeking knowledge on this topic. I will search the web.",
290
- ])
291
- answer = f"{plural.capitalize()} are {snippet}"
292
- elif frame == "what_is_singular" and ent_type in ["plant", "fruit", "vegetable", "animal", "creature", "artifact", "instrument", "vehicle"]:
293
- article = "an" if ent[0].lower() in "aeiou" else "a"
294
- q_mark = "?" if has_punct else ""
295
- user_prompt = f"What is {article} {ent}{q_mark}"
296
- search_q = ent
297
- thought = rng.choice([
298
- "The user is asking a factual question. I will query the search tool.",
299
- "The user is inquiring about this entity. I will query the search tool.",
300
- "The user requests factual information. Querying search.",
301
- ])
302
- answer = f"{ent.capitalize()} is {snippet}"
303
- elif frame == "define":
304
- p_mark = "." if has_punct else ""
305
- user_prompt = f"Define {ent}{p_mark}"
306
- search_q = ent
307
- thought = rng.choice([
308
- "The user asks for a definition. Calling search tool.",
309
- "The user is asking a factual question. I will query the search tool.",
310
- "The user requests factual information. Querying search.",
311
- ])
312
- answer = f"{ent.capitalize()} is defined as {snippet}"
313
- elif frame == "tell_me_about":
314
- p_mark = "." if has_punct else ""
315
- user_prompt = f"Tell me about {ent}{p_mark}"
316
- search_q = ent
317
- thought = rng.choice([
318
- "The user requests factual background information. Querying search.",
319
- "The user is asking a factual question. I will query the search tool.",
320
- "The user is seeking knowledge on this topic. I will search the web.",
321
- ])
322
- answer = f"According to verified search results, {ent} is {snippet}"
323
- elif frame == "search_for":
324
- p_mark = "." if has_punct else ""
325
- user_prompt = f"Search for {ent}{p_mark}"
326
- search_q = ent
327
- thought = rng.choice([
328
- "The user explicitly requests a search. Calling search tool.",
329
- "The user requests factual information. Querying search.",
330
- "The user is asking a factual question. I will query the search tool.",
331
- ])
332
- answer = f"Search results indicate {ent} is {snippet}"
333
- elif frame == "who_is" and ent_type in ["person", "creature"]:
334
- q_mark = "?" if has_punct else ""
335
- user_prompt = f"Who was {ent}{q_mark}" if rng.random() < 0.5 else f"Who is {ent}{q_mark}"
336
- search_q = ent
337
- thought = rng.choice([
338
- "The user requests biographical information. Calling search.",
339
- "The user is asking a factual question. I will query the search tool.",
340
- "The user requests factual information. Querying search.",
341
- ])
342
- answer = f"{ent} was {snippet}" if "was" in user_prompt else f"{ent} is {snippet}"
343
- elif frame == "explain":
344
- p_mark = "." if has_punct else ""
345
- user_prompt = f"Explain {ent}{p_mark}"
346
- search_q = ent
347
- thought = rng.choice([
348
- "The user asks for an explanation of this subject. I will query the search tool.",
349
- "The user is asking a factual question. I will query the search tool.",
350
- "The user requests factual information. Querying search.",
351
- ])
352
- answer = f"{ent.capitalize()} is {snippet}"
353
- else:
354
- q_mark = "?" if has_punct else ""
355
- user_prompt = f"What is {ent}{q_mark}"
356
- search_q = ent
357
- thought = rng.choice([
358
- "The user is asking a factual question. I will query the search tool.",
359
- "The user requests factual information. Querying search.",
360
- "The user is seeking knowledge on this topic. I will search the web.",
361
- ])
362
- answer = f"{ent.capitalize()} is {snippet}"
363
-
364
- call = f"search(query=\"{search_q}\")"
365
- result_json = json.dumps({"snippet": snippet})
366
- full_text = (
367
- f"<USER>{user_prompt}</USER>"
368
- f"<THINK>{thought}</THINK>"
369
- f"<CALL>{call}</CALL>"
370
- f"<RESULT>{result_json}</RESULT>"
371
- f"<ANSWER>{answer}</ANSWER>"
372
- )
373
-
374
- samples.append({
375
- "task_type": "tool",
376
- "category": "search_delegation",
377
- "prompt": f"<USER>{user_prompt}</USER>",
378
- "full_text": full_text,
379
- })
380
-
381
- return samples
382
-
383
-
384
- # -----------------------------------------------------------------------------
385
- # 1. High-Digit Arithmetic Dataset Generator
386
- # -----------------------------------------------------------------------------
387
-
388
- def generate_arithmetic_equations(
389
- n_samples: int = 4000,
390
- seed: int = 42,
391
- max_train_digits: int = 15,
392
- ) -> List[Dict[str, str]]:
393
- """
394
- Generates balanced multi-digit addition equations with column carry scratchpad.
395
- Curriculum: 1 to max_train_digits (default 15) digits, plus edge cases:
396
- - Zero padding
397
- - Cascade carries (e.g. 9999 + 1)
398
- - Equal length and asymmetric lengths
399
- """
400
- rng = random.Random(seed)
401
- samples: List[Dict[str, str]] = []
402
-
403
- for _ in range(n_samples):
404
- mode = rng.random()
405
-
406
- if mode < 0.20:
407
- # Cascade carry stress cases (999... + 1 or 999... + small)
408
- digits = rng.randint(2, max_train_digits)
409
- a = int("9" * digits)
410
- b = rng.randint(1, 9)
411
- elif mode < 0.35:
412
- # Round numbers / power of 10 zero tracking
413
- d1 = rng.randint(2, max_train_digits)
414
- d2 = rng.randint(2, max_train_digits)
415
- a = rng.randint(1, 9) * (10 ** (d1 - 1))
416
- b = rng.randint(1, 9) * (10 ** (d2 - 1))
417
- elif mode < 0.60:
418
- # Uniform length random digits (1 to max_train_digits)
419
- d = rng.randint(1, max_train_digits)
420
- min_v = 10 ** (d - 1) if d > 1 else 0
421
- max_v = (10 ** d) - 1
422
- a = rng.randint(min_v, max_v)
423
- b = rng.randint(min_v, max_v)
424
- else:
425
- # Asymmetric length random digits
426
- d1 = rng.randint(1, max_train_digits)
427
- d2 = rng.randint(1, max_train_digits)
428
- min_v1 = 10 ** (d1 - 1) if d1 > 1 else 0
429
- max_v1 = (10 ** d1) - 1
430
- min_v2 = 10 ** (d2 - 1) if d2 > 1 else 0
431
- max_v2 = (10 ** d2) - 1
432
- a = rng.randint(min_v1, max_v1)
433
- b = rng.randint(min_v2, max_v2)
434
-
435
- full_text = format_addition_char_scratchpad(a, b)
436
- prompt = full_text.split("<THINK>")[0]
437
- samples.append({
438
- "task_type": "math",
439
- "prompt": prompt,
440
- "full_text": full_text,
441
- })
442
-
443
- return samples
444
-
445
-
446
- # -----------------------------------------------------------------------------
447
- # 2. Expanded Tool Repertoire Generator
448
- # -----------------------------------------------------------------------------
449
-
450
- def generate_tool_example(rng: random.Random, category: Optional[str] = None) -> Dict[str, str]:
451
- """
452
- Generates a structured synthetic tool interaction from 6 expanded categories:
453
- 1. Single-tool calling (calculator, weather, search, time, unit_convert)
454
- 2. Multi-tool sequential chaining
455
- 3. Nested JSON parameter schemas
456
- 4. Error handling and recovery
457
- 5. Multi-turn dialogues
458
- 6. Direct conversational routing (no-tool)
459
-
460
- Enforces 1-Hop Induction Copying:
461
- - <THINK> emits a clean, entity-agnostic routing intent (no variable copying in <THINK>).
462
- - <CALL> copies arguments directly from <USER> via 1-hop attention pointers.
463
- - Symmetric punctuation curriculum (50% with '?'/'.', 50% raw conversational text).
464
- `category` forces a specific category instead of sampling one.
465
- """
466
- category = category or rng.choices(
467
- [
468
- "single_tool",
469
- "multi_tool_chain",
470
- "nested_json_schema",
471
- "error_recovery",
472
- "multi_turn",
473
- "no_tool",
474
- ],
475
- weights=[0.55, 0.12, 0.12, 0.08, 0.06, 0.07],
476
- k=1,
477
- )[0]
478
- has_punct = (rng.random() < 0.50)
479
- q_mark = "?" if has_punct else ""
480
- p_mark = "." if has_punct else ""
481
-
482
- if category == "single_tool":
483
- sub = rng.choices(
484
- ["weather", "calculator", "time", "unit_convert", "search"],
485
- weights=[0.28, 0.24, 0.20, 0.20, 0.08],
486
- k=1,
487
- )[0]
488
- if sub == "calculator":
489
- op = rng.choice(["+", "-", "*", "/"])
490
- if op == "+":
491
- a, b = rng.randint(10, 9999), rng.randint(10, 9999)
492
- res = a + b
493
- elif op == "-":
494
- a, b = rng.randint(100, 9999), rng.randint(10, 99)
495
- res = a - b
496
- elif op == "*":
497
- a, b = rng.randint(2, 250), rng.randint(2, 99)
498
- res = a * b
499
- else:
500
- b = rng.randint(2, 50)
501
- res = rng.randint(2, 200)
502
- a = res * b
503
-
504
- use_spaces = (rng.random() < 0.50)
505
- expr_prompt = f"{a} {op} {b}" if use_spaces else f"{a}{op}{b}"
506
- # Keep call expression identical to prompt expression or normalized so 1-hop pointer copies cleanly
507
- call_expr = expr_prompt if rng.random() < 0.65 else f"{a}{op}{b}"
508
- user_prompt = rng.choice([
509
- f"Calculate {expr_prompt}{p_mark}",
510
- f"What is {expr_prompt}{q_mark}",
511
- f"Compute {expr_prompt}{p_mark}",
512
- f"Evaluate {expr_prompt}{p_mark}",
513
- f"Evaluate the mathematical expression {expr_prompt}{p_mark}",
514
- f"Solve {expr_prompt}{p_mark}",
515
- ])
516
- thought = rng.choice([
517
- "The user requests an arithmetic calculation. I will invoke the calculator tool.",
518
- "The user is asking to evaluate a mathematical expression. Calling calculator.",
519
- ])
520
- call = f"calculator(expr=\"{call_expr}\")"
521
- result = json.dumps({"result": res})
522
- answer = f"The evaluated result of {call_expr} is {res}."
523
-
524
- elif sub == "weather":
525
- if rng.random() < 0.35:
526
- # Nonce city to force pure copy induction (single or multi-word)
527
- nonce_city, _, _ = generate_nonce_entity(rng, allow_multi_word=True)
528
- city = " ".join(w.capitalize() for w in nonce_city.split())
529
- else:
530
- city = rng.choice(CITIES)
531
- temp = rng.randint(-5, 35)
532
- cond = rng.choice(CONDITIONS)
533
- user_prompt = rng.choice([
534
- f"What is the weather in {city}{q_mark}",
535
- f"What is the weather on {city}{q_mark}",
536
- f"What is the weather for {city}{q_mark}",
537
- f"What is the current weather forecast for {city}{q_mark}",
538
- f"What is the weather like in {city}{q_mark}",
539
- f"How is the weather in {city}{q_mark}",
540
- f"How is the weather on {city}{q_mark}",
541
- f"How is the weather for {city}{q_mark}",
542
- f"Weather in {city}{p_mark}",
543
- f"Weather on {city}{p_mark}",
544
- f"Weather for {city}{p_mark}",
545
- f"Check the weather in {city}{p_mark}",
546
- f"Check the weather for {city}{p_mark}",
547
- f"Tell me the weather in {city}{p_mark}",
548
- f"Forecast for {city}{p_mark}",
549
- f"{city} weather{p_mark}",
550
- ])
551
- thought = rng.choice([
552
- "The user requests current weather information. I will query the get_weather tool.",
553
- "The user is asking for weather conditions. Querying get_weather.",
554
- ])
555
- call = f"get_weather(city=\"{city}\")"
556
- result = json.dumps({"temp": temp, "condition": cond, "units": "celsius"})
557
- answer = f"The weather in {city} is currently {cond} with a temperature of {temp} degrees Celsius."
558
-
559
- elif sub == "search":
560
- search_mode = rng.random()
561
- if search_mode < 0.50:
562
- ent, plural, ent_type = generate_nonce_entity(rng, allow_multi_word=True)
563
- snippet = f"a hypothetical {ent_type} characterized in general knowledge."
564
- elif search_mode < 0.85 and bool(SEARCH_ENTITIES):
565
- ent_item = rng.choice(SEARCH_ENTITIES)
566
- ent = ent_item["entity"]
567
- plural = ent_item.get("plural", ent + "s")
568
- snippet = ent_item["snippet"]
569
- ent_type = ent_item.get("type", "concept")
570
- else:
571
- ent = None
572
-
573
- if ent is not None:
574
- variant = rng.choice([
575
- "what_are_plural",
576
- "what_is_a",
577
- "what_is",
578
- "tell_me_about",
579
- "explain",
580
- "define",
581
- "search_for",
582
- "lookup",
583
- "who_is",
584
- ])
585
-
586
- if variant == "what_are_plural":
587
- user_prompt = f"What are {plural}{q_mark}"
588
- search_q = plural
589
- answer = f"{plural.capitalize()} are {snippet}"
590
- elif variant == "what_is_a" and ent_type in ["plant", "fruit", "vegetable", "animal", "creature", "artifact", "instrument", "vehicle"]:
591
- article = "an" if ent[0].lower() in "aeiou" else "a"
592
- user_prompt = f"What is {article} {ent}{q_mark}"
593
- search_q = ent
594
- answer = f"{ent.capitalize()} is {snippet}"
595
- elif variant == "tell_me_about":
596
- user_prompt = f"Tell me about {ent}{p_mark}"
597
- search_q = ent
598
- answer = f"According to verified search results, {ent} is {snippet}"
599
- elif variant == "define":
600
- user_prompt = f"Define {ent}{p_mark}"
601
- search_q = ent
602
- answer = f"{ent.capitalize()} is defined as {snippet}"
603
- elif variant == "explain":
604
- user_prompt = f"Explain {ent}{p_mark}"
605
- search_q = ent
606
- answer = f"{ent.capitalize()} is {snippet}"
607
- elif variant == "search_for":
608
- user_prompt = f"Search for {ent}{p_mark}"
609
- search_q = ent
610
- answer = f"Search results indicate {ent} is {snippet}"
611
- elif variant == "lookup":
612
- user_prompt = f"Look up {ent}{p_mark}"
613
- search_q = ent
614
- answer = f"Information for {ent}: {snippet}"
615
- elif variant == "who_is" and ent_type in ["person", "creature"]:
616
- prefix = "Who was" if rng.random() < 0.5 else "Who is"
617
- user_prompt = f"{prefix} {ent}{q_mark}"
618
- search_q = ent
619
- answer = f"{ent} is {snippet}"
620
- else:
621
- user_prompt = f"What is {ent}{q_mark}"
622
- search_q = ent
623
- answer = f"{ent.capitalize()} is {snippet}"
624
-
625
- thought = rng.choice([
626
- "The user is asking a factual question. I will query the search tool.",
627
- "The user requests factual information. Querying search.",
628
- "The user is seeking knowledge on this topic. I will search the web.",
629
- ])
630
- call = f"search(query=\"{search_q}\")"
631
- result = json.dumps({"snippet": snippet})
632
- else:
633
- topic, fact, _ = rng.choice(SEARCH_FACTS)
634
- user_prompt = rng.choice([
635
- f"Who or what is {topic}{q_mark}",
636
- f"What is {topic}{q_mark}",
637
- f"Tell me about {topic}{p_mark}",
638
- ])
639
- thought = "The user is asking a factual question. I will query the search tool."
640
- call = f"search(query=\"{topic}\")"
641
- result = json.dumps({"snippet": fact})
642
- answer = f"According to verified search results, {topic} is {fact}."
643
-
644
- elif sub == "time":
645
- tz = rng.choice(TIMEZONES)
646
- hr = rng.randint(0, 23)
647
- mn = rng.randint(0, 59)
648
- time_str = f"{hr:02d}:{mn:02d}"
649
- user_prompt = rng.choice([
650
- f"What is the current time in {tz}{q_mark}",
651
- f"What is the time in {tz}{q_mark}",
652
- f"What time is it in {tz}{q_mark}",
653
- f"Current time in {tz}{p_mark}",
654
- f"Time in {tz}{p_mark}",
655
- f"Tell me the time in {tz}{p_mark}",
656
- f"Check the time for {tz}{p_mark}",
657
- ])
658
- thought = rng.choice([
659
- "The user requests the current time for a timezone. I will query the get_time tool.",
660
- "The user is asking for the current time. Querying get_time.",
661
- ])
662
- call = f"get_time(timezone=\"{tz}\")"
663
- result = json.dumps({"time": time_str, "timezone": tz})
664
- answer = f"The current time in {tz} is {time_str}."
665
-
666
- else: # unit_convert
667
- val = rng.randint(5, 500)
668
- from_u, to_u, factor = rng.choice([
669
- ("km", "miles", 0.621371),
670
- ("miles", "km", 1.60934),
671
- ("kg", "lbs", 2.20462),
672
- ("lbs", "kg", 0.453592),
673
- ])
674
- converted = round(val * factor, 2)
675
- user_prompt = rng.choice([
676
- f"Convert {val} {from_u} to {to_u}{p_mark}",
677
- f"How many {to_u} is {val} {from_u}{q_mark}",
678
- f"How many {to_u} are in {val} {from_u}{q_mark}",
679
- f"What is {val} {from_u} in {to_u}{q_mark}",
680
- f"Change {val} {from_u} into {to_u}{p_mark}",
681
- ])
682
- thought = rng.choice([
683
- "The user requests a unit conversion. I will invoke the unit_convert tool.",
684
- "The user is asking to convert between measurement units. Calling unit_convert.",
685
- ])
686
- call = f"unit_convert(value={val}, from_unit=\"{from_u}\", to_unit=\"{to_u}\")"
687
- result = json.dumps({"value": converted, "unit": to_u})
688
- answer = f"{val} {from_u} is equal to {converted} {to_u}."
689
-
690
- full_text = (
691
- f"<USER>{user_prompt}</USER>"
692
- f"<THINK>{thought}</THINK>"
693
- f"<CALL>{call}</CALL>"
694
- f"<RESULT>{result}</RESULT>"
695
- f"<ANSWER>{answer}</ANSWER>"
696
- )
697
-
698
- elif category == "multi_tool_chain":
699
- # Chained sequential reasoning: Step 1 -> Tool 1 -> Result 1 -> Step 2 -> Tool 2 -> Result 2 -> Final Answer
700
- chain_type = rng.choice(["fact_calc", "dual_weather", "tz_calc"])
701
- if chain_type == "fact_calc":
702
- pop = 14000000
703
- area = 2194
704
- density = round(pop / area, 2)
705
- user_prompt = f"What is the population density of Tokyo based on its population and area of 2194 sq km{q_mark}"
706
- thought1 = "First, I need to search for Tokyo's population."
707
- call1 = "search(query=\"Tokyo population\")"
708
- result1 = json.dumps({"population": pop})
709
- thought2 = "Now I will calculate the population density using the calculator tool."
710
- call2 = f"calculator(expr=\"{pop} / {area}\")"
711
- result2 = json.dumps({"result": density})
712
- answer = f"Tokyo's population is {pop:,}. Dividing by 2,194 sq km gives a density of approximately {density} people per sq km."
713
-
714
- elif chain_type == "dual_weather":
715
- c1, c2 = rng.sample(CITIES, 2)
716
- t1, t2 = rng.randint(0, 30), rng.randint(0, 30)
717
- diff = abs(t1 - t2)
718
- user_prompt = rng.choice([
719
- f"Compare the temperature between {c1} and {c2}{p_mark}",
720
- f"Compare the weather in {c1} and {c2}{p_mark}",
721
- f"What is the temperature difference between {c1} and {c2}{q_mark}",
722
- ])
723
- thought1 = "I will first query the weather for the first city."
724
- call1 = f"get_weather(city=\"{c1}\")"
725
- result1 = json.dumps({"temp": t1, "condition": "Cloudy"})
726
- thought2 = "Next, I will query the weather for the second city."
727
- call2 = f"get_weather(city=\"{c2}\")"
728
- result2 = json.dumps({"temp": t2, "condition": "Sunny"})
729
- answer = f"{c1} is {t1}C and {c2} is {t2}C, with a temperature difference of {diff} degrees."
730
-
731
- else: # tz_calc
732
- tz1, tz2 = rng.sample(TIMEZONES[:6], 2)
733
- user_prompt = f"What is the time difference between {tz1} and {tz2}{q_mark}"
734
- thought1 = "First, I will query the current time in the first timezone."
735
- call1 = f"get_time(timezone=\"{tz1}\")"
736
- result1 = json.dumps({"time": "08:00", "utc_offset": -5})
737
- thought2 = "Next, I will query the current time in the second timezone."
738
- call2 = f"get_time(timezone=\"{tz2}\")"
739
- result2 = json.dumps({"time": "22:00", "utc_offset": 9})
740
- answer = f"{tz1} is UTC-5 and {tz2} is UTC+9, representing a 14-hour time difference."
741
-
742
- full_text = (
743
- f"<USER>{user_prompt}</USER>"
744
- f"<THINK>{thought1}</THINK>"
745
- f"<CALL>{call1}</CALL>"
746
- f"<RESULT>{result1}</RESULT>"
747
- f"<THINK>{thought2}</THINK>"
748
- f"<CALL>{call2}</CALL>"
749
- f"<RESULT>{result2}</RESULT>"
750
- f"<ANSWER>{answer}</ANSWER>"
751
- )
752
-
753
- elif category == "nested_json_schema":
754
- # Complex tools with nested dictionaries, lists, and boolean flags
755
- schema_type = rng.choice(["db_query", "event_create", "file_manage", "api_fetch"])
756
- if schema_type == "db_query":
757
- dept = rng.choice(DEPARTMENTS)
758
- limit = rng.randint(3, 10)
759
- user_prompt = rng.choice([
760
- f"Query the top {limit} active records in the {dept} department{p_mark}",
761
- f"Query active records in the {dept} department with limit {limit}{p_mark}",
762
- f"Fetch the top {limit} active entries from {dept}{p_mark}",
763
- ])
764
- thought = "I will invoke database_query with a nested filter and options schema."
765
- call = f"database_query(filter={{\"department\": \"{dept}\", \"status\": \"active\"}}, options={{\"limit\": {limit}, \"sort\": \"desc\"}})"
766
- result = json.dumps({"status": "success", "count": limit, "records": [{"id": 100 + i, "dept": dept} for i in range(min(limit, 3))]})
767
- answer = f"Found {limit} active records in {dept} according to your filter specifications."
768
-
769
- elif schema_type == "event_create":
770
- attendees = ["alice@corp.internal", "bob@corp.internal"]
771
- user_prompt = f"Schedule a Sprint Sync meeting for tomorrow from 10:00 to 11:00 UTC with Alice and Bob{p_mark}"
772
- thought = "I need to call create_event with a nested event schedule and attendee list."
773
- call = "create_event(event={\"title\": \"Sprint Sync\", \"schedule\": {\"date\": \"2026-11-01\", \"start\": \"10:00\", \"end\": \"11:00\", \"tz\": \"UTC\"}}, attendees=[\"alice@corp.internal\", \"bob@corp.internal\"])"
774
- result = json.dumps({"event_id": "evt_98214", "status": "confirmed"})
775
- answer = "Successfully scheduled the Sprint Sync meeting with Alice and Bob for 10:00-11:00 UTC (Event ID: evt_98214)."
776
-
777
- elif schema_type == "file_manage":
778
- path = "/configs/model_t4.json"
779
- user_prompt = f"Save model configuration to {path} with fp16 enabled and 2 workers{p_mark}"
780
- thought = "Calling file_manager with action write and a nested file payload."
781
- call = f"file_manager(action=\"write\", file={{\"path\": \"{path}\", \"format\": \"json\"}}, content={{\"fp16\": true, \"workers\": 2}})"
782
- result = json.dumps({"status": "ok", "bytes_written": 48})
783
- answer = f"Configuration file has been written to {path} with fp16=True and 2 workers."
784
-
785
- else: # api_fetch
786
- endpoint = "/v2/metrics/latency"
787
- user_prompt = f"Fetch latency metrics for EU region from {endpoint}{p_mark}"
788
- thought = "Calling fetch_api with endpoint and nested filter parameters."
789
- call = f"fetch_api(endpoint=\"{endpoint}\", params={{\"region\": \"EU\", \"metric\": \"p99\"}}, method=\"GET\")"
790
- result = json.dumps({"status": 200, "data": {"p99_ms": 14.2}})
791
- answer = f"Retrieved p99 latency for EU region from {endpoint}: 14.2 ms."
792
-
793
- full_text = (
794
- f"<USER>{user_prompt}</USER>"
795
- f"<THINK>{thought}</THINK>"
796
- f"<CALL>{call}</CALL>"
797
- f"<RESULT>{result}</RESULT>"
798
- f"<ANSWER>{answer}</ANSWER>"
799
- )
800
-
801
- elif category == "error_recovery":
802
- # Robust error reasoning & recovery paths
803
- err_type = rng.choice(["div_zero", "city_not_found", "api_503_fallback", "table_missing"])
804
- if err_type == "div_zero":
805
- num = rng.choice([100, 150, 250, 500])
806
- user_prompt = f"Calculate {num} / 0"
807
- thought1 = "The user requests an arithmetic calculation. I will invoke the calculator tool."
808
- call1 = f"calculator(expr=\"{num}/0\")"
809
- result1 = json.dumps({"error": "ZeroDivisionError: division by zero"})
810
- thought2 = "The calculator returned ZeroDivisionError. Division by zero is mathematically undefined. I will explain this clearly."
811
- answer = "Error: Division by zero is undefined in mathematics. Please specify a non-zero denominator."
812
- full_text = (
813
- f"<USER>{user_prompt}</USER>"
814
- f"<THINK>{thought1}</THINK>"
815
- f"<CALL>{call1}</CALL>"
816
- f"<RESULT>{result1}</RESULT>"
817
- f"<THINK>{thought2}</THINK>"
818
- f"<ANSWER>{answer}</ANSWER>"
819
- )
820
-
821
- elif err_type == "city_not_found":
822
- city = rng.choice(["Atlantis", "Narnia"])
823
- user_prompt = f"What is the weather in {city}{q_mark}"
824
- thought1 = "The user requests current weather information. I will query the get_weather tool."
825
- call1 = f"get_weather(city=\"{city}\")"
826
- result1 = json.dumps({"status": "error", "code": 404, "message": f"City '{city}' not found"})
827
- thought2 = "get_weather returned 404. The requested city is not recognized. I must inform the user."
828
- answer = f"I could not retrieve weather data for {city} because it is not recognized in our weather database."
829
- full_text = (
830
- f"<USER>{user_prompt}</USER>"
831
- f"<THINK>{thought1}</THINK>"
832
- f"<CALL>{call1}</CALL>"
833
- f"<RESULT>{result1}</RESULT>"
834
- f"<THINK>{thought2}</THINK>"
835
- f"<ANSWER>{answer}</ANSWER>"
836
- )
837
-
838
- elif err_type == "api_503_fallback":
839
- city = "Tokyo"
840
- user_prompt = f"Check the primary API forecast for {city}{p_mark}"
841
- thought1 = "Attempting primary weather API fetch."
842
- call1 = f"fetch_api(endpoint=\"/v1/weather\", params={{\"city\": \"{city}\"}}, method=\"GET\")"
843
- result1 = json.dumps({"status": "error", "code": 503, "message": "Service Unavailable"})
844
- thought2 = "Primary API failed with 503. Recovering via fallback tool get_weather."
845
- call2 = f"get_weather(city=\"{city}\")"
846
- result2 = json.dumps({"temp": 22, "condition": "Sunny"})
847
- answer = "Primary service was temporarily unavailable, but fallback weather confirmed Tokyo is Sunny at 22C."
848
- full_text = (
849
- f"<USER>{user_prompt}</USER>"
850
- f"<THINK>{thought1}</THINK>"
851
- f"<CALL>{call1}</CALL>"
852
- f"<RESULT>{result1}</RESULT>"
853
- f"<THINK>{thought2}</THINK>"
854
- f"<CALL>{call2}</CALL>"
855
- f"<RESULT>{result2}</RESULT>"
856
- f"<ANSWER>{answer}</ANSWER>"
857
- )
858
-
859
- else: # table_missing
860
- user_prompt = f"Query records from table 'legacy_metrics'{p_mark}"
861
- thought1 = "Querying database for the requested table."
862
- call1 = "database_query(filter={\"table\": \"legacy_metrics\"}, options={\"limit\": 5})"
863
- result1 = json.dumps({"status": "error", "code": "TABLE_NOT_FOUND", "message": "Table does not exist"})
864
- thought2 = "TABLE_NOT_FOUND error encountered. Informing user of invalid table name."
865
- answer = "The requested database table 'legacy_metrics' was not found. Please verify table name."
866
- full_text = (
867
- f"<USER>{user_prompt}</USER>"
868
- f"<THINK>{thought1}</THINK>"
869
- f"<CALL>{call1}</CALL>"
870
- f"<RESULT>{result1}</RESULT>"
871
- f"<THINK>{thought2}</THINK>"
872
- f"<ANSWER>{answer}</ANSWER>"
873
- )
874
-
875
- elif category == "multi_turn":
876
- # Multi-turn conversational dialogue with state continuity
877
- city = rng.choice(CITIES)
878
- temp = rng.randint(10, 25)
879
- w_q = rng.choice([
880
- f"How is the weather in {city}{q_mark}",
881
- f"What is the weather in {city}{q_mark}",
882
- f"What is the weather on {city}{q_mark}",
883
- ])
884
- full_text = (
885
- f"<USER>Hi there!</USER>"
886
- f"<THINK>Conversational greeting without tool.</THINK>"
887
- f"<ANSWER>Hello! How can I assist you today?</ANSWER>"
888
- f"<USER>{w_q}</USER>"
889
- f"<THINK>The user requests current weather information. I will query the get_weather tool.</THINK>"
890
- f"<CALL>get_weather(city=\"{city}\")</CALL>"
891
- f"<RESULT>{{\"temp\": {temp}, \"condition\": \"Clear\"}}</RESULT>"
892
- f"<ANSWER>{city} is currently Clear with a temperature of {temp} degrees Celsius.</ANSWER>"
893
- f"<USER>Thank you!</USER>"
894
- f"<THINK>User acknowledges. Polite closing response.</THINK>"
895
- f"<ANSWER>You are very welcome! Let me know if you need anything else.</ANSWER>"
896
- )
897
-
898
- else: # no_tool
899
- greeting = rng.choice([
900
- ("Hello! How are you?", "I am doing well, thank you! How may I assist you today?"),
901
- ("Hi there!", "Hello! What can I help you with today?"),
902
- ("Thank you for your assistance.", "You are welcome! Let me know if you have any more questions."),
903
- ("Thank you!", "You are very welcome! Let me know if you need anything else."),
904
- ("Good morning!", "Good morning! How may I help you?"),
905
- ("Who are you?", "I am ScaledGPT, an AI assistant with structured tool-calling and high-digit arithmetic capabilities."),
906
- ("Explain the concept of gradient descent.", "Gradient descent is an optimization algorithm that iteratively moves model weights in the direction of steepest loss decrease."),
907
- ])
908
- user_prompt, ans = greeting
909
- thought = "Conversational request that does not require any tool. I will answer directly."
910
- full_text = (
911
- f"<USER>{user_prompt}</USER>"
912
- f"<THINK>{thought}</THINK>"
913
- f"<ANSWER>{ans}</ANSWER>"
914
- )
915
-
916
- return {
917
- "category": category,
918
- "full_text": full_text,
919
- }
920
-
921
-
922
- def generate_tool_dataset(n_samples: int = 4000, seed: int = 1337) -> List[Dict[str, str]]:
923
- """Generates an expanded tool interaction dataset across all 6 categories."""
924
- rng = random.Random(seed)
925
- dataset: List[Dict[str, str]] = []
926
- for _ in range(n_samples):
927
- ex = generate_tool_example(rng)
928
- full_text = ex["full_text"]
929
- prompt = full_text.split("</USER>")[0] + "</USER>"
930
- dataset.append({
931
- "task_type": "tools",
932
- "prompt": prompt,
933
- "full_text": full_text,
934
- "category": ex["category"],
935
- })
936
- return dataset
937
-
938
-
939
- # -----------------------------------------------------------------------------
940
- # 2b. Curriculum V5: Broad-Distribution Copy Spans & Parameterized Schemas
941
- # -----------------------------------------------------------------------------
942
- # V4 copy spans came from one narrow phonotactic inventory, so real-world
943
- # spellings outside it (doubled vowels, lowercase connectors, leading articles)
944
- # drifted at copy time. V5 widens the span distribution, strips articles from
945
- # arguments, parameterizes every nested schema, and keeps <THINK> short so
946
- # nested calls close within a 140-token decode budget.
947
-
948
- SPAN_ONSETS_EXT = NONCE_ONSETS + [
949
- "", "", "b", "c", "d", "f", "g", "h", "l", "m", "n", "p", "r", "s", "t", "w", "y",
950
- "ph", "sh", "kh", "zh", "ts", "ng", "nj", "sr", "vr", "hy", "mb", "nd", "rh", "x",
951
- ]
952
- SPAN_NUCLEI_EXT = NONCE_NUCLEI + ["aa", "ii", "uu", "ae", "ia", "io", "ua", "ei", "au", "eo", "ie", "ya"]
953
- SPAN_CODAS_EXT = NONCE_CODAS + ["", "", "", "n", "r", "s", "l", "ng", "rm", "nd", "st", "x", "ch", "lt", "ss"]
954
- SPAN_CONNECTORS = ["of", "van", "de", "da", "del", "es", "la", "al", "bin", "von", "le", "and", "y", "el"]
955
- LETTERS = "abcdefghijklmnopqrstuvwxyz"
956
-
957
- V5_TIMEZONES = TIMEZONES + ["MST", "EET", "BST", "SGT", "NZST", "PDT", "EDT", "WET", "KST", "MSK"]
958
- V5_REGIONS = ["EU", "US", "APAC", "LATAM", "MEA", "UK", "JP", "IN", "BR", "CA"]
959
- V5_METRICS = ["latency", "errors", "uptime", "cpu", "memory", "requests", "traffic", "bandwidth"]
960
- V5_DAYS = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday", "tomorrow", "next week"]
961
- V5_FILE_EXTS = ["txt", "json", "csv", "md", "log", "py", "yaml", "cfg"]
962
- V5_RECORD_STATUSES = ["active", "inactive", "pending", "archived"]
963
-
964
- NO_TOOL_OPENERS = [
965
- "Hello", "Hi", "Hey", "Good morning", "Good afternoon", "Good evening",
966
- "Greetings", "Hi there", "Hello there", "Hey there", "Howdy",
967
- ]
968
- NO_TOOL_BODIES = [
969
- "how are you", "how are you today", "how are you doing", "how is it going",
970
- "how have you been", "what's up", "nice to meet you", "how is your day",
971
- "how is your day going", "how are things", "are you there", "how are you doing today",
972
- ]
973
- NO_TOOL_STANDALONE = [
974
- ("thanks", "Thank you"), ("thanks", "Thanks a lot"), ("thanks", "Thanks for the help"),
975
- ("thanks", "Thank you so much"), ("thanks", "Much appreciated"), ("thanks", "Great, thanks"),
976
- ("bye", "Goodbye"), ("bye", "See you later"), ("bye", "Good night"), ("bye", "Bye for now"),
977
- ("identity", "Who are you"), ("identity", "What can you do"), ("identity", "What is your name"),
978
- ("identity", "Are you a robot"), ("ack", "Okay"), ("ack", "Nice work"), ("ack", "That was helpful"),
979
- ("ack", "Cool"), ("ack", "Sounds good"), ("ack", "Got it"),
980
- ]
981
- NO_TOOL_REPLIES = {
982
- "greeting": [
983
- "Hello! I am doing well, thank you. How can I help you today?",
984
- "Hi! Everything is running smoothly. What can I do for you?",
985
- "Hello! How may I assist you today?",
986
- ],
987
- "thanks": [
988
- "You are welcome! Let me know if you need anything else.",
989
- "Happy to help! Feel free to ask another question.",
990
- ],
991
- "bye": ["Goodbye! Have a great day.", "See you soon! Take care."],
992
- "identity": [
993
- "I am CBT, a compact assistant that can call tools for weather, time, search, unit conversion, and arithmetic.",
994
- ],
995
- "ack": ["Great! Let me know if there is anything else I can do.", "Glad to hear it. What else can I help with?"],
996
- }
997
-
998
-
999
- def _span_word(rng: random.Random) -> str:
1000
- """Samples one lowercase word from a mixture of phonotactic, nonce, random-letter, and alphanumeric shapes."""
1001
- r = rng.random()
1002
- if r < 0.55:
1003
- w = ""
1004
- while not 2 <= len(w) <= 14:
1005
- n_syl = rng.choice([1, 2, 2, 2, 3, 3, 4])
1006
- w = "".join(
1007
- rng.choice(SPAN_ONSETS_EXT) + rng.choice(SPAN_NUCLEI_EXT)
1008
- + (rng.choice(SPAN_CODAS_EXT) if rng.random() < 0.5 else "")
1009
- for _ in range(n_syl)
1010
- )
1011
- elif r < 0.75:
1012
- w, _, _ = generate_nonce_entity(rng, allow_multi_word=False)
1013
- w = w.lower()
1014
- elif r < 0.90:
1015
- w = "".join(rng.choice(LETTERS) for _ in range(rng.randint(3, 12)))
1016
- else:
1017
- stem = "".join(rng.choice(LETTERS) for _ in range(rng.randint(1, 6)))
1018
- num = str(rng.randint(0, 999))
1019
- w = rng.choice([stem + num, stem + "-" + num, num + stem, stem.upper() + num])
1020
- return w if len(w) >= 2 else w + rng.choice(LETTERS)
1021
-
1022
-
1023
- def generate_copy_span(rng: random.Random, max_words: int = 4, title_prob: float = 0.5) -> str:
1024
- """Samples a 1-4 word span with mixed casing and optional lowercase connectors ('X de Y' shape)."""
1025
- n_words = rng.choices([1, 2, 3, 4][:max_words], weights=[0.50, 0.30, 0.15, 0.05][:max_words], k=1)[0]
1026
- words = [_span_word(rng) for _ in range(n_words)]
1027
- while len(" ".join(words)) > 36:
1028
- words = [_span_word(rng) for _ in range(n_words)]
1029
- casing = rng.random()
1030
- if casing < title_prob:
1031
- words = [w[0].upper() + w[1:] for w in words]
1032
- elif casing < title_prob + 0.05:
1033
- words = [w.upper() for w in words]
1034
- elif casing < title_prob + 0.10:
1035
- words[0] = words[0][0].upper() + words[0][1:]
1036
- if n_words >= 2 and rng.random() < 0.25:
1037
- words.insert(rng.randint(1, n_words - 1), rng.choice(SPAN_CONNECTORS))
1038
- return " ".join(words)
1039
-
1040
-
1041
- def _v5_entity(rng: random.Random) -> Tuple[str, str]:
1042
- """Returns (entity, snippet), mixing broad copy spans with case-perturbed known entities."""
1043
- if rng.random() < 0.70:
1044
- ent = generate_copy_span(rng)
1045
- return ent, "a topic described in verified reference sources."
1046
- item = rng.choice(SEARCH_ENTITIES)
1047
- ent = item["entity"]
1048
- r = rng.random()
1049
- if r < 0.25:
1050
- ent = ent.lower()
1051
- elif r < 0.35:
1052
- ent = ent.title()
1053
- return ent, item["snippet"]
1054
-
1055
-
1056
- def _wrap(user_prompt: str, thought: str, call: str, result: str, answer: str) -> str:
1057
- return (
1058
- f"<USER>{user_prompt}</USER><THINK>{thought}</THINK><CALL>{call}</CALL>"
1059
- f"<RESULT>{result}</RESULT><ANSWER>{answer}</ANSWER>"
1060
- )
1061
-
1062
-
1063
- def generate_search_v5_example(rng: random.Random) -> Dict[str, str]:
1064
- """Search delegation with article-stripped arguments ('Tell me about the X' -> query='X')."""
1065
- ent, snippet = _v5_entity(rng)
1066
- q = "?" if rng.random() < 0.5 else ""
1067
- p = "." if rng.random() < 0.5 else ""
1068
- article = "an" if ent[0].lower() in "aeiou" else "a"
1069
- user_prompt = rng.choice([
1070
- f"What is {ent}{q}", f"What is the {ent}{q}", f"What is {article} {ent}{q}", f"What are {ent}{q}",
1071
- f"What are the {ent}{q}", f"Tell me about {ent}{p}", f"Tell me about the {ent}{p}",
1072
- f"Tell me about {article} {ent}{p}", f"Define {ent}{p}", f"Define the term {ent}{p}",
1073
- f"Explain {ent}{p}", f"Explain the {ent}{p}", f"Search for {ent}{p}", f"Look up {ent}{p}",
1074
- f"Who is {ent}{q}", f"Who was {ent}{q}", f"Find information about {ent}{p}",
1075
- f"Give me information on {ent}{p}", f"I want to learn about {ent}{p}",
1076
- ])
1077
- thought = rng.choice([
1078
- "The user is asking a factual question. I will query the search tool.",
1079
- "The user requests factual information. Querying search.",
1080
- "The user is seeking knowledge on this topic. I will search the web.",
1081
- ])
1082
- full_text = _wrap(
1083
- user_prompt, thought, f"search(query=\"{ent}\")",
1084
- json.dumps({"snippet": snippet}), f"According to search results, {ent} is {snippet}",
1085
- )
1086
- return {"category": "search_v5", "full_text": full_text}
1087
-
1088
-
1089
- def generate_single_tool_v5_example(rng: random.Random) -> Dict[str, str]:
1090
- """Weather / calculator / time / unit_convert with broad spans and verbatim calculator expressions."""
1091
- q = "?" if rng.random() < 0.5 else ""
1092
- p = "." if rng.random() < 0.5 else ""
1093
- sub = rng.choices(["weather", "calculator", "time", "unit_convert"], weights=[0.35, 0.35, 0.15, 0.15], k=1)[0]
1094
-
1095
- if sub == "weather":
1096
- city = generate_copy_span(rng, max_words=3, title_prob=0.85) if rng.random() < 0.55 else rng.choice(CITIES)
1097
- temp, cond = rng.randint(-10, 40), rng.choice(CONDITIONS)
1098
- user_prompt = rng.choice([
1099
- f"What is the weather in {city}{q}", f"What is the weather on {city}{q}",
1100
- f"What is the weather for {city}{q}", f"What's the weather in {city}{q}",
1101
- f"What is the weather like in {city}{q}", f"How is the weather in {city}{q}",
1102
- f"How is the weather on {city}{q}", f"Weather in {city}{p}", f"Weather on {city}{p}",
1103
- f"Weather for {city}{p}", f"Check the weather in {city}{p}", f"Check the weather for {city}{p}",
1104
- f"Tell me the weather in {city}{p}", f"Forecast for {city}{p}", f"{city} weather{p}",
1105
- f"Is it raining in {city}{q}", f"Temperature in {city}{p}",
1106
- ])
1107
- thought = rng.choice([
1108
- "The user requests current weather information. I will query the get_weather tool.",
1109
- "The user is asking for weather conditions. Querying get_weather.",
1110
- ])
1111
- call = f"get_weather(city=\"{city}\")"
1112
- result = json.dumps({"temp": temp, "condition": cond, "units": "celsius"})
1113
- answer = f"The weather in {city} is currently {cond} with a temperature of {temp} degrees Celsius."
1114
-
1115
- elif sub == "calculator":
1116
- op = rng.choice(["+", "-", "*", "/"])
1117
- if op == "/":
1118
- b = rng.randint(2, 99)
1119
- res = rng.randint(2, 999)
1120
- a = res * b
1121
- else:
1122
- a = rng.randint(1, 10 ** rng.randint(1, 5) - 1)
1123
- b = rng.randint(1, 10 ** rng.randint(1, 4) - 1)
1124
- res = a + b if op == "+" else (a - b if op == "-" else a * b)
1125
- expr = rng.choice([f"{a} {op} {b}", f"{a}{op}{b}"])
1126
- user_prompt = rng.choice([
1127
- f"Calculate {expr}{p}", f"What is {expr}{q}", f"Compute {expr}{p}", f"Evaluate {expr}{p}",
1128
- f"Solve {expr}{p}", f"What's {expr}{q}", f"How much is {expr}{q}",
1129
- ])
1130
- thought = rng.choice([
1131
- "The user requests an arithmetic calculation. I will invoke the calculator tool.",
1132
- "The user is asking to evaluate a mathematical expression. Calling calculator.",
1133
- ])
1134
- call = f"calculator(expr=\"{expr}\")"
1135
- result = json.dumps({"result": res})
1136
- answer = f"The evaluated result of {expr} is {res}."
1137
-
1138
- elif sub == "time":
1139
- # 40% synthetic uppercase codes so timezone is copied from the prompt, not recalled from a list
1140
- tz = rng.choice(V5_TIMEZONES) if rng.random() < 0.6 else "".join(rng.choice(LETTERS) for _ in range(rng.randint(2, 5))).upper()
1141
- time_str = f"{rng.randint(0, 23):02d}:{rng.randint(0, 59):02d}"
1142
- user_prompt = rng.choice([
1143
- f"What is the current time in {tz}{q}", f"What is the time in {tz}{q}", f"What time is it in {tz}{q}",
1144
- f"Current time in {tz}{p}", f"Time in {tz}{p}", f"Tell me the time in {tz}{p}",
1145
- f"Check the time for {tz}{p}",
1146
- ])
1147
- thought = "The user is asking for the current time. Querying get_time."
1148
- call = f"get_time(timezone=\"{tz}\")"
1149
- result = json.dumps({"time": time_str, "timezone": tz})
1150
- answer = f"The current time in {tz} is {time_str}."
1151
-
1152
- else:
1153
- val = rng.randint(1, 1000)
1154
- from_u, to_u, factor = rng.choice([
1155
- ("km", "miles", 0.621371), ("miles", "km", 1.60934), ("kg", "lbs", 2.20462), ("lbs", "kg", 0.453592),
1156
- ])
1157
- user_prompt = rng.choice([
1158
- f"Convert {val} {from_u} to {to_u}{p}", f"How many {to_u} is {val} {from_u}{q}",
1159
- f"How many {to_u} are in {val} {from_u}{q}", f"What is {val} {from_u} in {to_u}{q}",
1160
- f"Change {val} {from_u} into {to_u}{p}",
1161
- ])
1162
- thought = "The user requests a unit conversion. I will invoke the unit_convert tool."
1163
- call = f"unit_convert(value={val}, from_unit=\"{from_u}\", to_unit=\"{to_u}\")"
1164
- result = json.dumps({"value": round(val * factor, 2), "unit": to_u})
1165
- answer = f"{val} {from_u} is equal to {round(val * factor, 2)} {to_u}."
1166
-
1167
- return {"category": f"single_{sub}", "full_text": _wrap(user_prompt, thought, call, result, answer)}
1168
-
1169
-
1170
- def generate_nested_v5_example(rng: random.Random) -> Dict[str, str]:
1171
- """Parameterized compact nested schemas; short <THINK> keeps the full call under the decode budget."""
1172
- p = "." if rng.random() < 0.5 else ""
1173
- schema = rng.choice(["db_query", "event_create", "file_manage", "api_fetch"])
1174
-
1175
- if schema == "db_query":
1176
- dept = rng.choice(DEPARTMENTS) if rng.random() < 0.6 else _span_word(rng).lower()[:12]
1177
- n = rng.randint(1, 20)
1178
- status = rng.choice(V5_RECORD_STATUSES)
1179
- user_prompt = rng.choice([
1180
- f"Query the top {n} {status} records in the {dept} department{p}",
1181
- f"Query top {n} {status} records in {dept}{p}",
1182
- f"Get {n} {status} records from {dept}{p}",
1183
- f"Show the top {n} {status} {dept} records{p}",
1184
- f"Fetch the top {n} {status} entries from {dept}{p}",
1185
- f"List {n} {status} records in the {dept} team{p}",
1186
- ])
1187
- thought = "Calling database_query."
1188
- call = f"database_query(filter={{\"department\": \"{dept}\", \"status\": \"{status}\"}}, options={{\"limit\": {n}}})"
1189
- result = json.dumps({"status": "success", "count": n})
1190
- answer = f"Found {n} {status} records in {dept}."
1191
-
1192
- elif schema == "event_create":
1193
- title = generate_copy_span(rng, max_words=3, title_prob=0.9)
1194
- h = rng.randint(0, 22)
1195
- start = f"{h:02d}:{rng.choice([0, 15, 30, 45]):02d}"
1196
- end = f"{h + 1:02d}:{start[-2:]}"
1197
- mode = rng.random()
1198
- if mode < 0.3:
1199
- day = rng.choice(V5_DAYS)
1200
- on = "" if day in ("tomorrow", "next week") else "on "
1201
- user_prompt = rng.choice([
1202
- f"Create a calendar event for {title} {on}{day}{p}",
1203
- f"Schedule {title} {on}{day}{p}",
1204
- f"Add {title} to my calendar {on}{day}{p}",
1205
- f"Book {title} {on}{day}{p}",
1206
- ])
1207
- call = f"create_event(event={{\"title\": \"{title}\", \"date\": \"{day}\"}})"
1208
- start = day
1209
- elif mode < 0.55:
1210
- user_prompt = rng.choice([
1211
- f"Schedule {title} from {start} to {end}{p}",
1212
- f"Create a calendar event for {title} from {start} to {end}{p}",
1213
- f"Book {title} between {start} and {end}{p}",
1214
- ])
1215
- call = f"create_event(event={{\"title\": \"{title}\", \"start\": \"{start}\", \"end\": \"{end}\"}})"
1216
- else:
1217
- user_prompt = rng.choice([
1218
- f"Create a calendar event for {title} at {start}{p}",
1219
- f"Create an event called {title} at {start}{p}",
1220
- f"Schedule {title} at {start}{p}",
1221
- f"Add {title} to my calendar at {start}{p}",
1222
- f"Set up {title} at {start} tomorrow{p}",
1223
- ])
1224
- call = f"create_event(event={{\"title\": \"{title}\", \"start\": \"{start}\"}})"
1225
- thought = "Calling create_event."
1226
- result = json.dumps({"event_id": f"evt_{rng.randint(10000, 99999)}", "status": "confirmed"})
1227
- answer = f"Scheduled {title} for {start}."
1228
-
1229
- elif schema == "file_manage":
1230
- fname = f"{_span_word(rng).lower()[:10]}.{rng.choice(V5_FILE_EXTS)}"
1231
- if rng.random() < 0.3:
1232
- fname = f"{rng.choice(['docs', 'data', 'logs', 'tmp', 'notes'])}/{fname}"
1233
- action = rng.choices(["write", "read", "delete"], weights=[0.7, 0.15, 0.15], k=1)[0]
1234
- if action == "write":
1235
- content = generate_copy_span(rng, max_words=3, title_prob=0.6)
1236
- user_prompt = rng.choice([
1237
- f"Save {fname} with content {content}{p}",
1238
- f"Write {content} to {fname}{p}",
1239
- f"Create file {fname} containing {content}{p}",
1240
- f"Save the text {content} in {fname}{p}",
1241
- ])
1242
- call = f"file_manager(action=\"write\", file={{\"path\": \"{fname}\"}}, content=\"{content}\")"
1243
- else:
1244
- verb = "Read" if action == "read" else "Delete"
1245
- user_prompt = rng.choice([f"{verb} {fname}{p}", f"{verb} the file {fname}{p}"])
1246
- call = f"file_manager(action=\"{action}\", file={{\"path\": \"{fname}\"}})"
1247
- thought = "Calling file_manager."
1248
- result = json.dumps({"status": "ok"})
1249
- answer = f"File operation {action} completed for {fname}."
1250
-
1251
- else:
1252
- segs = [_span_word(rng).lower()[:8] for _ in range(rng.randint(1, 2))]
1253
- endpoint = f"/v{rng.randint(1, 3)}/" + "/".join(segs)
1254
- region, metric = rng.choice(V5_REGIONS), rng.choice(V5_METRICS)
1255
- user_prompt = rng.choice([
1256
- f"Fetch {metric} metrics for {region} from {endpoint}{p}",
1257
- f"Fetch {metric} metrics for {region} region from {endpoint}{p}",
1258
- f"Get {metric} data from {endpoint} for {region}{p}",
1259
- f"Call {endpoint} for {region} {metric}{p}",
1260
- ])
1261
- thought = "Calling fetch_api."
1262
- call = f"fetch_api(endpoint=\"{endpoint}\", params={{\"region\": \"{region}\", \"metric\": \"{metric}\"}})"
1263
- result = json.dumps({"status": 200, "data": {"value": round(rng.random() * 100, 1)}})
1264
- answer = f"Retrieved {metric} for {region} from {endpoint}."
1265
-
1266
- return {"category": f"nested_{schema}", "full_text": _wrap(user_prompt, thought, call, result, answer)}
1267
-
1268
-
1269
- def generate_error_v5_example(rng: random.Random) -> Dict[str, str]:
1270
- """Error recovery traces using the V5 compact schemas."""
1271
- err = rng.choice(["div_zero", "city_not_found", "api_503_fallback", "table_missing"])
1272
- if err == "div_zero":
1273
- num = rng.randint(1, 9999)
1274
- user_prompt = f"Calculate {num} / 0"
1275
- steps = [
1276
- ("The user requests an arithmetic calculation. I will invoke the calculator tool.",
1277
- f"calculator(expr=\"{num} / 0\")", json.dumps({"error": "ZeroDivisionError: division by zero"})),
1278
- ]
1279
- final_thought = "The calculator returned ZeroDivisionError. I will explain this clearly."
1280
- answer = "Division by zero is undefined. Please provide a non-zero denominator."
1281
- elif err == "city_not_found":
1282
- city = generate_copy_span(rng, max_words=2, title_prob=1.0)
1283
- user_prompt = f"What is the weather in {city}"
1284
- steps = [
1285
- ("The user requests current weather information. I will query the get_weather tool.",
1286
- f"get_weather(city=\"{city}\")", json.dumps({"status": "error", "code": 404})),
1287
- ]
1288
- final_thought = "get_weather returned 404. I must inform the user."
1289
- answer = f"I could not retrieve weather data for {city} because it is not recognized."
1290
- elif err == "api_503_fallback":
1291
- city = rng.choice(CITIES)
1292
- user_prompt = f"Check the primary API forecast for {city}"
1293
- steps = [
1294
- ("Calling fetch_api.", f"fetch_api(endpoint=\"/v1/weather\", params={{\"city\": \"{city}\"}})",
1295
- json.dumps({"status": 503})),
1296
- ("Primary API failed with 503. Falling back to get_weather.", f"get_weather(city=\"{city}\")",
1297
- json.dumps({"temp": 22, "condition": "Sunny"})),
1298
- ]
1299
- final_thought = ""
1300
- answer = f"The primary service was unavailable, but the fallback reports {city} is Sunny at 22C."
1301
- else:
1302
- table = _span_word(rng).lower()[:12]
1303
- user_prompt = f"Query records from table {table}"
1304
- steps = [
1305
- ("Calling database_query.", f"database_query(filter={{\"table\": \"{table}\"}}, options={{\"limit\": 5}})",
1306
- json.dumps({"status": "error", "code": "TABLE_NOT_FOUND"})),
1307
- ]
1308
- final_thought = "TABLE_NOT_FOUND. Informing the user."
1309
- answer = f"The table {table} was not found. Please verify the table name."
1310
-
1311
- text = f"<USER>{user_prompt}</USER>"
1312
- for thought, call, result in steps:
1313
- text += f"<THINK>{thought}</THINK><CALL>{call}</CALL><RESULT>{result}</RESULT>"
1314
- if final_thought:
1315
- text += f"<THINK>{final_thought}</THINK>"
1316
- text += f"<ANSWER>{answer}</ANSWER>"
1317
- return {"category": "error_v5", "full_text": text}
1318
-
1319
-
1320
- def generate_no_tool_v5_example(rng: random.Random) -> Dict[str, str]:
1321
- """Compositional chit-chat (greetings with 'today'/'doing' etc.) routed directly to <ANSWER>."""
1322
- if rng.random() < 0.6:
1323
- opener, body = rng.choice(NO_TOOL_OPENERS), rng.choice(NO_TOOL_BODIES)
1324
- sep = rng.choice([", ", "! ", " ", ". "])
1325
- user_prompt = f"{opener}{sep}{body}{rng.choice(['?', '', '!'])}"
1326
- if rng.random() < 0.15:
1327
- user_prompt = user_prompt.lower()
1328
- kind = "greeting"
1329
- else:
1330
- kind, user_prompt = rng.choice(NO_TOOL_STANDALONE)
1331
- user_prompt += rng.choice(["", ".", "!", "?"] if kind == "identity" else ["", ".", "!"])
1332
- thought = "Conversational request that does not require any tool. I will answer directly."
1333
- full_text = f"<USER>{user_prompt}</USER><THINK>{thought}</THINK><ANSWER>{rng.choice(NO_TOOL_REPLIES[kind])}</ANSWER>"
1334
- return {"category": "no_tool_v5", "full_text": full_text}
1335
-
1336
-
1337
- def generate_v5_tool_example(rng: random.Random) -> Dict[str, str]:
1338
- """Samples one non-search V5 tool trace; chains and multi-turn reuse the V4 generator."""
1339
- category = rng.choices(
1340
- ["single", "nested", "no_tool", "error", "chain", "multi_turn"],
1341
- weights=[0.42, 0.24, 0.14, 0.08, 0.06, 0.06],
1342
- k=1,
1343
- )[0]
1344
- if category == "single":
1345
- return generate_single_tool_v5_example(rng)
1346
- if category == "nested":
1347
- return generate_nested_v5_example(rng)
1348
- if category == "no_tool":
1349
- return generate_no_tool_v5_example(rng)
1350
- if category == "error":
1351
- return generate_error_v5_example(rng)
1352
- return generate_tool_example(rng, category="multi_tool_chain" if category == "chain" else "multi_turn")
1353
-
1354
-
1355
- def to_tool_item(example: Dict[str, str]) -> Dict[str, str]:
1356
- """Wraps a generated trace into the encode_joint_batch item format."""
1357
- full_text = example["full_text"]
1358
- return {
1359
- "task_type": "tools",
1360
- "prompt": full_text.split("</USER>")[0] + "</USER>",
1361
- "full_text": full_text,
1362
- "category": example["category"],
1363
- }
1364
-
1365
-
1366
- def generate_joint_dataset(
1367
- n_math: int = 4000,
1368
- n_tools: int = 4000,
1369
- seed: int = 42,
1370
- max_train_digits: int = 15,
1371
- ) -> Tuple[List[Dict[str, str]], List[Dict[str, str]]]:
1372
- """Generates dual datasets for math curriculum and expanded tools."""
1373
- math_samples = generate_arithmetic_equations(n_samples=n_math, seed=seed, max_train_digits=max_train_digits)
1374
- tool_samples = generate_tool_dataset(n_samples=n_tools, seed=seed + 100)
1375
- return math_samples, tool_samples
1376
-
1377
-
1378
- # -----------------------------------------------------------------------------
1379
- # 3. Batch Encoder & Strict Loss Masker
1380
- # -----------------------------------------------------------------------------
1381
-
1382
- def encode_joint_batch(
1383
- math_items: List[Dict[str, str]],
1384
- tool_items: List[Dict[str, str]],
1385
- tokenizer: ScaledTokenizer,
1386
- max_len: Optional[int] = None,
1387
- device: Optional[torch.device] = None,
1388
- ) -> Tuple[torch.Tensor, torch.Tensor]:
1389
- """
1390
- Encodes balanced joint batch of math and tool sequences.
1391
- Strict Prompt & Environment Loss Masking:
1392
- - Math: masks everything up to and including '<THINK>' (loss on scratchpad + answer).
1393
- - Tools:
1394
- - Masks everything inside '<USER>...</USER>' except at '</USER>' (where target is '<THINK>').
1395
- - Masks everything inside '<RESULT>...</RESULT>' except at '</RESULT>' (where target is '<THINK>' or '<ANSWER>').
1396
- - Supervised: reasoning traces '<THINK>', tool calls '<CALL>', and answers '<ANSWER>'.
1397
- """
1398
- items = math_items + tool_items
1399
- B = len(items)
1400
-
1401
- encoded_seqs = [tokenizer.encode(item["full_text"], add_bos=True, add_eos=True) for item in items]
1402
- calc_max_len = max(len(s) for s in encoded_seqs) - 1
1403
- if max_len is not None:
1404
- batch_max = min(calc_max_len, max_len)
1405
- else:
1406
- batch_max = calc_max_len
1407
-
1408
- x_tensor = torch.full((B, batch_max), tokenizer.pad_id, dtype=torch.long)
1409
- y_tensor = torch.full((B, batch_max), -100, dtype=torch.long)
1410
-
1411
- think_s_id = tokenizer.think_s_id
1412
- user_s_id = tokenizer.user_s_id
1413
- user_e_id = tokenizer.user_e_id
1414
- result_s_id = tokenizer.result_s_id
1415
- result_e_id = tokenizer.result_e_id
1416
-
1417
- for i, (item, tokens) in enumerate(zip(items, encoded_seqs)):
1418
- # Truncate if exceeding batch_max + 1
1419
- if len(tokens) > batch_max + 1:
1420
- tokens = tokens[: batch_max + 1]
1421
-
1422
- n = len(tokens) - 1
1423
- t_tok = torch.tensor(tokens, dtype=torch.long)
1424
- x_tensor[i, :n] = t_tok[:-1]
1425
- y_tensor[i, :n] = t_tok[1:].clone()
1426
-
1427
- if item["task_type"] == "math":
1428
- # Mask prompt up to '<THINK>'
1429
- think_matches = (t_tok[:-1] == think_s_id).nonzero(as_tuple=True)[0]
1430
- if len(think_matches) > 0:
1431
- mask_until = think_matches[0].item()
1432
- y_tensor[i, :mask_until] = -100
1433
- else:
1434
- # Segment-aware masking for single-turn, multi-turn, and tool responses
1435
- mask = torch.zeros(n, dtype=torch.bool)
1436
- in_user = False
1437
- in_result = False
1438
-
1439
- for pos in range(n):
1440
- tok_id = t_tok[pos].item()
1441
- if tok_id == user_s_id:
1442
- in_user = True
1443
- elif tok_id == result_s_id:
1444
- in_result = True
1445
-
1446
- if in_user:
1447
- if tok_id == user_e_id:
1448
- in_user = False
1449
- mask[pos] = False
1450
- else:
1451
- mask[pos] = True
1452
- elif in_result:
1453
- if tok_id == result_e_id:
1454
- in_result = False
1455
- mask[pos] = False
1456
- else:
1457
- mask[pos] = True
1458
- elif pos == 0:
1459
- # BOS token preceding <USER>
1460
- mask[pos] = True
1461
-
1462
- y_tensor[i, :n][mask] = -100
1463
-
1464
- if device is not None:
1465
- x_tensor = x_tensor.to(device)
1466
- y_tensor = y_tensor.to(device)
1467
-
1468
- return x_tensor, y_tensor
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
equalized_15d_1024_and_ruoss_sweep.json DELETED
@@ -1,890 +0,0 @@
1
- {
2
- "part3_18m_1d_at_1024_context": {
3
- "25d_cascade": {
4
- "passed": false,
5
- "zero_loop": false,
6
- "extracted_answer_prefix": "9991000900999",
7
- "gen_tail": "+9+0)=10(1+9+0)=10(1+0+0)=01(0+9+0)=09(0+9+0)=09(0+9+0)=09</THINK>=9991000900999"
8
- },
9
- "35d_ones_eights": {
10
- "passed": false,
11
- "zero_loop": false,
12
- "extracted_answer_prefix": "08(0+0+0+0101999",
13
- "gen_tail": "+8)=09(0+1+8)=09(0+8+8)=16(1+1+0)=02(0+1+8)=09(0+1+8)=09(0+0+8)=08(0+0+0+0101999"
14
- },
15
- "50d_comp": {
16
- "passed": false,
17
- "zero_loop": false,
18
- "extracted_answer_prefix": "10880+08+8+6+5)=1",
19
- "gen_tail": "10(1+6+0)=07(0+6+0)=06(0+0+0)=00(0+1+0+0+0+4+1+0+0)=01</THINK>=10880+08+8+6+5)=1"
20
- },
21
- "50d_cascade": {
22
- "passed": false,
23
- "zero_loop": false,
24
- "extracted_answer_prefix": "<THINK>99999999999999999999999",
25
- "gen_tail": "99999999999999999999999999999999999999999999999999999999999999999999999999999999"
26
- }
27
- },
28
- "equalized_15d_1024_comparison": {
29
- "1d_standard_15d_1024": {
30
- "meta": {
31
- "id": "1d_standard_15d_1024",
32
- "label": "1D-RoPE Standard (1-15d, T=1024)",
33
- "pos_mode": "1d_standard",
34
- "k_shift": 0,
35
- "final_loss": 0.09822,
36
- "train_sec": 238.36
37
- },
38
- "pooled_seen_8_15d": {
39
- "passed": 0,
40
- "n": 600,
41
- "accuracy": 0.0,
42
- "wilson_95_ci": [
43
- 0.0,
44
- 0.6
45
- ]
46
- },
47
- "pooled_unseen_16_40d": {
48
- "passed": 0,
49
- "n": 1200,
50
- "accuracy": 0.0,
51
- "wilson_95_ci": [
52
- 0.0,
53
- 0.3
54
- ]
55
- },
56
- "pooled_unseen_16_50d": {
57
- "passed": 0,
58
- "n": 1600,
59
- "accuracy": 0.0,
60
- "wilson_95_ci": [
61
- 0.0,
62
- 0.2
63
- ]
64
- },
65
- "per_length": {
66
- "8": {
67
- "digits": 8,
68
- "n": 200,
69
- "passed": 0,
70
- "accuracy": 0.0,
71
- "wilson_95_ci": [
72
- 0.0,
73
- 1.9
74
- ],
75
- "elapsed_sec": 1.39
76
- },
77
- "12": {
78
- "digits": 12,
79
- "n": 200,
80
- "passed": 0,
81
- "accuracy": 0.0,
82
- "wilson_95_ci": [
83
- 0.0,
84
- 1.9
85
- ],
86
- "elapsed_sec": 0.85
87
- },
88
- "15": {
89
- "digits": 15,
90
- "n": 200,
91
- "passed": 0,
92
- "accuracy": 0.0,
93
- "wilson_95_ci": [
94
- 0.0,
95
- 1.9
96
- ],
97
- "elapsed_sec": 1.2
98
- },
99
- "16": {
100
- "digits": 16,
101
- "n": 200,
102
- "passed": 0,
103
- "accuracy": 0.0,
104
- "wilson_95_ci": [
105
- 0.0,
106
- 1.9
107
- ],
108
- "elapsed_sec": 1.28
109
- },
110
- "20": {
111
- "digits": 20,
112
- "n": 200,
113
- "passed": 0,
114
- "accuracy": 0.0,
115
- "wilson_95_ci": [
116
- 0.0,
117
- 1.9
118
- ],
119
- "elapsed_sec": 1.75
120
- },
121
- "25": {
122
- "digits": 25,
123
- "n": 200,
124
- "passed": 0,
125
- "accuracy": 0.0,
126
- "wilson_95_ci": [
127
- 0.0,
128
- 1.9
129
- ],
130
- "elapsed_sec": 2.53
131
- },
132
- "30": {
133
- "digits": 30,
134
- "n": 200,
135
- "passed": 0,
136
- "accuracy": 0.0,
137
- "wilson_95_ci": [
138
- 0.0,
139
- 1.9
140
- ],
141
- "elapsed_sec": 0.39
142
- },
143
- "35": {
144
- "digits": 35,
145
- "n": 200,
146
- "passed": 0,
147
- "accuracy": 0.0,
148
- "wilson_95_ci": [
149
- 0.0,
150
- 1.9
151
- ],
152
- "elapsed_sec": 0.37
153
- },
154
- "40": {
155
- "digits": 40,
156
- "n": 200,
157
- "passed": 0,
158
- "accuracy": 0.0,
159
- "wilson_95_ci": [
160
- 0.0,
161
- 1.9
162
- ],
163
- "elapsed_sec": 0.44
164
- },
165
- "45": {
166
- "digits": 45,
167
- "n": 200,
168
- "passed": 0,
169
- "accuracy": 0.0,
170
- "wilson_95_ci": [
171
- 0.0,
172
- 1.9
173
- ],
174
- "elapsed_sec": 0.54
175
- },
176
- "50": {
177
- "digits": 50,
178
- "n": 200,
179
- "passed": 0,
180
- "accuracy": 0.0,
181
- "wilson_95_ci": [
182
- 0.0,
183
- 1.9
184
- ],
185
- "elapsed_sec": 0.63
186
- },
187
- "55": {
188
- "digits": 55,
189
- "n": 200,
190
- "passed": 0,
191
- "accuracy": 0.0,
192
- "wilson_95_ci": [
193
- 0.0,
194
- 1.9
195
- ],
196
- "elapsed_sec": 0.65
197
- },
198
- "60": {
199
- "digits": 60,
200
- "n": 200,
201
- "passed": 0,
202
- "accuracy": 0.0,
203
- "wilson_95_ci": [
204
- 0.0,
205
- 1.9
206
- ],
207
- "elapsed_sec": 0.89
208
- },
209
- "70": {
210
- "digits": 70,
211
- "n": 200,
212
- "passed": 0,
213
- "accuracy": 0.0,
214
- "wilson_95_ci": [
215
- 0.0,
216
- 1.9
217
- ],
218
- "elapsed_sec": 0.7
219
- },
220
- "80": {
221
- "digits": 80,
222
- "n": 200,
223
- "passed": 0,
224
- "accuracy": 0.0,
225
- "wilson_95_ci": [
226
- 0.0,
227
- 1.9
228
- ],
229
- "elapsed_sec": 0.84
230
- },
231
- "100": {
232
- "digits": 100,
233
- "n": 200,
234
- "passed": 0,
235
- "accuracy": 0.0,
236
- "wilson_95_ci": [
237
- 0.0,
238
- 1.9
239
- ],
240
- "elapsed_sec": 0.37
241
- }
242
- }
243
- },
244
- "1d_shift512_15d_1024": {
245
- "meta": {
246
- "id": "1d_shift512_15d_1024",
247
- "label": "1D-RoPE + Random Offset (1-15d, T=1024, Shift=512)",
248
- "pos_mode": "1d_shift",
249
- "k_shift": 512,
250
- "final_loss": 0.09582,
251
- "train_sec": 237.03
252
- },
253
- "pooled_seen_8_15d": {
254
- "passed": 0,
255
- "n": 600,
256
- "accuracy": 0.0,
257
- "wilson_95_ci": [
258
- 0.0,
259
- 0.6
260
- ]
261
- },
262
- "pooled_unseen_16_40d": {
263
- "passed": 0,
264
- "n": 1200,
265
- "accuracy": 0.0,
266
- "wilson_95_ci": [
267
- 0.0,
268
- 0.3
269
- ]
270
- },
271
- "pooled_unseen_16_50d": {
272
- "passed": 0,
273
- "n": 1600,
274
- "accuracy": 0.0,
275
- "wilson_95_ci": [
276
- 0.0,
277
- 0.2
278
- ]
279
- },
280
- "per_length": {
281
- "8": {
282
- "digits": 8,
283
- "n": 200,
284
- "passed": 0,
285
- "accuracy": 0.0,
286
- "wilson_95_ci": [
287
- 0.0,
288
- 1.9
289
- ],
290
- "elapsed_sec": 0.46
291
- },
292
- "12": {
293
- "digits": 12,
294
- "n": 200,
295
- "passed": 0,
296
- "accuracy": 0.0,
297
- "wilson_95_ci": [
298
- 0.0,
299
- 1.9
300
- ],
301
- "elapsed_sec": 0.84
302
- },
303
- "15": {
304
- "digits": 15,
305
- "n": 200,
306
- "passed": 0,
307
- "accuracy": 0.0,
308
- "wilson_95_ci": [
309
- 0.0,
310
- 1.9
311
- ],
312
- "elapsed_sec": 1.19
313
- },
314
- "16": {
315
- "digits": 16,
316
- "n": 200,
317
- "passed": 0,
318
- "accuracy": 0.0,
319
- "wilson_95_ci": [
320
- 0.0,
321
- 1.9
322
- ],
323
- "elapsed_sec": 1.17
324
- },
325
- "20": {
326
- "digits": 20,
327
- "n": 200,
328
- "passed": 0,
329
- "accuracy": 0.0,
330
- "wilson_95_ci": [
331
- 0.0,
332
- 1.9
333
- ],
334
- "elapsed_sec": 1.66
335
- },
336
- "25": {
337
- "digits": 25,
338
- "n": 200,
339
- "passed": 0,
340
- "accuracy": 0.0,
341
- "wilson_95_ci": [
342
- 0.0,
343
- 1.9
344
- ],
345
- "elapsed_sec": 2.38
346
- },
347
- "30": {
348
- "digits": 30,
349
- "n": 200,
350
- "passed": 0,
351
- "accuracy": 0.0,
352
- "wilson_95_ci": [
353
- 0.0,
354
- 1.9
355
- ],
356
- "elapsed_sec": 3.0
357
- },
358
- "35": {
359
- "digits": 35,
360
- "n": 200,
361
- "passed": 0,
362
- "accuracy": 0.0,
363
- "wilson_95_ci": [
364
- 0.0,
365
- 1.9
366
- ],
367
- "elapsed_sec": 3.87
368
- },
369
- "40": {
370
- "digits": 40,
371
- "n": 200,
372
- "passed": 0,
373
- "accuracy": 0.0,
374
- "wilson_95_ci": [
375
- 0.0,
376
- 1.9
377
- ],
378
- "elapsed_sec": 4.56
379
- },
380
- "45": {
381
- "digits": 45,
382
- "n": 200,
383
- "passed": 0,
384
- "accuracy": 0.0,
385
- "wilson_95_ci": [
386
- 0.0,
387
- 1.9
388
- ],
389
- "elapsed_sec": 5.78
390
- },
391
- "50": {
392
- "digits": 50,
393
- "n": 200,
394
- "passed": 0,
395
- "accuracy": 0.0,
396
- "wilson_95_ci": [
397
- 0.0,
398
- 1.9
399
- ],
400
- "elapsed_sec": 6.28
401
- },
402
- "55": {
403
- "digits": 55,
404
- "n": 200,
405
- "passed": 0,
406
- "accuracy": 0.0,
407
- "wilson_95_ci": [
408
- 0.0,
409
- 1.9
410
- ],
411
- "elapsed_sec": 8.48
412
- },
413
- "60": {
414
- "digits": 60,
415
- "n": 200,
416
- "passed": 0,
417
- "accuracy": 0.0,
418
- "wilson_95_ci": [
419
- 0.0,
420
- 1.9
421
- ],
422
- "elapsed_sec": 9.89
423
- },
424
- "70": {
425
- "digits": 70,
426
- "n": 200,
427
- "passed": 0,
428
- "accuracy": 0.0,
429
- "wilson_95_ci": [
430
- 0.0,
431
- 1.9
432
- ],
433
- "elapsed_sec": 14.4
434
- },
435
- "80": {
436
- "digits": 80,
437
- "n": 200,
438
- "passed": 0,
439
- "accuracy": 0.0,
440
- "wilson_95_ci": [
441
- 0.0,
442
- 1.9
443
- ],
444
- "elapsed_sec": 16.89
445
- },
446
- "100": {
447
- "digits": 100,
448
- "n": 200,
449
- "passed": 0,
450
- "accuracy": 0.0,
451
- "wilson_95_ci": [
452
- 0.0,
453
- 1.9
454
- ],
455
- "elapsed_sec": 26.57
456
- }
457
- }
458
- },
459
- "1d_ruoss_15d_1024": {
460
- "meta": {
461
- "id": "1d_ruoss_15d_1024",
462
- "label": "1D-RoPE + Randomized Positions (Ruoss et al. 2023, 1-15d, M=1024)",
463
- "pos_mode": "1d_ruoss",
464
- "k_shift": 0,
465
- "final_loss": 0.23179,
466
- "train_sec": 271.63
467
- },
468
- "pooled_seen_8_15d": {
469
- "passed": 0,
470
- "n": 600,
471
- "accuracy": 0.0,
472
- "wilson_95_ci": [
473
- 0.0,
474
- 0.6
475
- ]
476
- },
477
- "pooled_unseen_16_40d": {
478
- "passed": 0,
479
- "n": 1200,
480
- "accuracy": 0.0,
481
- "wilson_95_ci": [
482
- 0.0,
483
- 0.3
484
- ]
485
- },
486
- "pooled_unseen_16_50d": {
487
- "passed": 0,
488
- "n": 1600,
489
- "accuracy": 0.0,
490
- "wilson_95_ci": [
491
- 0.0,
492
- 0.2
493
- ]
494
- },
495
- "per_length": {
496
- "8": {
497
- "digits": 8,
498
- "n": 200,
499
- "passed": 0,
500
- "accuracy": 0.0,
501
- "wilson_95_ci": [
502
- 0.0,
503
- 1.9
504
- ],
505
- "elapsed_sec": 0.58
506
- },
507
- "12": {
508
- "digits": 12,
509
- "n": 200,
510
- "passed": 0,
511
- "accuracy": 0.0,
512
- "wilson_95_ci": [
513
- 0.0,
514
- 1.9
515
- ],
516
- "elapsed_sec": 0.81
517
- },
518
- "15": {
519
- "digits": 15,
520
- "n": 200,
521
- "passed": 0,
522
- "accuracy": 0.0,
523
- "wilson_95_ci": [
524
- 0.0,
525
- 1.9
526
- ],
527
- "elapsed_sec": 1.11
528
- },
529
- "16": {
530
- "digits": 16,
531
- "n": 200,
532
- "passed": 0,
533
- "accuracy": 0.0,
534
- "wilson_95_ci": [
535
- 0.0,
536
- 1.9
537
- ],
538
- "elapsed_sec": 1.18
539
- },
540
- "20": {
541
- "digits": 20,
542
- "n": 200,
543
- "passed": 0,
544
- "accuracy": 0.0,
545
- "wilson_95_ci": [
546
- 0.0,
547
- 1.9
548
- ],
549
- "elapsed_sec": 1.72
550
- },
551
- "25": {
552
- "digits": 25,
553
- "n": 200,
554
- "passed": 0,
555
- "accuracy": 0.0,
556
- "wilson_95_ci": [
557
- 0.0,
558
- 1.9
559
- ],
560
- "elapsed_sec": 2.32
561
- },
562
- "30": {
563
- "digits": 30,
564
- "n": 200,
565
- "passed": 0,
566
- "accuracy": 0.0,
567
- "wilson_95_ci": [
568
- 0.0,
569
- 1.9
570
- ],
571
- "elapsed_sec": 3.05
572
- },
573
- "35": {
574
- "digits": 35,
575
- "n": 200,
576
- "passed": 0,
577
- "accuracy": 0.0,
578
- "wilson_95_ci": [
579
- 0.0,
580
- 1.9
581
- ],
582
- "elapsed_sec": 3.78
583
- },
584
- "40": {
585
- "digits": 40,
586
- "n": 200,
587
- "passed": 0,
588
- "accuracy": 0.0,
589
- "wilson_95_ci": [
590
- 0.0,
591
- 1.9
592
- ],
593
- "elapsed_sec": 4.09
594
- },
595
- "45": {
596
- "digits": 45,
597
- "n": 200,
598
- "passed": 0,
599
- "accuracy": 0.0,
600
- "wilson_95_ci": [
601
- 0.0,
602
- 1.9
603
- ],
604
- "elapsed_sec": 4.98
605
- },
606
- "50": {
607
- "digits": 50,
608
- "n": 200,
609
- "passed": 0,
610
- "accuracy": 0.0,
611
- "wilson_95_ci": [
612
- 0.0,
613
- 1.9
614
- ],
615
- "elapsed_sec": 7.21
616
- },
617
- "55": {
618
- "digits": 55,
619
- "n": 200,
620
- "passed": 0,
621
- "accuracy": 0.0,
622
- "wilson_95_ci": [
623
- 0.0,
624
- 1.9
625
- ],
626
- "elapsed_sec": 7.39
627
- },
628
- "60": {
629
- "digits": 60,
630
- "n": 200,
631
- "passed": 0,
632
- "accuracy": 0.0,
633
- "wilson_95_ci": [
634
- 0.0,
635
- 1.9
636
- ],
637
- "elapsed_sec": 10.29
638
- },
639
- "70": {
640
- "digits": 70,
641
- "n": 200,
642
- "passed": 0,
643
- "accuracy": 0.0,
644
- "wilson_95_ci": [
645
- 0.0,
646
- 1.9
647
- ],
648
- "elapsed_sec": 14.0
649
- },
650
- "80": {
651
- "digits": 80,
652
- "n": 200,
653
- "passed": 0,
654
- "accuracy": 0.0,
655
- "wilson_95_ci": [
656
- 0.0,
657
- 1.9
658
- ],
659
- "elapsed_sec": 16.05
660
- },
661
- "100": {
662
- "digits": 100,
663
- "n": 200,
664
- "passed": 0,
665
- "accuracy": 0.0,
666
- "wilson_95_ci": [
667
- 0.0,
668
- 1.9
669
- ],
670
- "elapsed_sec": 24.49
671
- }
672
- }
673
- },
674
- "2d_shift32_15d_1024": {
675
- "meta": {
676
- "id": "2d_shift32_15d_1024",
677
- "label": "Decoupled 2D-RoPE + Shift (Ours, 1-15d, T=1024, k=32)",
678
- "pos_mode": "2d_rope",
679
- "k_shift": 32,
680
- "final_loss": 0.00597,
681
- "train_sec": 352.24
682
- },
683
- "pooled_seen_8_15d": {
684
- "passed": 417,
685
- "n": 600,
686
- "accuracy": 69.5,
687
- "wilson_95_ci": [
688
- 65.7,
689
- 73.1
690
- ]
691
- },
692
- "pooled_unseen_16_40d": {
693
- "passed": 977,
694
- "n": 1200,
695
- "accuracy": 81.42,
696
- "wilson_95_ci": [
697
- 79.1,
698
- 83.5
699
- ]
700
- },
701
- "pooled_unseen_16_50d": {
702
- "passed": 1037,
703
- "n": 1600,
704
- "accuracy": 64.81,
705
- "wilson_95_ci": [
706
- 62.4,
707
- 67.1
708
- ]
709
- },
710
- "per_length": {
711
- "8": {
712
- "digits": 8,
713
- "n": 200,
714
- "passed": 108,
715
- "accuracy": 54.0,
716
- "wilson_95_ci": [
717
- 47.1,
718
- 60.8
719
- ],
720
- "elapsed_sec": 1.99
721
- },
722
- "12": {
723
- "digits": 12,
724
- "n": 200,
725
- "passed": 137,
726
- "accuracy": 68.5,
727
- "wilson_95_ci": [
728
- 61.8,
729
- 74.5
730
- ],
731
- "elapsed_sec": 1.53
732
- },
733
- "15": {
734
- "digits": 15,
735
- "n": 200,
736
- "passed": 172,
737
- "accuracy": 86.0,
738
- "wilson_95_ci": [
739
- 80.5,
740
- 90.1
741
- ],
742
- "elapsed_sec": 2.02
743
- },
744
- "16": {
745
- "digits": 16,
746
- "n": 200,
747
- "passed": 184,
748
- "accuracy": 92.0,
749
- "wilson_95_ci": [
750
- 87.4,
751
- 95.0
752
- ],
753
- "elapsed_sec": 3.12
754
- },
755
- "20": {
756
- "digits": 20,
757
- "n": 200,
758
- "passed": 180,
759
- "accuracy": 90.0,
760
- "wilson_95_ci": [
761
- 85.1,
762
- 93.4
763
- ],
764
- "elapsed_sec": 3.89
765
- },
766
- "25": {
767
- "digits": 25,
768
- "n": 200,
769
- "passed": 184,
770
- "accuracy": 92.0,
771
- "wilson_95_ci": [
772
- 87.4,
773
- 95.0
774
- ],
775
- "elapsed_sec": 3.71
776
- },
777
- "30": {
778
- "digits": 30,
779
- "n": 200,
780
- "passed": 170,
781
- "accuracy": 85.0,
782
- "wilson_95_ci": [
783
- 79.4,
784
- 89.3
785
- ],
786
- "elapsed_sec": 5.5
787
- },
788
- "35": {
789
- "digits": 35,
790
- "n": 200,
791
- "passed": 156,
792
- "accuracy": 78.0,
793
- "wilson_95_ci": [
794
- 71.8,
795
- 83.2
796
- ],
797
- "elapsed_sec": 8.06
798
- },
799
- "40": {
800
- "digits": 40,
801
- "n": 200,
802
- "passed": 103,
803
- "accuracy": 51.5,
804
- "wilson_95_ci": [
805
- 44.6,
806
- 58.3
807
- ],
808
- "elapsed_sec": 9.16
809
- },
810
- "45": {
811
- "digits": 45,
812
- "n": 200,
813
- "passed": 60,
814
- "accuracy": 30.0,
815
- "wilson_95_ci": [
816
- 24.1,
817
- 36.7
818
- ],
819
- "elapsed_sec": 11.37
820
- },
821
- "50": {
822
- "digits": 50,
823
- "n": 200,
824
- "passed": 0,
825
- "accuracy": 0.0,
826
- "wilson_95_ci": [
827
- 0.0,
828
- 1.9
829
- ],
830
- "elapsed_sec": 11.7
831
- },
832
- "55": {
833
- "digits": 55,
834
- "n": 200,
835
- "passed": 0,
836
- "accuracy": 0.0,
837
- "wilson_95_ci": [
838
- 0.0,
839
- 1.9
840
- ],
841
- "elapsed_sec": 12.22
842
- },
843
- "60": {
844
- "digits": 60,
845
- "n": 200,
846
- "passed": 0,
847
- "accuracy": 0.0,
848
- "wilson_95_ci": [
849
- 0.0,
850
- 1.9
851
- ],
852
- "elapsed_sec": 14.38
853
- },
854
- "70": {
855
- "digits": 70,
856
- "n": 200,
857
- "passed": 0,
858
- "accuracy": 0.0,
859
- "wilson_95_ci": [
860
- 0.0,
861
- 1.9
862
- ],
863
- "elapsed_sec": 23.93
864
- },
865
- "80": {
866
- "digits": 80,
867
- "n": 200,
868
- "passed": 0,
869
- "accuracy": 0.0,
870
- "wilson_95_ci": [
871
- 0.0,
872
- 1.9
873
- ],
874
- "elapsed_sec": 30.33
875
- },
876
- "100": {
877
- "digits": 100,
878
- "n": 200,
879
- "passed": 0,
880
- "accuracy": 0.0,
881
- "wilson_95_ci": [
882
- 0.0,
883
- 1.9
884
- ],
885
- "elapsed_sec": 35.26
886
- }
887
- }
888
- }
889
- }
890
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
cbt_2d_arithmetic_20_70d.json → evaluation/cbt_2d/addition_20-70_digits.json RENAMED
File without changes
cbt_2d_arithmetic_4_16d.json → evaluation/cbt_2d/addition_4-16_digits.json RENAMED
File without changes
cbt_2d_ood.json → evaluation/cbt_2d/tools_out_of_distribution.json RENAMED
File without changes
cbt_2d_phrasing.json → evaluation/cbt_2d/tools_rephrased.json RENAMED
File without changes
cbt_2d_unseen_copy.json → evaluation/cbt_2d/tools_unseen_entities.json RENAMED
File without changes
cbt1d_v5_arithmetic.json → evaluation/twin_1d/addition_20-70_digits.json RENAMED
File without changes
cbt_1d_twin_arithmetic_4_16d.json → evaluation/twin_1d/addition_4-16_digits.json RENAMED
File without changes
champion_v3_ood.json → evaluation/twin_1d/tools_out_of_distribution.json RENAMED
@@ -1,5 +1,5 @@
1
  {
2
- "checkpoint": "champion_model/checkpoints/champion_v3_final.zip",
3
  "max_train_operand_digits": 25,
4
  "ood": {
5
  "total": 18,
 
1
  {
2
+ "checkpoint": "checkpoints/cbt/cbt_18m_1d_rope_R3_k45_sftv5_best.pt",
3
  "max_train_operand_digits": 25,
4
  "ood": {
5
  "total": 18,
champion_v3_phrasing.json → evaluation/twin_1d/tools_rephrased.json RENAMED
@@ -1,17 +1,17 @@
1
  {
2
- "checkpoint": "champion_model/checkpoints/champion_v3_final.zip",
3
  "max_train_operand_digits": 25,
4
  "phrasing": {
5
  "total": 26,
6
- "routing_passed": 26,
7
  "routing_ci": [
8
- 87.1,
9
- 100.0
10
  ],
11
- "exact_args_passed": 26,
12
  "exact_args_ci": [
13
- 87.1,
14
- 100.0
15
  ]
16
  }
17
  }
 
1
  {
2
+ "checkpoint": "checkpoints/cbt/cbt_18m_1d_rope_R3_k45_sftv5_best.pt",
3
  "max_train_operand_digits": 25,
4
  "phrasing": {
5
  "total": 26,
6
+ "routing_passed": 25,
7
  "routing_ci": [
8
+ 81.1,
9
+ 99.3
10
  ],
11
+ "exact_args_passed": 25,
12
  "exact_args_ci": [
13
+ 81.1,
14
+ 99.3
15
  ]
16
  }
17
  }
champion_v3_unseen_ood.json → evaluation/twin_1d/tools_unseen_entities.json RENAMED
@@ -37,7 +37,7 @@
37
  "prompt": "<USER>Who was Hypatia of Alexandria?</USER><THINK>",
38
  "expected_tool": "search",
39
  "expected_arg": "Hypatia of Alexandria",
40
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"Hypatia of Alexandria\")</CALL>",
41
  "parsed_call": "search(query=\"Hypatia of Alexandria\")",
42
  "route_ok": true,
43
  "exact_copy_ok": true
@@ -48,7 +48,7 @@
48
  "prompt": "<USER>Who was Ludwig van Beethoven?</USER><THINK>",
49
  "expected_tool": "search",
50
  "expected_arg": "Ludwig van Beethoven",
51
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"Ludwig van Beethoven\")</CALL>",
52
  "parsed_call": "search(query=\"Ludwig van Beethoven\")",
53
  "route_ok": true,
54
  "exact_copy_ok": true
@@ -59,7 +59,7 @@
59
  "prompt": "<USER>Who was Johannes Vermeer?</USER><THINK>",
60
  "expected_tool": "search",
61
  "expected_arg": "Johannes Vermeer",
62
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"Johannes Vermeer\")</CALL>",
63
  "parsed_call": "search(query=\"Johannes Vermeer\")",
64
  "route_ok": true,
65
  "exact_copy_ok": true
@@ -70,7 +70,7 @@
70
  "prompt": "<USER>Who is Ada Yonath?</USER><THINK>",
71
  "expected_tool": "search",
72
  "expected_arg": "Ada Yonath",
73
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"Ada Yonath\")</CALL>",
74
  "parsed_call": "search(query=\"Ada Yonath\")",
75
  "route_ok": true,
76
  "exact_copy_ok": true
@@ -81,7 +81,7 @@
81
  "prompt": "<USER>Who was Srinivasa Ramanujan?</USER><THINK>",
82
  "expected_tool": "search",
83
  "expected_arg": "Srinivasa Ramanujan",
84
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"Srinivasa Ramanujan\")</CALL>",
85
  "parsed_call": "search(query=\"Srinivasa Ramanujan\")",
86
  "route_ok": true,
87
  "exact_copy_ok": true
@@ -114,7 +114,7 @@
114
  "prompt": "<USER>Weather on Antananarivo</USER><THINK>",
115
  "expected_tool": "get_weather",
116
  "expected_arg": "Antananarivo",
117
- "raw_response": "The user is asking for weather conditions. Querying get_weather.</THINK><CALL>get_weather(city=\"Antananarivo\")</CALL>",
118
  "parsed_call": "get_weather(city=\"Antananarivo\")",
119
  "route_ok": true,
120
  "exact_copy_ok": true
@@ -202,7 +202,7 @@
202
  "prompt": "<USER>Explain bose einstein condensate</USER><THINK>",
203
  "expected_tool": "search",
204
  "expected_arg": "bose einstein condensate",
205
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"bose einstein condensate\")</CALL>",
206
  "parsed_call": "search(query=\"bose einstein condensate\")",
207
  "route_ok": true,
208
  "exact_copy_ok": true
@@ -235,7 +235,7 @@
235
  "prompt": "<USER>Define baklava</USER><THINK>",
236
  "expected_tool": "search",
237
  "expected_arg": "baklava",
238
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"baklava\")</CALL>",
239
  "parsed_call": "search(query=\"baklava\")",
240
  "route_ok": true,
241
  "exact_copy_ok": true
@@ -246,7 +246,7 @@
246
  "prompt": "<USER>What is shakshuka?</USER><THINK>",
247
  "expected_tool": "search",
248
  "expected_arg": "shakshuka",
249
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"shakshuka\")</CALL>",
250
  "parsed_call": "search(query=\"shakshuka\")",
251
  "route_ok": true,
252
  "exact_copy_ok": true
@@ -301,7 +301,7 @@
301
  "prompt": "<USER>Define Zyntral Krelvok</USER><THINK>",
302
  "expected_tool": "search",
303
  "expected_arg": "Zyntral Krelvok",
304
- "raw_response": "The user is asking a factual question. I will query the search tool.</THINK><CALL>search(query=\"Zyntral Krelvok\")</CALL>",
305
  "parsed_call": "search(query=\"Zyntral Krelvok\")",
306
  "route_ok": true,
307
  "exact_copy_ok": true
 
37
  "prompt": "<USER>Who was Hypatia of Alexandria?</USER><THINK>",
38
  "expected_tool": "search",
39
  "expected_arg": "Hypatia of Alexandria",
40
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Hypatia of Alexandria\")</CALL>",
41
  "parsed_call": "search(query=\"Hypatia of Alexandria\")",
42
  "route_ok": true,
43
  "exact_copy_ok": true
 
48
  "prompt": "<USER>Who was Ludwig van Beethoven?</USER><THINK>",
49
  "expected_tool": "search",
50
  "expected_arg": "Ludwig van Beethoven",
51
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Ludwig van Beethoven\")</CALL>",
52
  "parsed_call": "search(query=\"Ludwig van Beethoven\")",
53
  "route_ok": true,
54
  "exact_copy_ok": true
 
59
  "prompt": "<USER>Who was Johannes Vermeer?</USER><THINK>",
60
  "expected_tool": "search",
61
  "expected_arg": "Johannes Vermeer",
62
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Johannes Vermeer\")</CALL>",
63
  "parsed_call": "search(query=\"Johannes Vermeer\")",
64
  "route_ok": true,
65
  "exact_copy_ok": true
 
70
  "prompt": "<USER>Who is Ada Yonath?</USER><THINK>",
71
  "expected_tool": "search",
72
  "expected_arg": "Ada Yonath",
73
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Ada Yonath\")</CALL>",
74
  "parsed_call": "search(query=\"Ada Yonath\")",
75
  "route_ok": true,
76
  "exact_copy_ok": true
 
81
  "prompt": "<USER>Who was Srinivasa Ramanujan?</USER><THINK>",
82
  "expected_tool": "search",
83
  "expected_arg": "Srinivasa Ramanujan",
84
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Srinivasa Ramanujan\")</CALL>",
85
  "parsed_call": "search(query=\"Srinivasa Ramanujan\")",
86
  "route_ok": true,
87
  "exact_copy_ok": true
 
114
  "prompt": "<USER>Weather on Antananarivo</USER><THINK>",
115
  "expected_tool": "get_weather",
116
  "expected_arg": "Antananarivo",
117
+ "raw_response": "The user requests current weather information. I will query the get_weather tool.</THINK><CALL>get_weather(city=\"Antananarivo\")</CALL>",
118
  "parsed_call": "get_weather(city=\"Antananarivo\")",
119
  "route_ok": true,
120
  "exact_copy_ok": true
 
202
  "prompt": "<USER>Explain bose einstein condensate</USER><THINK>",
203
  "expected_tool": "search",
204
  "expected_arg": "bose einstein condensate",
205
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"bose einstein condensate\")</CALL>",
206
  "parsed_call": "search(query=\"bose einstein condensate\")",
207
  "route_ok": true,
208
  "exact_copy_ok": true
 
235
  "prompt": "<USER>Define baklava</USER><THINK>",
236
  "expected_tool": "search",
237
  "expected_arg": "baklava",
238
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"baklava\")</CALL>",
239
  "parsed_call": "search(query=\"baklava\")",
240
  "route_ok": true,
241
  "exact_copy_ok": true
 
246
  "prompt": "<USER>What is shakshuka?</USER><THINK>",
247
  "expected_tool": "search",
248
  "expected_arg": "shakshuka",
249
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"shakshuka\")</CALL>",
250
  "parsed_call": "search(query=\"shakshuka\")",
251
  "route_ok": true,
252
  "exact_copy_ok": true
 
301
  "prompt": "<USER>Define Zyntral Krelvok</USER><THINK>",
302
  "expected_tool": "search",
303
  "expected_arg": "Zyntral Krelvok",
304
+ "raw_response": "The user is seeking knowledge on this topic. I will search the web.</THINK><CALL>search(query=\"Zyntral Krelvok\")</CALL>",
305
  "parsed_call": "search(query=\"Zyntral Krelvok\")",
306
  "route_ok": true,
307
  "exact_copy_ok": true
interactive.py DELETED
@@ -1,329 +0,0 @@
1
- #!/usr/bin/env python3
2
- """
3
- Interactive CLI Runner for ScaledGPT (~18.95M Parameter Recurrent 2D-RoPE Transformer).
4
- Supports live interactive inference across:
5
- 1. High-digit column scratchpad arithmetic (up to 50 digits).
6
- 2. Expanded tool execution (Single-tool, Chaining, Nested JSON schemas, Error handling).
7
- 3. Multi-turn interactive dialogues.
8
- """
9
-
10
- import os
11
- import sys
12
- import re
13
- import json
14
- import time
15
- import argparse
16
- from typing import Tuple, Optional, Dict, Any
17
-
18
- import torch
19
-
20
- ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
21
- if ROOT_DIR not in sys.path:
22
- sys.path.insert(0, ROOT_DIR)
23
- THIS_DIR = os.path.dirname(os.path.abspath(__file__))
24
- if THIS_DIR not in sys.path:
25
- sys.path.insert(0, THIS_DIR)
26
-
27
- if hasattr(sys.stdout, "reconfigure"):
28
- sys.stdout.reconfigure(line_buffering=True)
29
-
30
- try:
31
- from champion_model.scaled_t4.model import ScaledGPT, ScaledGPTConfig
32
- from champion_model.scaled_t4.tokenizer import ScaledTokenizer
33
- from champion_model.scaled_t4.dataset import extract_scratchpad_answer
34
- except ImportError:
35
- from model import ScaledGPT, ScaledGPTConfig
36
- from tokenizer import ScaledTokenizer
37
- from dataset import extract_scratchpad_answer
38
-
39
-
40
- def mock_execute_tool(call_text: str) -> str:
41
- """Executes or mocks tool calls parsed from <CALL>...</CALL>."""
42
- # 1. calculator(expr="...")
43
- calc_m = re.match(r'calculator\(expr="([^"]+)"\)', call_text)
44
- if calc_m:
45
- expr = calc_m.group(1)
46
- try:
47
- allowed = re.match(r"^[\d\+\-\*\/\(\)\s\.]+$", expr)
48
- if allowed:
49
- val = eval(expr)
50
- return json.dumps({"result": val})
51
- except ZeroDivisionError:
52
- return json.dumps({"error": "ZeroDivisionError: division by zero"})
53
- except Exception as e:
54
- return json.dumps({"error": str(e)})
55
-
56
- # 2. get_weather(city="...")
57
- w_m = re.match(r'get_weather\(city="([^"]+)"\)', call_text)
58
- if w_m:
59
- city = w_m.group(1)
60
- if city.lower() in ["atlantis", "narnia"]:
61
- return json.dumps({"status": "error", "code": 404, "message": f"City '{city}' not found"})
62
- return json.dumps({"temp": 22, "condition": "Sunny", "city": city})
63
-
64
- # 3. get_time(timezone="...")
65
- t_m = re.match(r'get_time\(timezone="([^"]+)"\)', call_text)
66
- if t_m:
67
- tz = t_m.group(1)
68
- return json.dumps({"time": "14:30", "timezone": tz})
69
-
70
- # 4. unit_convert
71
- u_m = re.match(r'unit_convert\(value=([\d\.]+),\s*from_unit="([^"]+)",\s*to_unit="([^"]+)"\)', call_text)
72
- if u_m:
73
- val = float(u_m.group(1))
74
- f_u, t_u = u_m.group(2), u_m.group(3)
75
- factors = {
76
- ("km", "miles"): 0.621371,
77
- ("miles", "km"): 1.60934,
78
- ("kg", "lbs"): 2.20462,
79
- ("lbs", "kg"): 0.453592,
80
- }
81
- factor = factors.get((f_u, t_u), 1.0)
82
- return json.dumps({"value": round(val * factor, 2), "unit": t_u})
83
-
84
- # 5. database_query
85
- if call_text.startswith("database_query("):
86
- return json.dumps({"status": "success", "count": 3, "records": [{"id": 101, "status": "active"}]})
87
-
88
- # 6. create_event
89
- if call_text.startswith("create_event("):
90
- return json.dumps({"event_id": "evt_98214", "status": "confirmed"})
91
-
92
- # 7. file_manager
93
- if call_text.startswith("file_manager("):
94
- return json.dumps({"status": "ok", "bytes_written": 64})
95
-
96
- # 8. fetch_api
97
- if call_text.startswith("fetch_api("):
98
- return json.dumps({"status": 200, "data": {"status": "healthy"}})
99
-
100
- # 9. search(query="...")
101
- s_m = re.match(r'search\(query="([^"]+)"\)', call_text)
102
- if s_m:
103
- query = s_m.group(1).strip()
104
- q_lower = query.lower()
105
-
106
- # Offline dictionary lookup for instant, deterministic testing
107
- kb = {
108
- "tomato": "the edible berry of the plant Solanum lycopersicum, commonly prepared as a culinary vegetable.",
109
- "tomatoes": "the edible berry of the plant Solanum lycopersicum, commonly prepared as a culinary vegetable.",
110
- "apple": "a round, edible fruit produced by an apple tree (Malus domestica), cultivated worldwide.",
111
- "apples": "a round, edible fruit produced by an apple tree (Malus domestica), cultivated worldwide.",
112
- "potato": "a starchy, tuberous root crop of the nightshade family Solanaceae.",
113
- "potatoes": "a starchy, tuberous root crop of the nightshade family Solanaceae.",
114
- "coffee": "a brewed drink prepared from roasted coffee beans of flowering Coffea species.",
115
- "banana": "an elongated, edible fruit produced by several kinds of large herbaceous plants in the genus Musa.",
116
- "bananas": "an elongated, edible fruit produced by several kinds of large herbaceous plants in the genus Musa.",
117
- "gravity": "a fundamental interaction causing mutual attraction between all things with mass or energy.",
118
- "photosynthesis": "a process used by plants to convert light energy into chemical energy stored in glucose.",
119
- "atom": "the basic unit of a chemical element, consisting of a dense central nucleus surrounded by electrons.",
120
- "python creator": "Guido van Rossum, a Dutch programmer.",
121
- "speed of light": "approximately 299,792 kilometers per second in a vacuum.",
122
- "mount everest": "Earth's highest mountain above sea level, located in the Himalayas.",
123
- }
124
-
125
- for k, v in kb.items():
126
- if k in q_lower or q_lower in k:
127
- return json.dumps({"snippet": v})
128
-
129
- # Try live Wikipedia REST API if network is available
130
- try:
131
- import urllib.request
132
- import urllib.parse
133
- clean_q = urllib.parse.quote(query.replace(" ", "_"))
134
- url = f"https://en.wikipedia.org/api/rest_v1/page/summary/{clean_q}"
135
- req = urllib.request.Request(url, headers={"User-Agent": "ScaledGPT-Search/1.0"})
136
- with urllib.request.urlopen(req, timeout=2.5) as resp:
137
- data = json.loads(resp.read().decode("utf-8"))
138
- if "extract" in data and data["extract"]:
139
- return json.dumps({"snippet": data["extract"][:220]})
140
- except Exception:
141
- pass
142
-
143
- return json.dumps({"snippet": f"{query.capitalize()} is an entity and topic in verified knowledge sources."})
144
-
145
- # Fallback
146
- return json.dumps({"status": "executed", "call": call_text})
147
-
148
-
149
- def print_human_readable_output(text: str, is_math: bool = False):
150
- """Formats internal model output into clean, human-readable terminal blocks."""
151
- print("\n" + "-" * 80)
152
- if is_math:
153
- m = re.search(r"(?:<BOS>)?(.*?)=(?:<THINK>)?(.*?)(?:</THINK>=)(.*?)(?:<EOS>|$)", text)
154
- if m:
155
- problem = m.group(1).strip()
156
- scratch = m.group(2).strip()
157
- ans = m.group(3).strip()
158
- print(f"Problem: {problem} = ?")
159
- print(f"Scratchpad: {scratch}")
160
- print(f"Answer: {ans}")
161
- try:
162
- if "+" in problem:
163
- parts = problem.split("+")
164
- if len(parts) == 2 and parts[0].isdigit() and parts[1].isdigit():
165
- expected = int(parts[0]) + int(parts[1])
166
- status = "[PASS]" if str(expected) == ans else "[FAIL]"
167
- print(f"Validation: {status} (Expected: {expected})")
168
- except Exception:
169
- pass
170
- else:
171
- print(f"Result: {text}")
172
- else:
173
- think_m = re.search(r"<THINK>(.*?)</THINK>", text)
174
- call_m = re.search(r"<CALL>(.*?)</CALL>", text)
175
- result_m = re.search(r"<RESULT>(.*?)</RESULT>", text)
176
- answer_m = re.search(r"<ANSWER>(.*?)</ANSWER>", text)
177
-
178
- if think_m:
179
- print(f"Reasoning: {think_m.group(1).strip()}")
180
- if call_m:
181
- print(f"Tool Call: {call_m.group(1).strip()}")
182
- if result_m:
183
- print(f"Tool Result: {result_m.group(1).strip()}")
184
- if answer_m:
185
- print(f"Response: {answer_m.group(1).strip()}")
186
- if not any([think_m, call_m, result_m, answer_m]):
187
- print(f"Response: {text}")
188
-
189
- print("-" * 80)
190
- print(f"[RAW TOKENS] {text}\n", flush=True)
191
-
192
-
193
- def run_math_prompt(model: ScaledGPT, tokenizer: ScaledTokenizer, prompt: str, device: torch.device) -> str:
194
- """Processes a multi-digit arithmetic expression."""
195
- clean_prompt = prompt.strip()
196
- if not clean_prompt.endswith("="):
197
- clean_prompt += "="
198
- full_prompt = clean_prompt + "<THINK>"
199
-
200
- tokens = tokenizer.encode(full_prompt, add_bos=True, add_eos=False)
201
- x = torch.tensor([tokens], dtype=torch.long, device=device)
202
-
203
- with torch.no_grad():
204
- out = model.generate(x, max_new_tokens=700, temperature=0.0, stop_tokens=[tokenizer.eos_id])
205
-
206
- gen_text = tokenizer.decode(out[0].tolist(), skip_special=False)
207
- print_human_readable_output(gen_text, is_math=True)
208
- return gen_text
209
-
210
-
211
- def run_dialogue_prompt(model: ScaledGPT, tokenizer: ScaledTokenizer, user_input: str, device: torch.device) -> str:
212
- """Processes a user dialogue turn with live tool execution."""
213
- prompt = f"<USER>{user_input}</USER><THINK>"
214
- tokens = tokenizer.encode(prompt, add_bos=True, add_eos=False)
215
- x = torch.tensor([tokens], dtype=torch.long, device=device)
216
-
217
- with torch.no_grad():
218
- stop_ids = [tokenizer.call_e_id, tokenizer.answer_e_id, tokenizer.eos_id]
219
- out = model.generate(x, max_new_tokens=256, temperature=0.0, stop_tokens=stop_ids)
220
-
221
- gen_text = tokenizer.decode(out[0].tolist(), skip_special=False)
222
-
223
- if "<CALL>" in gen_text and "</CALL>" in gen_text:
224
- call_part = gen_text.split("<CALL>")[1].split("</CALL>")[0]
225
- result_str = mock_execute_tool(call_part)
226
-
227
- # Continue generation with the tool result (append tokens directly to avoid double BOS)
228
- res_suffix = f"<RESULT>{result_str}</RESULT>"
229
- res_tokens = tokenizer.encode(res_suffix, add_bos=False, add_eos=False)
230
- c_x = torch.cat([out, torch.tensor([res_tokens], dtype=torch.long, device=device)], dim=1)
231
-
232
- with torch.no_grad():
233
- out2 = model.generate(c_x, max_new_tokens=256, temperature=0.0, stop_tokens=[tokenizer.answer_e_id, tokenizer.eos_id])
234
-
235
- full_dialogue = tokenizer.decode(out2[0].tolist(), skip_special=False)
236
- else:
237
- full_dialogue = gen_text
238
-
239
- print_human_readable_output(full_dialogue, is_math=False)
240
- return full_dialogue
241
-
242
-
243
- def main():
244
- parser = argparse.ArgumentParser(description="ScaledGPT Interactive CLI")
245
- parser.add_argument("--ckpt", type=str, default="checkpoints/cbt/cbt_18m_2d_rope_R3_k45_sft.pt", help="Path to checkpoint")
246
- parser.add_argument("--prompt", type=str, default=None, help="Execute a single prompt and exit")
247
- parser.add_argument("--threads", type=int, default=8, help="Number of CPU threads for inference")
248
- args = parser.parse_args()
249
-
250
- device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
251
- if device.type == "cpu":
252
- torch.set_num_threads(args.threads)
253
- tokenizer = ScaledTokenizer()
254
- config = ScaledGPTConfig(vocab_size=tokenizer.vocab_size)
255
- model = ScaledGPT(config, tokenizer=tokenizer).to(device)
256
-
257
- # Search candidate paths for the trained checkpoint
258
- candidates = [
259
- args.ckpt,
260
- os.path.join(ROOT_DIR, args.ckpt),
261
- os.path.join(ROOT_DIR, "checkpoints/cbt/cbt_18m_2d_rope_R3_k45_sft.pt"),
262
- os.path.join(ROOT_DIR, "champion_final.pt"),
263
- os.path.join(ROOT_DIR, "checkpoints/cbt/cbt_18m_2d_rope_R3_k32_base.pt"),
264
- os.path.join(ROOT_DIR, "checkpoints/cbt/cbt_18m_2d_rope_R3_k32_pretrain.pt"),
265
- ]
266
- resolved_path = None
267
- for p in candidates:
268
- if os.path.exists(p) and not os.path.isdir(p):
269
- resolved_path = p
270
- break
271
-
272
- if resolved_path is None:
273
- try:
274
- from huggingface_hub import hf_hub_download
275
- print("[INFO] Local weights not found. Automatically pulling champion weights from Hugging Face (Neithanvarvas/CBT)...")
276
- target_dir = os.path.join(ROOT_DIR, "checkpoints", "cbt")
277
- resolved_path = hf_hub_download(repo_id="Neithanvarvas/CBT", filename="champion_final.pt", local_dir=target_dir)
278
- local_path = os.path.join(ROOT_DIR, "checkpoints/cbt/cbt_18m_2d_rope_R3_k45_sft.pt")
279
- os.replace(resolved_path, local_path)
280
- resolved_path = local_path
281
- print(f"[PASS] Successfully pulled weights to: {resolved_path}")
282
- except Exception as e:
283
- print(f"[WARN] Automatic Hugging Face pull unsuccessful ({e}).")
284
-
285
- if resolved_path is not None:
286
- ckpt = torch.load(resolved_path, map_location=device, weights_only=False)
287
- model.load_state_dict(ckpt["model_state_dict"])
288
- step_info = f" (Step {ckpt.get('step')})" if 'step' in ckpt else ""
289
- print(f"[INFO] Successfully loaded trained checkpoint{step_info} from: {resolved_path}")
290
- else:
291
- print(f"[WARN] Running with randomly initialized weights.")
292
-
293
- model.eval()
294
-
295
- if args.prompt:
296
- user_input = args.prompt.strip()
297
- is_math = bool(re.match(r"^[\d\s\+\-\=]+$", user_input) and ("+" in user_input or "-" in user_input))
298
- if is_math:
299
- run_math_prompt(model, tokenizer, user_input, device)
300
- else:
301
- run_dialogue_prompt(model, tokenizer, user_input, device)
302
- return
303
-
304
- print("\n" + "=" * 80)
305
- print("CHARACTER BASED TRANSFORMER (CBT, ~18.95M) INTERACTIVE TERMINAL")
306
- print("Type an arithmetic equation (e.g. '123456789+987654321=') or ask a question.")
307
- print("Type 'exit' or 'quit' to terminate.")
308
- print("=" * 80 + "\n")
309
-
310
- while True:
311
- try:
312
- user_input = input("User >> ").strip()
313
- except (EOFError, KeyboardInterrupt):
314
- print("\n[INFO] Exiting interactive CLI.")
315
- break
316
-
317
- if not user_input or user_input.lower() in ["exit", "quit"]:
318
- break
319
-
320
- # Check if input is arithmetic (contains only digits and + or -)
321
- is_math = bool(re.match(r"^[\d\s\+\-\=]+$", user_input) and ("+" in user_input or "-" in user_input))
322
- if is_math:
323
- run_math_prompt(model, tokenizer, user_input, device)
324
- else:
325
- run_dialogue_prompt(model, tokenizer, user_input, device)
326
-
327
-
328
- if __name__ == "__main__":
329
- main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
champion_final.pt → model.safetensors RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c6fab36eaadbc402441691ded7ed9e26990f3718e6cefcb5a889691b7fc3b9e5
3
- size 75799812
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:44ffcb10b8b4bc31fa7ca7d8a8848c8ee0cbe8c4f2fff9ced9898787b7ce997d
3
+ size 76010552
RESEARCH_PAPER.md → paper/CBT-18.95M.md RENAMED
@@ -4,21 +4,38 @@
4
 
5
  ## Abstract
6
 
7
- We set out to train a small autoregressive language model that adds multi-digit integers correctly at lengths it has not seen. A standard 18.91M-parameter decoder with one-dimensional rotary positions (1D-RoPE), trained on 1-12-digit sums with a column scratchpad, solves short sums but scores 0/30 at every unseen length we tested (14-35 digits), with either a character or a 50k-token BPE vocabulary; three alternative architectures (looped, hierarchical, hybrid) do the same, and two others did not fit the task. Giving each token a two-dimensional coordinate, the decimal place $p$ and the step $\tau$ within a column, through Decoupled 2D-RoPE changes the outcome: a 1.57M-parameter model trained on the same 1-12-digit range is exact on 150/150 sums of 14-25 digits and on 47/60 sums of 30-35 digits, and in controlled 100k-parameter runs the extrapolation limit moves with the coordinate range seen in training ($p_{\max} = N_{\text{train}} + k$). Our interpretation is that the coordinates turn place-value alignment into one fixed relation ($\Delta p = 0$) that does not change with length, so the model learns it once. Character-level models have no meaning-bearing subword embeddings and must assemble every word from characters, which prior work finds hard. We therefore asked whether the same relational layout, with $p$ indexing the dialogue segment and $\tau$ the character within it, could support a compact character-level model that also handles words. The resulting Character-Based Transformer (CBT-18.95M) is exact on 210/210 random sums of 26-55 digits, all longer than any sum in training (254/270 up to 70 digits), and exact on all 74 prompts of three tool-use suites (26/26 phrasing, 18/18 out-of-distribution, and 30/30 copies of entities absent from the training corpus, against 4/30 for the pre-fine-tuning base). Its final fine-tuning data was broadened after earlier failures were seen, so the unseen-entity set is the cleaner evidence. To attribute these results we trained a 1D-RoPE twin with the same architecture, data and step count. It scores 0/270 on every unseen length from 26 to 70 digits, but matches the 2D model on tool use (73/74 exact calls against 74/74). The 2D coordinates are therefore responsible for length generalization, while the tool-use and copying results are not attributable to them.
8
 
9
  ## 1. Introduction
10
 
11
  Small Transformers trained from scratch learn multi-digit addition within their training lengths but fail beyond them [1, 2, 3, 4, 5]. Addition is a convenient test of whether a model has learned an algorithm rather than a lookup over seen lengths: the procedure is a small finite-state machine applied once per column, and the answer is exactly checkable at any length. This paper reports a sequence of experiments that began with that question and ended with a compact character-level model. We present them in the order we ran them.
12
 
13
- 1. **A standard language model.** We trained an 18.91M-parameter decoder with 1D-RoPE [6] on 1-12-digit sums with an explicit column scratchpad [7, 8], using a character vocabulary and, separately, a 50k BPE vocabulary. It solved 4- and 8-digit sums, scored at most 14/30 at 12 digits, and scored 0/30 at every unseen length (Section 3).
14
- 2. **Other architectures.** Looped, hierarchical, hybrid recurrent, harmonic-register-machine and Abacus-looped variants [9, 10, 5], trained the same way, either scored 0/30 on every length from 12 digits up (three models) or did not fit the task (two models), and randomized 1D positions [11] did not help a 100k model (Section 4).
15
- 3. **Two-dimensional coordinates.** Assigning each token a coordinate $(p, \tau)$, place value and step within the column, and rotating queries and keys by both (Decoupled 2D-RoPE) made the models extrapolate. The extrapolation limit tracks the coordinate range seen in training, which suggests the model learns a fixed relation between positions of equal place value (Section 5).
16
- 4. **From digits to characters.** Character-level models must compose words from characters, which is a known weakness [12, 13, 14]. We asked whether the same relational layout helps a character model with words, and built the best small model we could around it: CBT-18.95M (Sections 6-8).
17
- 5. **A 1D control.** To find out which results the layout is responsible for, we trained a twin with standard 1D-RoPE on identical data. It does not extrapolate, but it matches the 2D model on tool use (Section 9).
18
 
19
- **Contributions.** (i) A re-evaluation of nine 1D-position checkpoints (standard, BPE and alternative architectures) on a common random-sum benchmark, none of which solves any unseen length. (ii) Evidence from controlled ablations that 2D place-value coordinates are the factor that enables extrapolation, and that the extrapolation limit is set by the trained coordinate range. (iii) CBT-18.95M, an 18.9M-parameter character model that combines arithmetic extrapolation with tool calling and entity copying. (iv) A matched 1D-RoPE twin of CBT-18.95M, which shows that the 2D coordinates account for the arithmetic extrapolation (254/270 against 0/270 on unseen lengths) and not for the tool-use results. All results are single training runs; test sets and their sizes are given with every number.
20
 
21
- ## 2. Task and Evaluation Protocol
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
 
23
  **Addition as a state machine.** For operands with digits $a_i, b_i$ (column $i=0$ is the units digit) the sum is produced by
24
 
@@ -28,9 +45,9 @@ with $c_{-1}=0$. The same 200-entry transition table applies at every column, so
28
 
29
  **Scratchpad format.** Character models are trained on `A+B=<THINK>` followed by one block `(c+a+b)=c's` per column, from the units column up, then `</THINK>=S`. For example, `47+85=<THINK>(0+7+5)=12(1+4+8)=13</THINK>=132`. BPE models use the same steps in words (`Column 0: 7+5=12, digit 2, carry 1. ...`). The loss is masked on the prompt `A+B=`.
30
 
31
- **Benchmark.** For each length $d$ we draw 30 problems with two $d$-digit operands from a stratified generator: 70% uniform random digits, 15% dense carries (all digits 5-9), and 15% long carry cascades. Decoding is greedy. A problem counts as correct only if the answer after `</THINK>=` matches the true sum exactly. We report $k/n$ with 95% Wilson intervals. Lengths inside the training range are labelled *seen*. Baseline traces up to 35 digits fit in their 512-token context, so truncation does not explain the failures below, with two exceptions noted where they occur: the place-value variant (Section 4) and the philosophy-mixed BPE model (Section 3).
32
 
33
- ## 3. Step 1: A Standard Language Model Does Not Extrapolate
34
 
35
  Table 1 shows the 18.9M-parameter baselines ($L=6$, $d_{\text{model}}=512$, 8 heads, 1D-RoPE, 400M training tokens, 1-12-digit operands). All three solve short sums: the character model is 30/30 at 4 and 8 digits. None of them reaches 12 digits reliably, which is the longest length in training: the character model scores 0/30 there and the BPE word-chain model 14/30. Past 12 digits every baseline scores 0/30 at every length we tested, up to 35 digits. So the failure is not confined to unseen lengths; it begins at the edge of the training range, where the model has seen few examples, and is complete beyond it. For the philosophy-mixed BPE model, 42 of its 60 working traces at 30 and 35 digits exceeded the 512-token window, so those two zeros partly reflect truncation; its zeros at 14-25 digits do not.
36
 
@@ -45,26 +62,22 @@ Table 1 shows the 18.9M-parameter baselines ($L=6$, $d_{\text{model}}=512$, 8 he
45
 
46
  **How they fail.** The wrong answers come from reading the wrong operand digit, not from arithmetic slips. For the 20-digit problem `64012498003866726193+44789680393999591374`, the character model's first column is `(0+9+4)=13`: it read the tens digit of the first operand (9) instead of its units digit (3). It then wrote only nine columns for a 20-column sum and answered `834569453`. At 12 digits (`320623293553+963365697522`) the first column is correct, the second column repeats it (`(0+3+2)=05(0+3+2)=05`, where the tens digits are 5 and 2), and the working stops one column early. With a flat index $t$, the distance between a scratchpad column and the operand digits it must read grows with the number of digits. At unseen lengths the model has to attend at relative offsets it never saw in training, and it reads the wrong digit. Adding 50% philosophy text to the BPE model lowered its short-sum accuracy (21/30 at 4 digits, 18/30 at 8) and did not change the result beyond 12 digits.
47
 
48
- ## 4. Step 2: Other Architectures Do Not Fix It
49
 
50
- We trained five other architectures on the same data and budget (Table 2): a looped Transformer that reuses two blocks six times [9, 10], a hierarchical fast/slow recurrent model, a hybrid of gated linear recurrence and attention, a harmonic neural register machine with a learned arithmetic unit (H-NRM), and an Abacus-looped model that adds place-value embeddings and shift augmentation [5] (ALSM). The looped, hierarchical and hybrid models behave like the plain Transformer: 30/30 at 4 and 8 digits, 0/30 at 12 and at every longer length. Recurrence and hierarchy do not make a flat index generalize.
51
 
52
- The other two did not learn the task in the first place. H-NRM and ALSM score 0/30 even at 4 digits. A teacher-forced check on 24 four-digit sums, given the gold working, separates this from a decoding problem: the plain character model and the hybrid predict 100% of the next characters, H-NRM 92.6% (enough per-character error to break a working of 60 or more characters) and ALSM 47.7%. Our ALSM variant (four core blocks looped six times, Abacus-style place indices with random shift $k=32$) is our own implementation, so this result says nothing about the original Abacus method [5]; it only shows that this variant did not fit within the same budget. A sixth baseline, a place-value variant trained with group relative policy optimization, scored 0/30 at every length. Its training traces were stored without line separators because the character vocabulary has no newline token in that format, so the terms ran together (`100050002006003070102`); we do not interpret this result.
53
-
54
- **Table 2.** Alternative architectures do not extrapolate. Same protocol and training data as Table 1; correct answers out of 30. Parameters are non-embedding. The last row is the 2D model of Section 5, shown for comparison.
55
 
56
  | Model | Params | 4 | 8 | 12 | 14 | 16 | 18 | 20 | 25 | 30 | 35 |
57
  | :--- | :---: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
58
  | Looped (2 blocks x 6) | 6.3M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
59
  | Hierarchical (fast 4, slow 3) | 7.3M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
60
  | Hybrid gated recurrence + attention | 18.9M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
61
- | Harmonic register machine (H-NRM) | 18.9M | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
62
- | Abacus-looped (ALSM) | 12.7M | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
63
  | **Decoupled 2D-RoPE ($k=32$)** | 1.57M | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 28 | 19 |
64
 
65
- **Randomized 1D positions.** We also trained four 100k-parameter character models ($L=2$, $d_{\text{model}}=64$) on the same 6,000 1-15-digit sums with a 1024-token context for 3,000 steps: standard 1D-RoPE, 1D-RoPE with a random offset $\Delta \sim \mathcal{U}\{0..512\}$, randomized sorted positions [11], and Decoupled 2D-RoPE with $k=32$ (Section 5). Evaluated on 200 random problems per length, all three 1D variants scored 0/600 even on seen 8-15-digit sums (final losses 0.098, 0.096 and 0.232), while the 2D model reached 417/600 on seen lengths and 977/1,200 (81.4%) on unseen 16-40 digits (identical on re-evaluation). Retraining 1D-RoPE for 6,000 steps with a tuned learning rate reached 56/200 at 8 digits and 0/200 from 12 digits on (Table 5). In this setting the 1D models do not learn the task at all, so this comparison concerns trainability; the length-generalization failure of 1D models that do fit the task is the one shown in Table 1.
66
 
67
- ## 5. Step 3: Two-Dimensional Coordinates
68
 
69
  **Coordinates.** Each token $x_t$ receives a coordinate $(p_t, \tau_t)$ from a deterministic parser applied inside the forward pass, identically in training and inference (Table 3, Part A). An operand digit gets $p$ equal to its place value (0 for units) and $\tau=0$. Inside the scratchpad, each `(` starts a new column, so $p$ advances by one and $\tau$ counts the tokens since the `(` (capped at 9). After `</THINK>=`, the $i$-th answer digit gets $p=\max(0, C-1-i)$, where $C$ is the number of scratchpad columns the model emitted. The model never predicts coordinates, but it controls column advancement by emitting `(`.
70
 
@@ -84,7 +97,7 @@ $$p_t \leftarrow p_t + \Delta \cdot \mathbb{I}(p_t > 0),\qquad \Delta \sim \math
84
 
85
  so that place coordinates up to $p_{\max} = N_{\text{train}} + k$ appear in training while the prompt start stays at $p=0$. This is a coordinate-space version of randomized positions [11].
86
 
87
- **Table 3.** Coordinates assigned by the parser. *Part A, arithmetic*: prompt `47+85=`, scratchpad `<THINK>(0+7+5)=12(1+4+8)=13</THINK>`, answer `=132`. Digits of the same place value share $p$ in the prompt and scratchpad, whatever the operand length. In the answer, $p$ counts down from the last scratchpad column, so when the sum has a carry-out digit (as here, `132`) the answer coordinates are one place lower than the true place values; the model is exact on such sums (Section 8). *Part B, dialogue*: $p$ is the index of the most recent segment tag and $\tau$ counts characters since that tag.
88
 
89
  Part A (arithmetic; the middle of the second scratchpad column is omitted, so the 1D index jumps from 17 to 28):
90
 
@@ -101,7 +114,7 @@ Part B (dialogue):
101
  | Segment $p_t$ | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 4 | 4 | 4 |
102
  | Character $\tau_t$ | 0 | 1 | 2 | 3 | 4 | 0 | 1 | 2 | 3 | 0 | 1 | 2 | 3 | 4 | 5 | 0 | 1 | 2 | 0 | 1 | 2 |
103
 
104
- **Result 1: a small 2D model extrapolates.** We trained a 1.57M-parameter (non-embedding) model with Decoupled 2D-RoPE and shift masking $k=32$ (embedding width 256, 4 heads, a two-level fast/slow recurrent core) on the same data as the baselines: 400M tokens of 1-12-digit sums with the same scratchpad. Table 2 (last row) shows it solves every sum we tested from 4 to 25 digits (240/240 over 8 lengths, including 12 digits, its longest training length), more than twice the training length, and scores 28/30 at 30 digits and 19/30 at 35. The 18.9M-parameter baselines with the same data and 12x more parameters score 0/30 at 14 digits. The accuracy begins to fall at 30 digits, below $p_{\max} = 12 + 32 = 44$, so the horizon is not sharp; we have no explanation for the gap beyond noting that the 100k models in Result 3 and CBT-18.95M in Section 8 also degrade somewhat before $p_{\max}$.
105
 
106
  **Sample efficiency.** The same architecture trained for 250-600 steps on only $N$ sums (Table 4) fits the training range with $N=1000$ (29/30 at 4 digits, 30/30 at 8, 28/30 at 12) but extrapolates only briefly (22/30 at 14 digits, 14/30 at 16, 1/30 at 25). With $N=100$ and $N=10$ it did not fit the task (2/30 and 0/30 at 4 digits). So the extrapolation we see with full training is not free: it needs the model to fit the training range first, and 1000 examples are enough only for a short margin.
107
 
@@ -137,13 +150,13 @@ Part B (dialogue):
137
 
138
  **Interpretation.** With a flat index, "read the operand digit of the current column" is a different relative offset for every column and every problem length, so a model trained on short problems learns offsets that do not exist at longer lengths. With place coordinates the same instruction becomes one relation, attend where $\Delta p = 0$, identical for every column and length. We think the 2D models extrapolate because they learn this single fixed relation, in effect memorizing a linear relation in coordinate space rather than a set of length-specific distances. The relation only holds for coordinate values the model has seen, which is why the limit follows $p_{\max}$ rather than the training length. This account is consistent with the shift sweep and the place-only ablation; we have not tested it directly, for example by probing attention patterns at unseen lengths.
139
 
140
- ## 6. Step 4: From Digits to Characters
141
 
142
  Subword vocabularies give a language model ready-made units that carry meaning. A character-level model has no such units: every word must be assembled from characters, sequences are several times longer, and prior work finds character models harder to train and weaker or slower than subword models at equal scale [12, 13, 14]. Character and byte models remain attractive because they have no tokenization artifacts and a tiny vocabulary [15, 16, 17, 18]. For arithmetic in particular, BPE merges digits of different place values into one token, which breaks the column alignment above.
143
 
144
- The addition results suggested a question for character models. If a fixed relation in coordinate space lets a model align digits regardless of length, can the same kind of coordinate help a character model treat a word as a unit? We extended the parser to dialogue (Table 3, Part B): $p$ indexes the segment (`<USER>`, `<THINK>`, `<CALL>`, `<RESULT>`, `<ANSWER>`) and $\tau$ counts characters within it. Copying an entity from the user message into a tool call then becomes a relation between two segments, from $p=0$ to $p=2$, at matching character offsets within the entity, analogous to $\Delta p=0$ in addition. The hypothesis we test is that a compact character model built on this layout can do both exact arithmetic and usable word-level tasks such as tool routing and entity copying. Section 9 tests whether the layout is responsible for the language results by training a 1D twin on the same data.
145
 
146
- ## 7. CBT-18.95M
147
 
148
  **Architecture.** CBT-18.95M is a decoder with $L=6$ blocks, $d_{\text{model}}=512$, 8 heads ($d_k=64$), pre-LayerNorm, no biases, tied input and output embeddings, and a 1024-token context. The six blocks are applied $R=3$ times with shared weights (18 effective layers) using a residual update scaled by $1/\sqrt{R}$. We kept recurrence as a parameter-efficient way to add depth, not because addition needs it: it gave no benefit in the 100k ablation, and we did not test $R=1$ at this scale. The model has 18,944,512 parameters, 18,887,680 of them outside the embedding table, or about 36 MB in FP16.
149
 
@@ -157,15 +170,47 @@ The addition results suggested a question for character models. If a fixed relat
157
  4. *Intermediate fine-tuning* (12,000 steps, final loss 0.0034, $k=45$): 45% multi-tool dialogues, 35% search copy and 20% arithmetic replay with 1-25-digit operands. This is the *intermediate model*.
158
  5. *Final fine-tuning* (23,927 steps, stopped by a 6-hour budget, $k=45$): a decontaminated online stream of 35% search copy, 20% arithmetic replay (1-25 digits) and the rest multi-tool and chat dialogues, with loss weight 2 on tool-call arguments. Its phrasings were broadened after the intermediate model's failures were seen. Held-out tool exactness at the end of training was 63/64 and held-out math 100%. **This is CBT-18.95M.**
159
 
160
- Because of the replay, sums of up to 25 digits are seen in training and $p_{\max} = 25 + 45 = 70$. All test prompts and entities were checked against the fine-tuning corpus. Section 8 evaluates CBT-18.95M and, where noted, the earlier stages; Section 9 compares it with a 1D-RoPE twin.
161
 
162
- ## 8. Results of CBT-18.95M
163
 
164
  All results in this section were obtained on CPU from the checkpoints, with greedy decoding. Tool calls are free-running and scored by exact match of the emitted call. Arithmetic is scored by teacher-forced greedy agreement: a sum counts as correct only if the greedy argmax equals the gold scratchpad and answer at every position after the prompt, which is the condition for free-running greedy decoding to reproduce the gold sequence. Each length has 30 random sums (70% uniform digits, 15% dense carries, 15% cascades such as $99\ldots9+b$).
165
 
166
  **Arithmetic.** Table 6 shows that CBT-18.95M is exact on all 150 sums at 4-16 digits, on all 60 at 20 and 25 digits, and on all 210 at 26-55 digits (Wilson 95% CI 98.2-100%), which are longer than any sum in training. Accuracy falls at 60 digits (25/30) and 70 digits (19/30), still inside $p_{\max}=70$ (254/270 pooled over 26-70 digits, CI 90.6-96.3%). Localizing the first wrong token of the 16 failures at 55-70 digits shows that all are uniform-digit sums with an error inside the scratchpad, at columns 59-70 (token positions 705-840). There is no error in columns 1-58 on any problem, and every cascade (17/17) and dense-carry sum (9/9) at 55-70 digits is exact, so the failures are local errors in a column's carry or digit, not the digit drift of the 1D baselines. At 80 and 100 digits the full trace would be about 1,150 and 1,430 tokens, longer than the 1,024-token context. For reference, the intermediate model scored free-running on a different random sample was exact on 240/240 sums of 16-50 digits, 14/20 at 60 and 6/15 at 70 digits.
167
 
168
- **Table 6.** Addition of CBT-18.95M (2D-RoPE) and its 1D-RoPE twin. Correct sums out of 30 per length. Training covers 1-25 digits; the twin is described in Section 9.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
169
 
170
  | Digits | Status | 2D-RoPE | 1D-RoPE |
171
  | :--- | :--- | :-: | :-: |
@@ -179,7 +224,7 @@ All results in this section were obtained on CPU from the checkpoints, with gree
179
 
180
  **Tool use.** The phrasing suite has 26 prompts with paraphrases and invented entities over weather, time, unit conversion, search, calculator, conversational reply and four nested-schema tools (database query, HTTP fetch, calendar event, file manager), whose schemas occur in the training data. The out-of-distribution (OOD) suite has 18 prompts whose entities do not occur in the fine-tuning corpus (audited: 0 overlaps). The unseen-copy suite has 30 prompts in five groups (historical figures, cities, science terms, concepts, invented words), none of whose entities appear in any training stage. Table 7 reports routing (right tool) and exact match (right tool and argument string).
181
 
182
- **Table 7.** Tool calls across fine-tuning stages. Right tool / exact call. The base is the pre-SFT checkpoint, Intermediate is the model after the first fine-tuning round, 2D is CBT-18.95M and 1D its twin. The base was scored only on unseen copy. The final fine-tuning data was broadened against the phrasing and OOD suites (Section 7).
183
 
184
  | Suite | Base | Interm. | 2D | 1D |
185
  | :--- | :-: | :-: | :-: | :-: |
@@ -189,33 +234,30 @@ All results in this section were obtained on CPU from the checkpoints, with gree
189
 
190
  CBT-18.95M routes and copies correctly on all 74 prompts (Wilson 95% lower bounds 87%, 82% and 89% for the three suites). The unseen-copy set is the cleanest evidence, since none of its entities occur in any training corpus: all five groups are 6/6, and the exact-copy rate is above 50% (one-sided binomial $p=9.3\times10^{-10}$). The phrasing and OOD suites are less clean because the final fine-tuning data was broadened after the intermediate model's failures on them were seen.
191
 
192
- **Effect of the fine-tuning stages on copying.** Copying is the weak part of a character model, and the earlier stages show it. On the unseen-copy suite the pre-SFT base copies 4/30 entities exactly and the intermediate model 17/30 (routing roughly constant, 28 and 29 of 30). The base's errors are mostly fragments (`Belo` for Belo Horizonte, `Cai` for Chiang Mai, `quarks` for quarks and leptons); the intermediate model's are mostly one- or two-character slips (`Barch Spinoza`, `Ada Nonath`, `Chiang Maix`) and, on the OOD and phrasing suites, corrupted spellings (`Auccra` for Accra, `Rerevan` for Yerevan, `zggdrasil`) and truncated names (`Santo` for Santo Domingo). The intermediate model also routed all four nested-schema phrasings wrongly (three of them to `search`). CBT-18.95M gets every one of these cases exact, including the city, time-zone and multi-word cases that the intermediate model corrupted. This is the word-formation weakness of character models discussed in Section 6, and the result shows that more targeted training data removes it at this scale.
193
 
194
- ## 9. Control: A 1D-RoPE Twin
195
 
196
- The experiments so far leave open which of CBT-18.95M's results the coordinate layout is responsible for. We trained a twin that differs in position encoding only: standard RoPE over the whole head dimension at positions $0,\dots,T-1$, with the same architecture (18,944,512 parameters), vocabulary, intermediate recipe (12,000 steps) and final recipe (23,927 steps, step-matched). The final-stage corpus is byte-identical after decompression (SHA256 of the text matches). A uniform shift of positions is an exact no-op under RoPE, so the twin has no shift augmentation, and the comparison is between the full 2D scheme and plain 1D-RoPE. The lineages also differ before the intermediate stage: the 2D model's pretraining stopped at step 45,059 while the twin completed 50,000 steps, and the 2D model also received the search-copy and all-tools stages (11,000 steps, neither with arithmetic) that the twin skipped. Both models were scored with the harness of Section 8.
197
 
198
- **Arithmetic.** The twin drops from 29/30 at 25 digits to 0/30 at 26, one digit past the training maximum, and scores 0 on every unseen length (Table 6, 0/270 pooled, CI 0.0-1.4%). Localizing the first wrong token of the twin's 150 failures at 26, 30, 40, 55 and 70 digits shows that all of them lie inside the scratchpad. For the 124 uniform and dense sums the first error is in column 1 (93), 2 (29) or 3 (2): the twin misreads an operand digit at the start of the scratchpad. For the 26 carry cascades it is in columns 22-27 and always writes `0` for an operand digit `9`, as if the operand of nines ended near the 25-digit training maximum. This contrasts with the 2D model, whose first error is never before column 59. The pattern is consistent with the Proposition in Section 5: under 1D positions the distance between a scratchpad column and the operand digit it must read grows with the operand length and leaves the trained range, while under 2D coordinates it does not. We did not test that explanation directly, for example with attention maps, and we did not find a single consistent wrong-place offset in the misread digits.
199
 
200
  **Tool use.** Table 7 shows no such gap. The twin routes and copies as well as the 2D model on all three suites (73/74 against 74/74 exact calls); its one failure, the HTTP-fetch phrasing, picks the right tool and endpoint and then breaks the `params` dictionary so that the call never closes.
201
 
202
- This changes how Section 6 should be read. The final fine-tuning data, not the layout, produced the gain in tool use and copying: the intermediate model rose from 17/30 to 30/30 exact copies on the unseen set (entities absent from its corpus), and a 1D model trained on the same data reached the same score. The hypothesis that segment and character-offset coordinates help a character model assemble words is therefore not supported at this scale and sample size; a character model with plain 1D-RoPE copies unseen entities just as well once the data cover the failure modes. What the layout does contribute is arithmetic length generalization. The other direction also holds: a character-level model with 1D positions can be a competent tool user and still be unable to add a 26-digit sum.
203
-
204
- ## 10. Discussion and Limitations
205
 
206
- **Relation to prior work.** Place-value alignment is the idea behind Abacus embeddings [5], index hints and positional formats [19, 20, 21], and randomized positions [11]; looped Transformers address the same problem through depth [22]. Our coordinates are another hand-designed alignment, delivered through the rotary subspace and extended to dialogue segments.
207
 
208
  **Limitations.**
209
 
210
  - The coordinates are assigned by a rule-based parser, not learned, and the parser assumes the `A+B=` format. The arithmetic result is conditional on correct coordinates and covers only addition with a fixed scratchpad.
211
- - The 1D control (Section 9) compares the full 2D scheme, coordinates and shift augmentation together, with plain 1D-RoPE, so it does not show which component is necessary. The two lineages also differ before the intermediate stage (the twin had 50,000 pretraining steps against 45,059 and skipped two tool fine-tuning stages with no arithmetic), each model is a single run, and the mechanism is inferred from where the twin fails, not tested directly.
212
  - The final fine-tuning data were broadened after the intermediate model's failures on the phrasing and OOD suites were seen, so CBT-18.95M's scores on those suites partly reflect targeted coverage. The unseen-copy set, whose entities are absent from the corpus, is the cleaner evidence. The twin shares this data, so the 2D/1D comparison is unaffected. The ceiling on the three suites (100% observed, lower Wilson bounds of 87%, 82% and 89%) also limits how small a difference they can detect.
213
- - Two of the alternative architectures (H-NRM and ALSM) did not fit the task, so Table 2 cannot show that those designs fail to extrapolate, only that they did not train here.
214
  - Every configuration is a single run, and several test sets are small (14 cases for the factor ablation, 26 and 18 for tools, 30 per length for arithmetic).
215
  - Extrapolation is bounded by $p_{\max}$, and CBT-18.95M already degrades below it (60 and 70 digits).
216
  - Generation is character by character with three recurrent passes, so per-token compute is about that of a 57M-parameter dense model; the 36 MB figure covers weights only.
217
 
218
- ## 11. Conclusion
219
 
220
  We asked a small autoregressive model to add integers longer than any it had seen. Standard 1D-RoPE Transformers with a character or BPE vocabulary solved short sums and then scored 0/30 at every length past 12 digits, and three recurrent or hierarchical variants did the same. Giving each token a two-dimensional coordinate for place value and step let a 1.57M-parameter model add 14-25-digit sums without error (150/150) after training on 1-12 digits, and controlled 100k-parameter runs show that its limit follows the range of coordinates seen in training. We read this as the model learning one fixed relation between digits of equal place value, which is a hypothesis we have not tested directly.
221
 
 
4
 
5
  ## Abstract
6
 
7
+ We set out to train a small autoregressive language model that adds multi-digit integers correctly at lengths it has not seen. A standard 18.91M-parameter decoder with one-dimensional rotary positions (1D-RoPE), trained on 1-12-digit sums with a column scratchpad, solves short sums but scores 0/30 at every unseen length we tested (14-35 digits), with either a character or a 50k-token BPE vocabulary; three alternative architectures (looped, hierarchical, hybrid) do the same. Giving each token a two-dimensional coordinate, the decimal place $p$ and the step $\tau$ within a column, through Decoupled 2D-RoPE changes the outcome: a 1.57M-parameter model trained on the same 1-12-digit range is exact on 150/150 sums of 14-25 digits and on 47/60 sums of 30-35 digits, and in controlled 100k-parameter runs the extrapolation limit moves with the coordinate range seen in training ($p_{\max} = N_{\text{train}} + k$). Our interpretation is that the coordinates turn place-value alignment into one fixed relation ($\Delta p = 0$) that does not change with length, so the model learns it once. Character-level models have no meaning-bearing subword embeddings and must assemble every word from characters, which prior work finds hard. We therefore asked whether the same relational layout, with $p$ indexing the dialogue segment and $\tau$ the character within it, could support a compact character-level model that also handles words. The resulting Character-Based Transformer (CBT-18.95M) is exact on 210/210 random sums of 26-55 digits, all longer than any sum in training (254/270 up to 70 digits), and exact on all 74 prompts of three tool-use suites (26/26 phrasing, 18/18 out-of-distribution, and 30/30 copies of entities absent from the training corpus, against 4/30 for the pre-fine-tuning base). Its final fine-tuning data was broadened after earlier failures were seen, so the unseen-entity set is the cleaner evidence. To attribute these results we trained a 1D-RoPE twin with the same architecture, data and step count. It scores 0/270 on every unseen length from 26 to 70 digits, but matches the 2D model on tool use (73/74 exact calls against 74/74). The 2D coordinates are therefore responsible for length generalization, while the tool-use and copying results are not attributable to them.
8
 
9
  ## 1. Introduction
10
 
11
  Small Transformers trained from scratch learn multi-digit addition within their training lengths but fail beyond them [1, 2, 3, 4, 5]. Addition is a convenient test of whether a model has learned an algorithm rather than a lookup over seen lengths: the procedure is a small finite-state machine applied once per column, and the answer is exactly checkable at any length. This paper reports a sequence of experiments that began with that question and ended with a compact character-level model. We present them in the order we ran them.
12
 
13
+ 1. **A standard language model.** We trained an 18.91M-parameter decoder with 1D-RoPE [6] on 1-12-digit sums with an explicit column scratchpad [7, 8], using a character vocabulary and, separately, a 50k BPE vocabulary. It solved 4- and 8-digit sums, scored at most 14/30 at 12 digits, and scored 0/30 at every unseen length (Section 4).
14
+ 2. **Other architectures.** Looped, hierarchical and hybrid recurrent variants [9, 10], trained the same way, either scored 0/30 on every length from 12 digits up (three models), and randomized 1D positions [11] did not help a 100k model (Section 5).
15
+ 3. **Two-dimensional coordinates.** Assigning each token a coordinate $(p, \tau)$, place value and step within the column, and rotating queries and keys by both (Decoupled 2D-RoPE) made the models extrapolate. The extrapolation limit tracks the coordinate range seen in training, which suggests the model learns a fixed relation between positions of equal place value (Section 6).
16
+ 4. **From digits to characters.** Character-level models must compose words from characters, which is a known weakness [12, 13, 14]. We asked whether the same relational layout helps a character model with words, and built the best small model we could around it: CBT-18.95M (Sections 7-9).
17
+ 5. **A 1D control.** To find out which results the layout is responsible for, we trained a twin with standard 1D-RoPE on identical data. It does not extrapolate, but it matches the 2D model on tool use (Section 10).
18
 
19
+ **Our contributions** are:
20
 
21
+ - A re-evaluation of six 1D-position checkpoints (standard, BPE and alternative architectures) on a common random-sum benchmark. None of them solves any unseen length.
22
+ - Evidence from controlled ablations that 2D place-value coordinates are the factor that enables extrapolation, and that the extrapolation limit is set by the trained coordinate range (Section 6).
23
+ - CBT-18.95M, an 18.9M-parameter character model that combines exact addition at lengths up to 55 digits (trained on at most 25) with tool calling and entity copying (Sections 8-9).
24
+ - A matched 1D-RoPE twin of CBT-18.95M, which shows that the 2D coordinates account for the arithmetic extrapolation (254/270 against 0/270 on unseen lengths) and not for the tool-use results (Section 10).
25
+
26
+ All results are single training runs; test sets and their sizes are given with every number. Figures (including the headline Figure 1) are in the PDF version of the paper.
27
+
28
+ ## 2. Related Work
29
+
30
+ **Length generalization in arithmetic.** Transformers trained on short sums fail on longer ones [1, 2, 3], and how well they extrapolate depends strongly on the position encoding [4]. Scratchpads of intermediate steps help within the training range [7, 8].
31
+
32
+ **Digit-aligned positions.** Abacus embeddings [5] add to each digit's token embedding a learned embedding indexed by its position within its number, so that digits of equal place value share an index, and they report addition far beyond the training length. Index hints and positional formats [20, 19, 21] insert or reformat tokens to align digits, and randomized positions [11] train on position values drawn from a wider range. Our coordinates enter differently: they drive a rotary rotation inside attention, so only coordinate differences matter ($\Delta p = 0$ for aligned digits), a second coordinate carries the step within a column, and the same mechanism extends to dialogue segments. We did not compare against Abacus embeddings or index hints under our protocol, so we make no claim that our scheme is better.
33
+
34
+ **Depth.** Looped and universal Transformers [9, 10, 22] address length generalization through iterated depth. CBT-18.95M uses weight-tied recurrence, but our ablation found no addition benefit from it.
35
+
36
+ **Character-level models.** Character and byte models avoid tokenization artifacts but must compose every word from characters and tend to be harder to train than subword models [12, 13, 14, 15, 16, 17, 18].
37
+
38
+ ## 3. Task and Evaluation Protocol
39
 
40
  **Addition as a state machine.** For operands with digits $a_i, b_i$ (column $i=0$ is the units digit) the sum is produced by
41
 
 
45
 
46
  **Scratchpad format.** Character models are trained on `A+B=<THINK>` followed by one block `(c+a+b)=c's` per column, from the units column up, then `</THINK>=S`. For example, `47+85=<THINK>(0+7+5)=12(1+4+8)=13</THINK>=132`. BPE models use the same steps in words (`Column 0: 7+5=12, digit 2, carry 1. ...`). The loss is masked on the prompt `A+B=`.
47
 
48
+ **Benchmark.** For each length $d$ we draw 30 problems with two $d$-digit operands from a stratified generator: 70% uniform random digits, 15% dense carries (all digits 5-9), and 15% long carry cascades. Decoding is greedy. A problem counts as correct only if the answer after `</THINK>=` matches the true sum exactly. We report $k/n$ with 95% Wilson intervals. Lengths inside the training range are labelled *seen*. Baseline traces up to 35 digits fit in their 512-token context, so truncation does not explain the failures below, with one exception noted where it occurs: the philosophy-mixed BPE model (Section 4).
49
 
50
+ ## 4. Step 1: A Standard Language Model Does Not Extrapolate
51
 
52
  Table 1 shows the 18.9M-parameter baselines ($L=6$, $d_{\text{model}}=512$, 8 heads, 1D-RoPE, 400M training tokens, 1-12-digit operands). All three solve short sums: the character model is 30/30 at 4 and 8 digits. None of them reaches 12 digits reliably, which is the longest length in training: the character model scores 0/30 there and the BPE word-chain model 14/30. Past 12 digits every baseline scores 0/30 at every length we tested, up to 35 digits. So the failure is not confined to unseen lengths; it begins at the edge of the training range, where the model has seen few examples, and is complete beyond it. For the philosophy-mixed BPE model, 42 of its 60 working traces at 30 and 35 digits exceeded the 512-token window, so those two zeros partly reflect truncation; its zeros at 14-25 digits do not.
53
 
 
62
 
63
  **How they fail.** The wrong answers come from reading the wrong operand digit, not from arithmetic slips. For the 20-digit problem `64012498003866726193+44789680393999591374`, the character model's first column is `(0+9+4)=13`: it read the tens digit of the first operand (9) instead of its units digit (3). It then wrote only nine columns for a 20-column sum and answered `834569453`. At 12 digits (`320623293553+963365697522`) the first column is correct, the second column repeats it (`(0+3+2)=05(0+3+2)=05`, where the tens digits are 5 and 2), and the working stops one column early. With a flat index $t$, the distance between a scratchpad column and the operand digits it must read grows with the number of digits. At unseen lengths the model has to attend at relative offsets it never saw in training, and it reads the wrong digit. Adding 50% philosophy text to the BPE model lowered its short-sum accuracy (21/30 at 4 digits, 18/30 at 8) and did not change the result beyond 12 digits.
64
 
65
+ ## 5. Step 2: Other Architectures Do Not Fix It
66
 
67
+ We trained three other architectures on the same data and budget (Table 2): a looped Transformer that reuses two blocks six times [9, 10], a hierarchical fast/slow recurrent model, and a hybrid of gated linear recurrence and attention. The looped, hierarchical and hybrid models behave like the plain Transformer: 30/30 at 4 and 8 digits, 0/30 at 12 and at every longer length. Recurrence and hierarchy do not make a flat index generalize.
68
 
69
+ **Table 2.** Alternative architectures do not extrapolate. Same protocol and training data as Table 1; correct answers out of 30. Parameters are non-embedding. The last row is the 2D model of Section 6, shown for comparison.
 
 
70
 
71
  | Model | Params | 4 | 8 | 12 | 14 | 16 | 18 | 20 | 25 | 30 | 35 |
72
  | :--- | :---: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
73
  | Looped (2 blocks x 6) | 6.3M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
74
  | Hierarchical (fast 4, slow 3) | 7.3M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
75
  | Hybrid gated recurrence + attention | 18.9M | 30 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
 
 
76
  | **Decoupled 2D-RoPE ($k=32$)** | 1.57M | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 28 | 19 |
77
 
78
+ **Randomized 1D positions.** We also trained four 100k-parameter character models ($L=2$, $d_{\text{model}}=64$) on the same 6,000 1-15-digit sums with a 1024-token context for 3,000 steps: standard 1D-RoPE, 1D-RoPE with a random offset $\Delta \sim \mathcal{U}\{0..512\}$, randomized sorted positions [11], and Decoupled 2D-RoPE with $k=32$ (Section 6). Evaluated on 200 random problems per length, all three 1D variants scored 0/600 even on seen 8-15-digit sums (final losses 0.098, 0.096 and 0.232), while the 2D model reached 417/600 on seen lengths and 977/1,200 (81.4%) on unseen 16-40 digits (identical on re-evaluation). Retraining 1D-RoPE for 6,000 steps with a tuned learning rate reached 56/200 at 8 digits and 0/200 from 12 digits on (Table 5). In this setting the 1D models do not learn the task at all, so this comparison concerns trainability; the length-generalization failure of 1D models that do fit the task is the one shown in Table 1.
79
 
80
+ ## 6. Step 3: Two-Dimensional Coordinates
81
 
82
  **Coordinates.** Each token $x_t$ receives a coordinate $(p_t, \tau_t)$ from a deterministic parser applied inside the forward pass, identically in training and inference (Table 3, Part A). An operand digit gets $p$ equal to its place value (0 for units) and $\tau=0$. Inside the scratchpad, each `(` starts a new column, so $p$ advances by one and $\tau$ counts the tokens since the `(` (capped at 9). After `</THINK>=`, the $i$-th answer digit gets $p=\max(0, C-1-i)$, where $C$ is the number of scratchpad columns the model emitted. The model never predicts coordinates, but it controls column advancement by emitting `(`.
83
 
 
97
 
98
  so that place coordinates up to $p_{\max} = N_{\text{train}} + k$ appear in training while the prompt start stays at $p=0$. This is a coordinate-space version of randomized positions [11].
99
 
100
+ **Table 3.** Coordinates assigned by the parser. *Part A, arithmetic*: prompt `47+85=`, scratchpad `<THINK>(0+7+5)=12(1+4+8)=13</THINK>`, answer `=132`. Digits of the same place value share $p$ in the prompt and scratchpad, whatever the operand length. In the answer, $p$ counts down from the last scratchpad column, so when the sum has a carry-out digit (as here, `132`) the answer coordinates are one place lower than the true place values; the model is exact on such sums (Section 9). *Part B, dialogue*: $p$ is the index of the most recent segment tag and $\tau$ counts characters since that tag.
101
 
102
  Part A (arithmetic; the middle of the second scratchpad column is omitted, so the 1D index jumps from 17 to 28):
103
 
 
114
  | Segment $p_t$ | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 4 | 4 | 4 |
115
  | Character $\tau_t$ | 0 | 1 | 2 | 3 | 4 | 0 | 1 | 2 | 3 | 0 | 1 | 2 | 3 | 4 | 5 | 0 | 1 | 2 | 0 | 1 | 2 |
116
 
117
+ **Result 1: a small 2D model extrapolates.** We trained a 1.57M-parameter (non-embedding) model with Decoupled 2D-RoPE and shift masking $k=32$ (embedding width 256, 4 heads, a two-level fast/slow recurrent core) on the same data as the baselines: 400M tokens of 1-12-digit sums with the same scratchpad. Table 2 (last row) shows it solves every sum we tested from 4 to 25 digits (240/240 over 8 lengths, including 12 digits, its longest training length), more than twice the training length, and scores 28/30 at 30 digits and 19/30 at 35. The 18.9M-parameter baselines with the same data and 12x more parameters score 0/30 at 14 digits. The accuracy begins to fall at 30 digits, below $p_{\max} = 12 + 32 = 44$, so the horizon is not sharp; we have no explanation for the gap beyond noting that the 100k models in Result 3 and CBT-18.95M in Section 9 also degrade somewhat before $p_{\max}$.
118
 
119
  **Sample efficiency.** The same architecture trained for 250-600 steps on only $N$ sums (Table 4) fits the training range with $N=1000$ (29/30 at 4 digits, 30/30 at 8, 28/30 at 12) but extrapolates only briefly (22/30 at 14 digits, 14/30 at 16, 1/30 at 25). With $N=100$ and $N=10$ it did not fit the task (2/30 and 0/30 at 4 digits). So the extrapolation we see with full training is not free: it needs the model to fit the training range first, and 1000 examples are enough only for a short margin.
120
 
 
150
 
151
  **Interpretation.** With a flat index, "read the operand digit of the current column" is a different relative offset for every column and every problem length, so a model trained on short problems learns offsets that do not exist at longer lengths. With place coordinates the same instruction becomes one relation, attend where $\Delta p = 0$, identical for every column and length. We think the 2D models extrapolate because they learn this single fixed relation, in effect memorizing a linear relation in coordinate space rather than a set of length-specific distances. The relation only holds for coordinate values the model has seen, which is why the limit follows $p_{\max}$ rather than the training length. This account is consistent with the shift sweep and the place-only ablation; we have not tested it directly, for example by probing attention patterns at unseen lengths.
152
 
153
+ ## 7. Step 4: From Digits to Characters
154
 
155
  Subword vocabularies give a language model ready-made units that carry meaning. A character-level model has no such units: every word must be assembled from characters, sequences are several times longer, and prior work finds character models harder to train and weaker or slower than subword models at equal scale [12, 13, 14]. Character and byte models remain attractive because they have no tokenization artifacts and a tiny vocabulary [15, 16, 17, 18]. For arithmetic in particular, BPE merges digits of different place values into one token, which breaks the column alignment above.
156
 
157
+ The addition results suggested a question for character models. If a fixed relation in coordinate space lets a model align digits regardless of length, can the same kind of coordinate help a character model treat a word as a unit? We extended the parser to dialogue (Table 3, Part B): $p$ indexes the segment (`<USER>`, `<THINK>`, `<CALL>`, `<RESULT>`, `<ANSWER>`) and $\tau$ counts characters within it. Copying an entity from the user message into a tool call then becomes a relation between two segments, from $p=0$ to $p=2$, at matching character offsets within the entity, analogous to $\Delta p=0$ in addition. The hypothesis we test is that a compact character model built on this layout can do both exact arithmetic and usable word-level tasks such as tool routing and entity copying. Section 10 tests whether the layout is responsible for the language results by training a 1D twin on the same data.
158
 
159
+ ## 8. CBT-18.95M
160
 
161
  **Architecture.** CBT-18.95M is a decoder with $L=6$ blocks, $d_{\text{model}}=512$, 8 heads ($d_k=64$), pre-LayerNorm, no biases, tied input and output embeddings, and a 1024-token context. The six blocks are applied $R=3$ times with shared weights (18 effective layers) using a residual update scaled by $1/\sqrt{R}$. We kept recurrence as a parameter-efficient way to add depth, not because addition needs it: it gave no benefit in the 100k ablation, and we did not test $R=1$ at this scale. The model has 18,944,512 parameters, 18,887,680 of them outside the embedding table, or about 36 MB in FP16.
162
 
 
170
  4. *Intermediate fine-tuning* (12,000 steps, final loss 0.0034, $k=45$): 45% multi-tool dialogues, 35% search copy and 20% arithmetic replay with 1-25-digit operands. This is the *intermediate model*.
171
  5. *Final fine-tuning* (23,927 steps, stopped by a 6-hour budget, $k=45$): a decontaminated online stream of 35% search copy, 20% arithmetic replay (1-25 digits) and the rest multi-tool and chat dialogues, with loss weight 2 on tool-call arguments. Its phrasings were broadened after the intermediate model's failures were seen. Held-out tool exactness at the end of training was 63/64 and held-out math 100%. **This is CBT-18.95M.**
172
 
173
+ Because of the replay, sums of up to 25 digits are seen in training and $p_{\max} = 25 + 45 = 70$. All test prompts and entities were checked against the fine-tuning corpus. Section 9 evaluates CBT-18.95M and, where noted, the earlier stages; Section 10 compares it with a 1D-RoPE twin.
174
 
175
+ ## 9. Results of CBT-18.95M
176
 
177
  All results in this section were obtained on CPU from the checkpoints, with greedy decoding. Tool calls are free-running and scored by exact match of the emitted call. Arithmetic is scored by teacher-forced greedy agreement: a sum counts as correct only if the greedy argmax equals the gold scratchpad and answer at every position after the prompt, which is the condition for free-running greedy decoding to reproduce the gold sequence. Each length has 30 random sums (70% uniform digits, 15% dense carries, 15% cascades such as $99\ldots9+b$).
178
 
179
  **Arithmetic.** Table 6 shows that CBT-18.95M is exact on all 150 sums at 4-16 digits, on all 60 at 20 and 25 digits, and on all 210 at 26-55 digits (Wilson 95% CI 98.2-100%), which are longer than any sum in training. Accuracy falls at 60 digits (25/30) and 70 digits (19/30), still inside $p_{\max}=70$ (254/270 pooled over 26-70 digits, CI 90.6-96.3%). Localizing the first wrong token of the 16 failures at 55-70 digits shows that all are uniform-digit sums with an error inside the scratchpad, at columns 59-70 (token positions 705-840). There is no error in columns 1-58 on any problem, and every cascade (17/17) and dense-carry sum (9/9) at 55-70 digits is exact, so the failures are local errors in a column's carry or digit, not the digit drift of the 1D baselines. At 80 and 100 digits the full trace would be about 1,150 and 1,430 tokens, longer than the 1,024-token context. For reference, the intermediate model scored free-running on a different random sample was exact on 240/240 sums of 16-50 digits, 14/20 at 60 and 6/15 at 70 digits.
180
 
181
+ Table 6b varies the two operand lengths independently, with 10 random sums per cell (rows: digits of the second operand b, columns: digits of the first operand a; entries are correct sums out of 10). The 2D model is exact on all 250 cells inside the trained range (both operands up to 25 digits) and on 644 of the 750 sums that have at least one longer operand; the twin gets 234/250 inside the range and 0/750 outside it. The 2D model is perfect for operands up to 50 digits in both positions, and its errors are concentrated where an operand has 60 or 70 digits (195/250 when both exceed 25 digits, with the weakest row and column at 70). So unequal operand lengths cause no extra difficulty beyond the length of the longer operand.
182
+
183
+ **Table 6b.** Addition accuracy by operand lengths, CBT-18.95M (2D-RoPE), correct out of 10 per cell. Operand lengths seen in training: at most 25 digits.
184
+
185
+ | b \ a | 5 | 10 | 15 | 20 | 25 | 30 | 40 | 50 | 60 | 70 |
186
+ | :-- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
187
+ | **5** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 9 | 9 |
188
+ | **10** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 7 | 7 |
189
+ | **15** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 8 | 7 |
190
+ | **20** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 9 | 6 |
191
+ | **25** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 2 |
192
+ | **30** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 9 | 3 |
193
+ | **40** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 4 |
194
+ | **50** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 8 | 5 |
195
+ | **60** | 9 | 10 | 9 | 9 | 9 | 9 | 9 | 9 | 10 | 6 |
196
+ | **70** | 8 | 6 | 6 | 6 | 3 | 5 | 4 | 4 | 4 | 6 |
197
+
198
+ **Table 6c.** The same grid for the 1D-RoPE twin.
199
+
200
+ | b \ a | 5 | 10 | 15 | 20 | 25 | 30 | 40 | 50 | 60 | 70 |
201
+ | :-- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
202
+ | **5** | 10 | 10 | 10 | 10 | 9 | 0 | 0 | 0 | 0 | 0 |
203
+ | **10** | 10 | 10 | 10 | 10 | 7 | 0 | 0 | 0 | 0 | 0 |
204
+ | **15** | 9 | 10 | 10 | 9 | 7 | 0 | 0 | 0 | 0 | 0 |
205
+ | **20** | 10 | 10 | 9 | 10 | 6 | 0 | 0 | 0 | 0 | 0 |
206
+ | **25** | 10 | 10 | 9 | 9 | 10 | 0 | 0 | 0 | 0 | 0 |
207
+ | **30** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
208
+ | **40** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
209
+ | **50** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
210
+ | **60** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
211
+ | **70** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
212
+
213
+ **Table 6.** Addition of CBT-18.95M (2D-RoPE) and its 1D-RoPE twin. Correct sums out of 30 per length. Training covers 1-25 digits; the twin is described in Section 10.
214
 
215
  | Digits | Status | 2D-RoPE | 1D-RoPE |
216
  | :--- | :--- | :-: | :-: |
 
224
 
225
  **Tool use.** The phrasing suite has 26 prompts with paraphrases and invented entities over weather, time, unit conversion, search, calculator, conversational reply and four nested-schema tools (database query, HTTP fetch, calendar event, file manager), whose schemas occur in the training data. The out-of-distribution (OOD) suite has 18 prompts whose entities do not occur in the fine-tuning corpus (audited: 0 overlaps). The unseen-copy suite has 30 prompts in five groups (historical figures, cities, science terms, concepts, invented words), none of whose entities appear in any training stage. Table 7 reports routing (right tool) and exact match (right tool and argument string).
226
 
227
+ **Table 7.** Tool calls across fine-tuning stages. Right tool / exact call. The base is the pre-SFT checkpoint, Intermediate is the model after the first fine-tuning round, 2D is CBT-18.95M and 1D its twin. The base was scored only on unseen copy. The final fine-tuning data was broadened against the phrasing and OOD suites (Section 8).
228
 
229
  | Suite | Base | Interm. | 2D | 1D |
230
  | :--- | :-: | :-: | :-: | :-: |
 
234
 
235
  CBT-18.95M routes and copies correctly on all 74 prompts (Wilson 95% lower bounds 87%, 82% and 89% for the three suites). The unseen-copy set is the cleanest evidence, since none of its entities occur in any training corpus: all five groups are 6/6, and the exact-copy rate is above 50% (one-sided binomial $p=9.3\times10^{-10}$). The phrasing and OOD suites are less clean because the final fine-tuning data was broadened after the intermediate model's failures on them were seen.
236
 
237
+ **Effect of the fine-tuning stages on copying.** Copying is the weak part of a character model, and the earlier stages show it. On the unseen-copy suite the pre-SFT base copies 4/30 entities exactly and the intermediate model 17/30 (routing roughly constant, 28 and 29 of 30). The base's errors are mostly fragments (`Belo` for Belo Horizonte, `Cai` for Chiang Mai, `quarks` for quarks and leptons); the intermediate model's are mostly one- or two-character slips (`Barch Spinoza`, `Ada Nonath`, `Chiang Maix`) and, on the OOD and phrasing suites, corrupted spellings (`Auccra` for Accra, `Rerevan` for Yerevan, `zggdrasil`) and truncated names (`Santo` for Santo Domingo). The intermediate model also routed all four nested-schema phrasings wrongly (three of them to `search`). CBT-18.95M gets every one of these cases exact, including the city, time-zone and multi-word cases that the intermediate model corrupted. This is the word-formation weakness of character models discussed in Section 7, and the result shows that more targeted training data removes it at this scale.
238
 
239
+ ## 10. Control: A 1D-RoPE Twin
240
 
241
+ The experiments so far leave open which of CBT-18.95M's results the coordinate layout is responsible for. We trained a twin that differs in position encoding only: standard RoPE over the whole head dimension at positions $0,\dots,T-1$, with the same architecture (18,944,512 parameters), vocabulary, intermediate recipe (12,000 steps) and final recipe (23,927 steps, step-matched). The final-stage corpus is byte-identical after decompression (SHA256 of the text matches). A uniform shift of positions is an exact no-op under RoPE, so the twin has no shift augmentation, and the comparison is between the full 2D scheme and plain 1D-RoPE. The lineages also differ before the intermediate stage: the 2D model's pretraining stopped at step 45,059 while the twin completed 50,000 steps, and the 2D model also received the search-copy and all-tools stages (11,000 steps, neither with arithmetic) that the twin skipped. Both models were scored with the harness of Section 9.
242
 
243
+ **Arithmetic.** The twin drops from 29/30 at 25 digits to 0/30 at 26, one digit past the training maximum, and scores 0 on every unseen length (Table 6, 0/270 pooled, CI 0.0-1.4%). Localizing the first wrong token of the twin's 150 failures at 26, 30, 40, 55 and 70 digits shows that all of them lie inside the scratchpad. For the 124 uniform and dense sums the first error is in column 1 (93), 2 (29) or 3 (2): the twin misreads an operand digit at the start of the scratchpad. For the 26 carry cascades it is in columns 22-27 and always writes `0` for an operand digit `9`, as if the operand of nines ended near the 25-digit training maximum. This contrasts with the 2D model, whose first error is never before column 59. The pattern is consistent with the Proposition in Section 6: under 1D positions the distance between a scratchpad column and the operand digit it must read grows with the operand length and leaves the trained range, while under 2D coordinates it does not. We did not test that explanation directly, for example with attention maps, and we did not find a single consistent wrong-place offset in the misread digits.
244
 
245
  **Tool use.** Table 7 shows no such gap. The twin routes and copies as well as the 2D model on all three suites (73/74 against 74/74 exact calls); its one failure, the HTTP-fetch phrasing, picks the right tool and endpoint and then breaks the `params` dictionary so that the call never closes.
246
 
247
+ This changes how Section 7 should be read. The final fine-tuning data, not the layout, produced the gain in tool use and copying: the intermediate model rose from 17/30 to 30/30 exact copies on the unseen set (entities absent from its corpus), and a 1D model trained on the same data reached the same score. The hypothesis that segment and character-offset coordinates help a character model assemble words is therefore not supported at this scale and sample size; a character model with plain 1D-RoPE copies unseen entities just as well once the data cover the failure modes. What the layout does contribute is arithmetic length generalization. The other direction also holds: a character-level model with 1D positions can be a competent tool user and still be unable to add a 26-digit sum.
 
 
248
 
249
+ ## 11. Discussion and Limitations
250
 
251
  **Limitations.**
252
 
253
  - The coordinates are assigned by a rule-based parser, not learned, and the parser assumes the `A+B=` format. The arithmetic result is conditional on correct coordinates and covers only addition with a fixed scratchpad.
254
+ - The 1D control (Section 10) compares the full 2D scheme, coordinates and shift augmentation together, with plain 1D-RoPE, so it does not show which component is necessary. The two lineages also differ before the intermediate stage (the twin had 50,000 pretraining steps against 45,059 and skipped two tool fine-tuning stages with no arithmetic), each model is a single run, and the mechanism is inferred from where the twin fails, not tested directly.
255
  - The final fine-tuning data were broadened after the intermediate model's failures on the phrasing and OOD suites were seen, so CBT-18.95M's scores on those suites partly reflect targeted coverage. The unseen-copy set, whose entities are absent from the corpus, is the cleaner evidence. The twin shares this data, so the 2D/1D comparison is unaffected. The ceiling on the three suites (100% observed, lower Wilson bounds of 87%, 82% and 89%) also limits how small a difference they can detect.
 
256
  - Every configuration is a single run, and several test sets are small (14 cases for the factor ablation, 26 and 18 for tools, 30 per length for arithmetic).
257
  - Extrapolation is bounded by $p_{\max}$, and CBT-18.95M already degrades below it (60 and 70 digits).
258
  - Generation is character by character with three recurrent passes, so per-token compute is about that of a 57M-parameter dense model; the 36 MB figure covers weights only.
259
 
260
+ ## 12. Conclusion
261
 
262
  We asked a small autoregressive model to add integers longer than any it had seen. Standard 1D-RoPE Transformers with a character or BPE vocabulary solved short sums and then scored 0/30 at every length past 12 digits, and three recurrent or hierarchical variants did the same. Giving each token a two-dimensional coordinate for place value and step let a 1.57M-parameter model add 14-25-digit sums without error (150/150) after training on 1-12 digits, and controlled 100k-parameter runs show that its limit follows the range of coordinates seen in training. We read this as the model learning one fixed relation between digits of equal place value, which is a hypothesis we have not tested directly.
263
 
RESEARCH_PAPER.pdf → paper/CBT-18.95M.pdf RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:63acd5f400fb015c5d06914187440079f3c15eb0fd2b88ee98ce8154e10b813c
3
- size 465243
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b0c1a95c219f2fd636cc748f8d63809049da42845d69ce70f9cddf059361186
3
+ size 389324
random_extrapolation_16_to_100d.json DELETED
@@ -1,655 +0,0 @@
1
- {
2
- "per_length": {
3
- "16": {
4
- "digits": 16,
5
- "n_samples": 30,
6
- "exact_match": 30,
7
- "accuracy_pct": 100.0,
8
- "wilson_95_ci": [
9
- 88.6,
10
- 100.0
11
- ],
12
- "think_exact": 30,
13
- "think_acc_pct": 100.0,
14
- "think_wilson_95_ci": [
15
- 88.6,
16
- 100.0
17
- ],
18
- "lparen_errors": 0,
19
- "off_by_one_readout_errors": 0,
20
- "seq_tokens": 226,
21
- "elapsed_sec": 20.63,
22
- "tok_per_sec": 328.6
23
- },
24
- "20": {
25
- "digits": 20,
26
- "n_samples": 30,
27
- "exact_match": 30,
28
- "accuracy_pct": 100.0,
29
- "wilson_95_ci": [
30
- 88.6,
31
- 100.0
32
- ],
33
- "think_exact": 30,
34
- "think_acc_pct": 100.0,
35
- "think_wilson_95_ci": [
36
- 88.6,
37
- 100.0
38
- ],
39
- "lparen_errors": 0,
40
- "off_by_one_readout_errors": 0,
41
- "seq_tokens": 278,
42
- "elapsed_sec": 21.19,
43
- "tok_per_sec": 393.5
44
- },
45
- "25": {
46
- "digits": 25,
47
- "n_samples": 30,
48
- "exact_match": 30,
49
- "accuracy_pct": 100.0,
50
- "wilson_95_ci": [
51
- 88.6,
52
- 100.0
53
- ],
54
- "think_exact": 30,
55
- "think_acc_pct": 100.0,
56
- "think_wilson_95_ci": [
57
- 88.6,
58
- 100.0
59
- ],
60
- "lparen_errors": 0,
61
- "off_by_one_readout_errors": 0,
62
- "seq_tokens": 343,
63
- "elapsed_sec": 28.65,
64
- "tok_per_sec": 359.2
65
- },
66
- "30": {
67
- "digits": 30,
68
- "n_samples": 30,
69
- "exact_match": 30,
70
- "accuracy_pct": 100.0,
71
- "wilson_95_ci": [
72
- 88.6,
73
- 100.0
74
- ],
75
- "think_exact": 30,
76
- "think_acc_pct": 100.0,
77
- "think_wilson_95_ci": [
78
- 88.6,
79
- 100.0
80
- ],
81
- "lparen_errors": 0,
82
- "off_by_one_readout_errors": 0,
83
- "seq_tokens": 408,
84
- "elapsed_sec": 46.16,
85
- "tok_per_sec": 265.2
86
- },
87
- "35": {
88
- "digits": 35,
89
- "n_samples": 30,
90
- "exact_match": 30,
91
- "accuracy_pct": 100.0,
92
- "wilson_95_ci": [
93
- 88.6,
94
- 100.0
95
- ],
96
- "think_exact": 30,
97
- "think_acc_pct": 100.0,
98
- "think_wilson_95_ci": [
99
- 88.6,
100
- 100.0
101
- ],
102
- "lparen_errors": 0,
103
- "off_by_one_readout_errors": 0,
104
- "seq_tokens": 473,
105
- "elapsed_sec": 55.95,
106
- "tok_per_sec": 253.6
107
- },
108
- "40": {
109
- "digits": 40,
110
- "n_samples": 30,
111
- "exact_match": 30,
112
- "accuracy_pct": 100.0,
113
- "wilson_95_ci": [
114
- 88.6,
115
- 100.0
116
- ],
117
- "think_exact": 30,
118
- "think_acc_pct": 100.0,
119
- "think_wilson_95_ci": [
120
- 88.6,
121
- 100.0
122
- ],
123
- "lparen_errors": 0,
124
- "off_by_one_readout_errors": 0,
125
- "seq_tokens": 538,
126
- "elapsed_sec": 61.79,
127
- "tok_per_sec": 261.2
128
- },
129
- "45": {
130
- "digits": 45,
131
- "n_samples": 30,
132
- "exact_match": 28,
133
- "accuracy_pct": 93.3,
134
- "wilson_95_ci": [
135
- 78.7,
136
- 98.2
137
- ],
138
- "think_exact": 29,
139
- "think_acc_pct": 96.7,
140
- "think_wilson_95_ci": [
141
- 83.3,
142
- 99.4
143
- ],
144
- "lparen_errors": 0,
145
- "off_by_one_readout_errors": 0,
146
- "seq_tokens": 603,
147
- "elapsed_sec": 63.52,
148
- "tok_per_sec": 284.8
149
- },
150
- "50": {
151
- "digits": 50,
152
- "n_samples": 30,
153
- "exact_match": 29,
154
- "accuracy_pct": 96.7,
155
- "wilson_95_ci": [
156
- 83.3,
157
- 99.4
158
- ],
159
- "think_exact": 29,
160
- "think_acc_pct": 96.7,
161
- "think_wilson_95_ci": [
162
- 83.3,
163
- 99.4
164
- ],
165
- "lparen_errors": 0,
166
- "off_by_one_readout_errors": 0,
167
- "seq_tokens": 668,
168
- "elapsed_sec": 85.37,
169
- "tok_per_sec": 234.7
170
- },
171
- "55": {
172
- "digits": 55,
173
- "n_samples": 20,
174
- "exact_match": 18,
175
- "accuracy_pct": 90.0,
176
- "wilson_95_ci": [
177
- 69.9,
178
- 97.2
179
- ],
180
- "think_exact": 19,
181
- "think_acc_pct": 95.0,
182
- "think_wilson_95_ci": [
183
- 76.4,
184
- 99.1
185
- ],
186
- "lparen_errors": 0,
187
- "off_by_one_readout_errors": 0,
188
- "seq_tokens": 799,
189
- "elapsed_sec": 86.78,
190
- "tok_per_sec": 184.2
191
- },
192
- "60": {
193
- "digits": 60,
194
- "n_samples": 20,
195
- "exact_match": 9,
196
- "accuracy_pct": 45.0,
197
- "wilson_95_ci": [
198
- 25.8,
199
- 65.8
200
- ],
201
- "think_exact": 9,
202
- "think_acc_pct": 45.0,
203
- "think_wilson_95_ci": [
204
- 25.8,
205
- 65.8
206
- ],
207
- "lparen_errors": 3,
208
- "off_by_one_readout_errors": 0,
209
- "seq_tokens": 869,
210
- "elapsed_sec": 81.06,
211
- "tok_per_sec": 214.4
212
- },
213
- "70": {
214
- "digits": 70,
215
- "n_samples": 15,
216
- "exact_match": 1,
217
- "accuracy_pct": 6.7,
218
- "wilson_95_ci": [
219
- 1.2,
220
- 29.8
221
- ],
222
- "think_exact": 1,
223
- "think_acc_pct": 6.7,
224
- "think_wilson_95_ci": [
225
- 1.2,
226
- 29.8
227
- ],
228
- "lparen_errors": 0,
229
- "off_by_one_readout_errors": 0,
230
- "seq_tokens": 928,
231
- "elapsed_sec": 68.66,
232
- "tok_per_sec": 202.7
233
- },
234
- "80": {
235
- "digits": 80,
236
- "n_samples": 10,
237
- "exact_match": 0,
238
- "accuracy_pct": 0.0,
239
- "wilson_95_ci": [
240
- 0.0,
241
- 27.8
242
- ],
243
- "think_exact": 0,
244
- "think_acc_pct": 0.0,
245
- "think_wilson_95_ci": [
246
- 0.0,
247
- 27.8
248
- ],
249
- "lparen_errors": 4,
250
- "off_by_one_readout_errors": 0,
251
- "seq_tokens": 1149,
252
- "elapsed_sec": 72.05,
253
- "tok_per_sec": 159.5
254
- },
255
- "100": {
256
- "digits": 100,
257
- "n_samples": 10,
258
- "exact_match": 0,
259
- "accuracy_pct": 0.0,
260
- "wilson_95_ci": [
261
- 0.0,
262
- 27.8
263
- ],
264
- "think_exact": 0,
265
- "think_acc_pct": 0.0,
266
- "think_wilson_95_ci": [
267
- 0.0,
268
- 27.8
269
- ],
270
- "lparen_errors": 0,
271
- "off_by_one_readout_errors": 0,
272
- "seq_tokens": 1429,
273
- "elapsed_sec": 101.11,
274
- "tok_per_sec": 141.3
275
- }
276
- },
277
- "failures": [
278
- {
279
- "digits": 45,
280
- "kind": "uniform_random",
281
- "expected_len": 46,
282
- "pred_len": 46,
283
- "think_ok": false,
284
- "expected_head": "10465044933514809662",
285
- "pred_head": "10465044933515809662"
286
- },
287
- {
288
- "digits": 45,
289
- "kind": "dense_carry",
290
- "expected_len": 46,
291
- "pred_len": 46,
292
- "think_ok": true,
293
- "expected_head": "19673285845443761366",
294
- "pred_head": "14673285845443761366"
295
- },
296
- {
297
- "digits": 50,
298
- "kind": "uniform_random",
299
- "expected_len": 50,
300
- "pred_len": 50,
301
- "think_ok": false,
302
- "expected_head": "68905429896446161053",
303
- "pred_head": "68905429896446161053"
304
- },
305
- {
306
- "digits": 55,
307
- "kind": "cascade_mix",
308
- "expected_len": 55,
309
- "pred_len": 133,
310
- "think_ok": true,
311
- "expected_head": "99999899800000895006",
312
- "pred_head": "99999899800000895006"
313
- },
314
- {
315
- "digits": 55,
316
- "kind": "cascade_mix",
317
- "expected_len": 56,
318
- "pred_len": 56,
319
- "think_ok": false,
320
- "expected_head": "18300979999929997971",
321
- "pred_head": "18290979999929997971"
322
- },
323
- {
324
- "digits": 60,
325
- "kind": "uniform_random",
326
- "expected_len": 61,
327
- "pred_len": 61,
328
- "think_ok": false,
329
- "expected_head": "17794705984524592751",
330
- "pred_head": "17784705984524592751"
331
- },
332
- {
333
- "digits": 60,
334
- "kind": "uniform_random",
335
- "expected_len": 61,
336
- "pred_len": 62,
337
- "think_ok": false,
338
- "expected_head": "10358362896129198913",
339
- "pred_head": "10035836289612919891"
340
- },
341
- {
342
- "digits": 60,
343
- "kind": "uniform_random",
344
- "expected_len": 60,
345
- "pred_len": 60,
346
- "think_ok": false,
347
- "expected_head": "99455736002861930179",
348
- "pred_head": "99445736002861930179"
349
- },
350
- {
351
- "digits": 60,
352
- "kind": "uniform_random",
353
- "expected_len": 61,
354
- "pred_len": 61,
355
- "think_ok": false,
356
- "expected_head": "13771859705436256056",
357
- "pred_head": "14771859705436256056"
358
- },
359
- {
360
- "digits": 60,
361
- "kind": "uniform_random",
362
- "expected_len": 61,
363
- "pred_len": 61,
364
- "think_ok": false,
365
- "expected_head": "11927411935032822174",
366
- "pred_head": "41927411935032822174"
367
- },
368
- {
369
- "digits": 60,
370
- "kind": "uniform_random",
371
- "expected_len": 61,
372
- "pred_len": 61,
373
- "think_ok": false,
374
- "expected_head": "11836982981890419337",
375
- "pred_head": "11736982981890419337"
376
- },
377
- {
378
- "digits": 60,
379
- "kind": "uniform_random",
380
- "expected_len": 61,
381
- "pred_len": 62,
382
- "think_ok": false,
383
- "expected_head": "13495686920877645844",
384
- "pred_head": "13595686920877645844"
385
- },
386
- {
387
- "digits": 60,
388
- "kind": "uniform_random",
389
- "expected_len": 60,
390
- "pred_len": 60,
391
- "think_ok": false,
392
- "expected_head": "33303324997839267821",
393
- "pred_head": "35303324997839267821"
394
- },
395
- {
396
- "digits": 60,
397
- "kind": "dense_carry",
398
- "expected_len": 61,
399
- "pred_len": 62,
400
- "think_ok": false,
401
- "expected_head": "17833947643427329535",
402
- "pred_head": "31783394764342732953"
403
- },
404
- {
405
- "digits": 60,
406
- "kind": "cascade_mix",
407
- "expected_len": 60,
408
- "pred_len": 133,
409
- "think_ok": false,
410
- "expected_head": "99999990999993999900",
411
- "pred_head": "49999999099999399990"
412
- },
413
- {
414
- "digits": 60,
415
- "kind": "cascade_mix",
416
- "expected_len": 61,
417
- "pred_len": 62,
418
- "think_ok": false,
419
- "expected_head": "15999869832993500000",
420
- "pred_head": "16999869832993500000"
421
- },
422
- {
423
- "digits": 70,
424
- "kind": "uniform_random",
425
- "expected_len": 71,
426
- "pred_len": 71,
427
- "think_ok": false,
428
- "expected_head": "13709577792207576758",
429
- "pred_head": "13610577792207576758"
430
- },
431
- {
432
- "digits": 70,
433
- "kind": "uniform_random",
434
- "expected_len": 71,
435
- "pred_len": 71,
436
- "think_ok": false,
437
- "expected_head": "18254804996856400492",
438
- "pred_head": "18274805096859400492"
439
- },
440
- {
441
- "digits": 70,
442
- "kind": "uniform_random",
443
- "expected_len": 71,
444
- "pred_len": 71,
445
- "think_ok": false,
446
- "expected_head": "15468076189969829446",
447
- "pred_head": "56459076190969829446"
448
- },
449
- {
450
- "digits": 70,
451
- "kind": "uniform_random",
452
- "expected_len": 71,
453
- "pred_len": 71,
454
- "think_ok": false,
455
- "expected_head": "17132113958771129774",
456
- "pred_head": "17292113958771129774"
457
- },
458
- {
459
- "digits": 70,
460
- "kind": "uniform_random",
461
- "expected_len": 70,
462
- "pred_len": 71,
463
- "think_ok": false,
464
- "expected_head": "81569945829520864708",
465
- "pred_head": "81669945829420864708"
466
- },
467
- {
468
- "digits": 70,
469
- "kind": "uniform_random",
470
- "expected_len": 71,
471
- "pred_len": 71,
472
- "think_ok": false,
473
- "expected_head": "14659845102561298448",
474
- "pred_head": "13657849102361298448"
475
- },
476
- {
477
- "digits": 70,
478
- "kind": "uniform_random",
479
- "expected_len": 71,
480
- "pred_len": 71,
481
- "think_ok": false,
482
- "expected_head": "12532903951038538802",
483
- "pred_head": "11332904551038538802"
484
- },
485
- {
486
- "digits": 70,
487
- "kind": "uniform_random",
488
- "expected_len": 70,
489
- "pred_len": 71,
490
- "think_ok": false,
491
- "expected_head": "91885390650615687834",
492
- "pred_head": "91885390980615687834"
493
- },
494
- {
495
- "digits": 70,
496
- "kind": "uniform_random",
497
- "expected_len": 71,
498
- "pred_len": 71,
499
- "think_ok": false,
500
- "expected_head": "17102804133024228749",
501
- "pred_head": "17102903132924228749"
502
- },
503
- {
504
- "digits": 70,
505
- "kind": "uniform_random",
506
- "expected_len": 71,
507
- "pred_len": 71,
508
- "think_ok": false,
509
- "expected_head": "14193174229527824431",
510
- "pred_head": "13273174229827824431"
511
- },
512
- {
513
- "digits": 70,
514
- "kind": "dense_carry",
515
- "expected_len": 71,
516
- "pred_len": 71,
517
- "think_ok": false,
518
- "expected_head": "15754532624644617966",
519
- "pred_head": "95754532624644617966"
520
- },
521
- {
522
- "digits": 70,
523
- "kind": "dense_carry",
524
- "expected_len": 71,
525
- "pred_len": 71,
526
- "think_ok": false,
527
- "expected_head": "15424597674457653251",
528
- "pred_head": "15624597684457653251"
529
- },
530
- {
531
- "digits": 70,
532
- "kind": "cascade_mix",
533
- "expected_len": 71,
534
- "pred_len": 71,
535
- "think_ok": false,
536
- "expected_head": "14007494074599980006",
537
- "pred_head": "13907494074599980006"
538
- },
539
- {
540
- "digits": 70,
541
- "kind": "cascade_mix",
542
- "expected_len": 71,
543
- "pred_len": 71,
544
- "think_ok": false,
545
- "expected_head": "17503600000090000000",
546
- "pred_head": "17503600000080000000"
547
- },
548
- {
549
- "digits": 80,
550
- "kind": "uniform_random",
551
- "expected_len": 81,
552
- "pred_len": 7,
553
- "think_ok": false,
554
- "expected_head": "12847570704495513468",
555
- "pred_head": "NO_STOP"
556
- },
557
- {
558
- "digits": 80,
559
- "kind": "uniform_random",
560
- "expected_len": 81,
561
- "pred_len": 84,
562
- "think_ok": false,
563
- "expected_head": "10585471026702913113",
564
- "pred_head": "50910585470037702913"
565
- },
566
- {
567
- "digits": 80,
568
- "kind": "uniform_random",
569
- "expected_len": 81,
570
- "pred_len": 81,
571
- "think_ok": false,
572
- "expected_head": "12278363141943820719",
573
- "pred_head": "41279373131933820717"
574
- },
575
- {
576
- "digits": 80,
577
- "kind": "uniform_random",
578
- "expected_len": 81,
579
- "pred_len": 82,
580
- "think_ok": false,
581
- "expected_head": "11724726771333240441",
582
- "pred_head": "41182576676137324034"
583
- },
584
- {
585
- "digits": 80,
586
- "kind": "uniform_random",
587
- "expected_len": 81,
588
- "pred_len": 133,
589
- "think_ok": false,
590
- "expected_head": "14074311451742727401",
591
- "pred_head": "40001406331265174282"
592
- },
593
- {
594
- "digits": 80,
595
- "kind": "uniform_random",
596
- "expected_len": 81,
597
- "pred_len": 7,
598
- "think_ok": false,
599
- "expected_head": "16952626939497179229",
600
- "pred_head": "NO_STOP"
601
- },
602
- {
603
- "digits": 80,
604
- "kind": "uniform_random",
605
- "expected_len": 80,
606
- "pred_len": 183,
607
- "think_ok": false,
608
- "expected_head": "82549613862831800793",
609
- "pred_head": "87848613863331800803"
610
- },
611
- {
612
- "digits": 80,
613
- "kind": "dense_carry",
614
- "expected_len": 81,
615
- "pred_len": 81,
616
- "think_ok": false,
617
- "expected_head": "13272527565535783883",
618
- "pred_head": "73272527565635784883"
619
- },
620
- {
621
- "digits": 80,
622
- "kind": "dense_carry",
623
- "expected_len": 81,
624
- "pred_len": 86,
625
- "think_ok": false,
626
- "expected_head": "16668493333645678724",
627
- "pred_head": "74810166685944336456"
628
- },
629
- {
630
- "digits": 80,
631
- "kind": "cascade_mix",
632
- "expected_len": 81,
633
- "pred_len": 173,
634
- "think_ok": false,
635
- "expected_head": "10002799999960006628",
636
- "pred_head": "49002600009959906628"
637
- }
638
- ],
639
- "summary_16_50": {
640
- "total_samples": 240,
641
- "exact_match": 237,
642
- "accuracy_pct": 98.75,
643
- "wilson_95_ci": [
644
- 96.4,
645
- 99.6
646
- ],
647
- "think_exact": 238,
648
- "think_acc_pct": 99.17,
649
- "think_wilson_95_ci": [
650
- 97.0,
651
- 99.8
652
- ]
653
- },
654
- "total_elapsed_sec": 792.96
655
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
random_extrapolation_champion_16_to_100d.json DELETED
@@ -1,610 +0,0 @@
1
- {
2
- "per_length": {
3
- "16": {
4
- "digits": 16,
5
- "n_samples": 30,
6
- "exact_match": 30,
7
- "accuracy_pct": 100.0,
8
- "wilson_95_ci": [
9
- 88.6,
10
- 100.0
11
- ],
12
- "think_exact": 30,
13
- "think_acc_pct": 100.0,
14
- "think_wilson_95_ci": [
15
- 88.6,
16
- 100.0
17
- ],
18
- "lparen_errors": 0,
19
- "off_by_one_readout_errors": 0,
20
- "seq_tokens": 226,
21
- "elapsed_sec": 27.6,
22
- "tok_per_sec": 245.6
23
- },
24
- "20": {
25
- "digits": 20,
26
- "n_samples": 30,
27
- "exact_match": 30,
28
- "accuracy_pct": 100.0,
29
- "wilson_95_ci": [
30
- 88.6,
31
- 100.0
32
- ],
33
- "think_exact": 30,
34
- "think_acc_pct": 100.0,
35
- "think_wilson_95_ci": [
36
- 88.6,
37
- 100.0
38
- ],
39
- "lparen_errors": 0,
40
- "off_by_one_readout_errors": 0,
41
- "seq_tokens": 278,
42
- "elapsed_sec": 38.68,
43
- "tok_per_sec": 215.6
44
- },
45
- "25": {
46
- "digits": 25,
47
- "n_samples": 30,
48
- "exact_match": 30,
49
- "accuracy_pct": 100.0,
50
- "wilson_95_ci": [
51
- 88.6,
52
- 100.0
53
- ],
54
- "think_exact": 30,
55
- "think_acc_pct": 100.0,
56
- "think_wilson_95_ci": [
57
- 88.6,
58
- 100.0
59
- ],
60
- "lparen_errors": 0,
61
- "off_by_one_readout_errors": 0,
62
- "seq_tokens": 343,
63
- "elapsed_sec": 54.9,
64
- "tok_per_sec": 187.4
65
- },
66
- "30": {
67
- "digits": 30,
68
- "n_samples": 30,
69
- "exact_match": 30,
70
- "accuracy_pct": 100.0,
71
- "wilson_95_ci": [
72
- 88.6,
73
- 100.0
74
- ],
75
- "think_exact": 30,
76
- "think_acc_pct": 100.0,
77
- "think_wilson_95_ci": [
78
- 88.6,
79
- 100.0
80
- ],
81
- "lparen_errors": 0,
82
- "off_by_one_readout_errors": 0,
83
- "seq_tokens": 408,
84
- "elapsed_sec": 79.13,
85
- "tok_per_sec": 154.7
86
- },
87
- "35": {
88
- "digits": 35,
89
- "n_samples": 30,
90
- "exact_match": 30,
91
- "accuracy_pct": 100.0,
92
- "wilson_95_ci": [
93
- 88.6,
94
- 100.0
95
- ],
96
- "think_exact": 30,
97
- "think_acc_pct": 100.0,
98
- "think_wilson_95_ci": [
99
- 88.6,
100
- 100.0
101
- ],
102
- "lparen_errors": 0,
103
- "off_by_one_readout_errors": 0,
104
- "seq_tokens": 473,
105
- "elapsed_sec": 95.72,
106
- "tok_per_sec": 148.3
107
- },
108
- "40": {
109
- "digits": 40,
110
- "n_samples": 30,
111
- "exact_match": 30,
112
- "accuracy_pct": 100.0,
113
- "wilson_95_ci": [
114
- 88.6,
115
- 100.0
116
- ],
117
- "think_exact": 30,
118
- "think_acc_pct": 100.0,
119
- "think_wilson_95_ci": [
120
- 88.6,
121
- 100.0
122
- ],
123
- "lparen_errors": 0,
124
- "off_by_one_readout_errors": 0,
125
- "seq_tokens": 538,
126
- "elapsed_sec": 155.8,
127
- "tok_per_sec": 103.6
128
- },
129
- "45": {
130
- "digits": 45,
131
- "n_samples": 30,
132
- "exact_match": 30,
133
- "accuracy_pct": 100.0,
134
- "wilson_95_ci": [
135
- 88.6,
136
- 100.0
137
- ],
138
- "think_exact": 30,
139
- "think_acc_pct": 100.0,
140
- "think_wilson_95_ci": [
141
- 88.6,
142
- 100.0
143
- ],
144
- "lparen_errors": 0,
145
- "off_by_one_readout_errors": 0,
146
- "seq_tokens": 603,
147
- "elapsed_sec": 182.82,
148
- "tok_per_sec": 99.0
149
- },
150
- "50": {
151
- "digits": 50,
152
- "n_samples": 30,
153
- "exact_match": 30,
154
- "accuracy_pct": 100.0,
155
- "wilson_95_ci": [
156
- 88.6,
157
- 100.0
158
- ],
159
- "think_exact": 30,
160
- "think_acc_pct": 100.0,
161
- "think_wilson_95_ci": [
162
- 88.6,
163
- 100.0
164
- ],
165
- "lparen_errors": 0,
166
- "off_by_one_readout_errors": 0,
167
- "seq_tokens": 668,
168
- "elapsed_sec": 166.06,
169
- "tok_per_sec": 120.7
170
- },
171
- "55": {
172
- "digits": 55,
173
- "n_samples": 20,
174
- "exact_match": 20,
175
- "accuracy_pct": 100.0,
176
- "wilson_95_ci": [
177
- 83.9,
178
- 100.0
179
- ],
180
- "think_exact": 20,
181
- "think_acc_pct": 100.0,
182
- "think_wilson_95_ci": [
183
- 83.9,
184
- 100.0
185
- ],
186
- "lparen_errors": 0,
187
- "off_by_one_readout_errors": 0,
188
- "seq_tokens": 733,
189
- "elapsed_sec": 134.79,
190
- "tok_per_sec": 108.8
191
- },
192
- "60": {
193
- "digits": 60,
194
- "n_samples": 20,
195
- "exact_match": 14,
196
- "accuracy_pct": 70.0,
197
- "wilson_95_ci": [
198
- 48.1,
199
- 85.5
200
- ],
201
- "think_exact": 14,
202
- "think_acc_pct": 70.0,
203
- "think_wilson_95_ci": [
204
- 48.1,
205
- 85.5
206
- ],
207
- "lparen_errors": 1,
208
- "off_by_one_readout_errors": 0,
209
- "seq_tokens": 798,
210
- "elapsed_sec": 150.02,
211
- "tok_per_sec": 106.4
212
- },
213
- "70": {
214
- "digits": 70,
215
- "n_samples": 15,
216
- "exact_match": 6,
217
- "accuracy_pct": 40.0,
218
- "wilson_95_ci": [
219
- 19.8,
220
- 64.3
221
- ],
222
- "think_exact": 6,
223
- "think_acc_pct": 40.0,
224
- "think_wilson_95_ci": [
225
- 19.8,
226
- 64.3
227
- ],
228
- "lparen_errors": 0,
229
- "off_by_one_readout_errors": 0,
230
- "seq_tokens": 928,
231
- "elapsed_sec": 161.99,
232
- "tok_per_sec": 85.9
233
- },
234
- "80": {
235
- "digits": 80,
236
- "n_samples": 10,
237
- "exact_match": 0,
238
- "accuracy_pct": 0.0,
239
- "wilson_95_ci": [
240
- 0.0,
241
- 27.8
242
- ],
243
- "think_exact": 0,
244
- "think_acc_pct": 0.0,
245
- "think_wilson_95_ci": [
246
- 0.0,
247
- 27.8
248
- ],
249
- "lparen_errors": 5,
250
- "off_by_one_readout_errors": 0,
251
- "seq_tokens": 1149,
252
- "elapsed_sec": 154.23,
253
- "tok_per_sec": 74.5
254
- },
255
- "100": {
256
- "digits": 100,
257
- "n_samples": 10,
258
- "exact_match": 0,
259
- "accuracy_pct": 0.0,
260
- "wilson_95_ci": [
261
- 0.0,
262
- 27.8
263
- ],
264
- "think_exact": 0,
265
- "think_acc_pct": 0.0,
266
- "think_wilson_95_ci": [
267
- 0.0,
268
- 27.8
269
- ],
270
- "lparen_errors": 0,
271
- "off_by_one_readout_errors": 0,
272
- "seq_tokens": 1429,
273
- "elapsed_sec": 200.31,
274
- "tok_per_sec": 71.3
275
- }
276
- },
277
- "failures": [
278
- {
279
- "digits": 60,
280
- "kind": "uniform_random",
281
- "expected_len": 60,
282
- "pred_len": 61,
283
- "think_ok": false,
284
- "expected_head": "99455736002861930179",
285
- "pred_head": "10045573600286193017"
286
- },
287
- {
288
- "digits": 60,
289
- "kind": "uniform_random",
290
- "expected_len": 61,
291
- "pred_len": 61,
292
- "think_ok": false,
293
- "expected_head": "13771859705436256056",
294
- "pred_head": "14771859705436256056"
295
- },
296
- {
297
- "digits": 60,
298
- "kind": "uniform_random",
299
- "expected_len": 60,
300
- "pred_len": 60,
301
- "think_ok": false,
302
- "expected_head": "93182197277112619803",
303
- "pred_head": "93182207277112619803"
304
- },
305
- {
306
- "digits": 60,
307
- "kind": "uniform_random",
308
- "expected_len": 61,
309
- "pred_len": 61,
310
- "think_ok": false,
311
- "expected_head": "13495686920877645844",
312
- "pred_head": "13595686920877645844"
313
- },
314
- {
315
- "digits": 60,
316
- "kind": "uniform_random",
317
- "expected_len": 60,
318
- "pred_len": 60,
319
- "think_ok": false,
320
- "expected_head": "33303324997839267821",
321
- "pred_head": "35303324997839267821"
322
- },
323
- {
324
- "digits": 60,
325
- "kind": "cascade_mix",
326
- "expected_len": 61,
327
- "pred_len": 61,
328
- "think_ok": false,
329
- "expected_head": "15999869832993500000",
330
- "pred_head": "16999869832993500000"
331
- },
332
- {
333
- "digits": 70,
334
- "kind": "uniform_random",
335
- "expected_len": 71,
336
- "pred_len": 71,
337
- "think_ok": false,
338
- "expected_head": "13709577792207576758",
339
- "pred_head": "13710577792207576758"
340
- },
341
- {
342
- "digits": 70,
343
- "kind": "uniform_random",
344
- "expected_len": 71,
345
- "pred_len": 71,
346
- "think_ok": false,
347
- "expected_head": "18254804996856400492",
348
- "pred_head": "18254805096856400492"
349
- },
350
- {
351
- "digits": 70,
352
- "kind": "uniform_random",
353
- "expected_len": 71,
354
- "pred_len": 71,
355
- "think_ok": false,
356
- "expected_head": "15468076189969829446",
357
- "pred_head": "15468076190969829446"
358
- },
359
- {
360
- "digits": 70,
361
- "kind": "uniform_random",
362
- "expected_len": 71,
363
- "pred_len": 71,
364
- "think_ok": false,
365
- "expected_head": "17132113958771129774",
366
- "pred_head": "17162113958771129774"
367
- },
368
- {
369
- "digits": 70,
370
- "kind": "uniform_random",
371
- "expected_len": 70,
372
- "pred_len": 70,
373
- "think_ok": false,
374
- "expected_head": "81569945829520864708",
375
- "pred_head": "81669945829520864708"
376
- },
377
- {
378
- "digits": 70,
379
- "kind": "uniform_random",
380
- "expected_len": 71,
381
- "pred_len": 71,
382
- "think_ok": false,
383
- "expected_head": "14659845102561298448",
384
- "pred_head": "14657845102561298448"
385
- },
386
- {
387
- "digits": 70,
388
- "kind": "uniform_random",
389
- "expected_len": 71,
390
- "pred_len": 71,
391
- "think_ok": false,
392
- "expected_head": "12532903951038538802",
393
- "pred_head": "12532903851038538802"
394
- },
395
- {
396
- "digits": 70,
397
- "kind": "uniform_random",
398
- "expected_len": 71,
399
- "pred_len": 71,
400
- "think_ok": false,
401
- "expected_head": "14193174229527824431",
402
- "pred_head": "14193174229827824431"
403
- },
404
- {
405
- "digits": 70,
406
- "kind": "cascade_mix",
407
- "expected_len": 71,
408
- "pred_len": 71,
409
- "think_ok": false,
410
- "expected_head": "17503600000090000000",
411
- "pred_head": "17503600000080000000"
412
- },
413
- {
414
- "digits": 80,
415
- "kind": "uniform_random",
416
- "expected_len": 81,
417
- "pred_len": 87,
418
- "think_ok": false,
419
- "expected_head": "12847570704495513468",
420
- "pred_head": "00000012847570704495"
421
- },
422
- {
423
- "digits": 80,
424
- "kind": "uniform_random",
425
- "expected_len": 81,
426
- "pred_len": 63,
427
- "think_ok": false,
428
- "expected_head": "10585471026702913113",
429
- "pred_head": "00000000000105854710"
430
- },
431
- {
432
- "digits": 80,
433
- "kind": "uniform_random",
434
- "expected_len": 81,
435
- "pred_len": 7,
436
- "think_ok": false,
437
- "expected_head": "12278363141943820719",
438
- "pred_head": "NO_STOP"
439
- },
440
- {
441
- "digits": 80,
442
- "kind": "uniform_random",
443
- "expected_len": 81,
444
- "pred_len": 88,
445
- "think_ok": false,
446
- "expected_head": "11724726771333240441",
447
- "pred_head": "00000001172472676133"
448
- },
449
- {
450
- "digits": 80,
451
- "kind": "uniform_random",
452
- "expected_len": 81,
453
- "pred_len": 7,
454
- "think_ok": false,
455
- "expected_head": "14074311451742727401",
456
- "pred_head": "NO_STOP"
457
- },
458
- {
459
- "digits": 80,
460
- "kind": "uniform_random",
461
- "expected_len": 81,
462
- "pred_len": 86,
463
- "think_ok": false,
464
- "expected_head": "16952626939497179229",
465
- "pred_head": "00000169526269394861"
466
- },
467
- {
468
- "digits": 80,
469
- "kind": "uniform_random",
470
- "expected_len": 80,
471
- "pred_len": 86,
472
- "think_ok": false,
473
- "expected_head": "82549613862831800793",
474
- "pred_head": "80000082849613863331"
475
- },
476
- {
477
- "digits": 80,
478
- "kind": "dense_carry",
479
- "expected_len": 81,
480
- "pred_len": 7,
481
- "think_ok": false,
482
- "expected_head": "13272527565535783883",
483
- "pred_head": "NO_STOP"
484
- },
485
- {
486
- "digits": 80,
487
- "kind": "dense_carry",
488
- "expected_len": 81,
489
- "pred_len": 7,
490
- "think_ok": false,
491
- "expected_head": "16668493333645678724",
492
- "pred_head": "NO_STOP"
493
- },
494
- {
495
- "digits": 80,
496
- "kind": "cascade_mix",
497
- "expected_len": 81,
498
- "pred_len": 7,
499
- "think_ok": false,
500
- "expected_head": "10002799999960006628",
501
- "pred_head": "NO_STOP"
502
- },
503
- {
504
- "digits": 100,
505
- "kind": "uniform_random",
506
- "expected_len": 101,
507
- "pred_len": 7,
508
- "think_ok": false,
509
- "expected_head": "11288401813144915928",
510
- "pred_head": "NO_STOP"
511
- },
512
- {
513
- "digits": 100,
514
- "kind": "uniform_random",
515
- "expected_len": 101,
516
- "pred_len": 7,
517
- "think_ok": false,
518
- "expected_head": "15225010932074486266",
519
- "pred_head": "NO_STOP"
520
- },
521
- {
522
- "digits": 100,
523
- "kind": "uniform_random",
524
- "expected_len": 101,
525
- "pred_len": 7,
526
- "think_ok": false,
527
- "expected_head": "17203428879913620849",
528
- "pred_head": "NO_STOP"
529
- },
530
- {
531
- "digits": 100,
532
- "kind": "uniform_random",
533
- "expected_len": 101,
534
- "pred_len": 7,
535
- "think_ok": false,
536
- "expected_head": "13949786348913299912",
537
- "pred_head": "NO_STOP"
538
- },
539
- {
540
- "digits": 100,
541
- "kind": "uniform_random",
542
- "expected_len": 101,
543
- "pred_len": 7,
544
- "think_ok": false,
545
- "expected_head": "12975883843361249782",
546
- "pred_head": "NO_STOP"
547
- },
548
- {
549
- "digits": 100,
550
- "kind": "uniform_random",
551
- "expected_len": 100,
552
- "pred_len": 7,
553
- "think_ok": false,
554
- "expected_head": "45147010533867848323",
555
- "pred_head": "NO_STOP"
556
- },
557
- {
558
- "digits": 100,
559
- "kind": "uniform_random",
560
- "expected_len": 101,
561
- "pred_len": 7,
562
- "think_ok": false,
563
- "expected_head": "13735571010375678962",
564
- "pred_head": "NO_STOP"
565
- },
566
- {
567
- "digits": 100,
568
- "kind": "dense_carry",
569
- "expected_len": 101,
570
- "pred_len": 7,
571
- "think_ok": false,
572
- "expected_head": "16553554674857513255",
573
- "pred_head": "NO_STOP"
574
- },
575
- {
576
- "digits": 100,
577
- "kind": "dense_carry",
578
- "expected_len": 101,
579
- "pred_len": 7,
580
- "think_ok": false,
581
- "expected_head": "18253466387437646273",
582
- "pred_head": "NO_STOP"
583
- },
584
- {
585
- "digits": 100,
586
- "kind": "cascade_mix",
587
- "expected_len": 101,
588
- "pred_len": 7,
589
- "think_ok": false,
590
- "expected_head": "17200006000000004996",
591
- "pred_head": "NO_STOP"
592
- }
593
- ],
594
- "summary_16_50": {
595
- "total_samples": 240,
596
- "exact_match": 240,
597
- "accuracy_pct": 100.0,
598
- "wilson_95_ci": [
599
- 98.4,
600
- 100.0
601
- ],
602
- "think_exact": 240,
603
- "think_acc_pct": 100.0,
604
- "think_wilson_95_ci": [
605
- 98.4,
606
- 100.0
607
- ]
608
- },
609
- "total_elapsed_sec": 1602.17
610
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
requirements.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ torch>=2.2.0
2
+ safetensors>=0.4.0
3
+ huggingface_hub>=0.20.0