WIKI300M holdout runs
72 language-model checkpoints trained on English Wikipedia and held out from the scaling-law fits,
for testing predictions made from the grid and colinear sweeps. Each .pt is one completed run.
| Checkpoints | 72 (203 GiB) |
| N | 22.8 M to 380.6 M parameters |
| D | 20.5 M to 1750.6 M tokens |
| Learning rate | 3e-4, weight decay 0.1, 1 epoch |
| Batch size | 96 to 384, sequence length 256 |
| Wall time per run | 0.14 h to 7.59 h on NVIDIA GH200 |
The parameter and token ranges extend past the fitted sweeps, to 380 M parameters and 1.75 B tokens, so these runs test extrapolation rather than interpolation.
Two families are present: 36 checkpoints prefixed WIKI300M_HOLDOUT held out from the NxD grid,
and 36 prefixed WIKI300M_COLINEAR_HOLDOUT held out from the colinear sweep, the latter carrying
a TPP ratio from 1.0 to 4.6.
Layout
N105.33M/D421.34M/WIKI300M_COLINEAR_HOLDOUT_N105.33M_TPP4.0_D421.34M_...pt
Checkpoints sit under N<params>M/D<tokens>M/, one per cell: every held-out run is a distinct
point. Eleven N folders span 22.80 M to 380.56 M, the upper five lying beyond the fitted sweeps.
Folder names are zero padded so that listing them alphabetically also orders them numerically.
Filename schema
WIKI300M_HOLDOUT_N<params>M_D<tokens>M_...
WIKI300M_COLINEAR_HOLDOUT_N<params>M_TPP<ratio>_D<tokens>M_...
_lr<lr>_distributed_epoch<n>_bs<batch>_seq<len>_wd<decay>
_time<hours>h_completedAt<timestamp>.pt
Loading
import torch
from huggingface_hub import hf_hub_download
path = hf_hub_download("leibnitz-lab/wiki300m-holdout", filename="<checkpoint>.pt")
state = torch.load(path, map_location="cpu")
These are raw PyTorch state dicts from a custom distributed trainer, not transformers models.
Related
- wiki300m-nxd-grid, the fitted grid
- wiki300m-colinear, the fitted colinear sweep