---
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
datasets:
- EleutherAI/the_pile_deduplicated
- nvidia/Llama-Nemotron-Post-Training-Dataset
base_model:
- allenai/Olmo-3-7B-Instruct
---

## Model Description

ADAPT is a technique that allows for size interpolation across different post-trained variants of the same base model. This is the student model distilled from Olmo-3-7B-Instruct from [our paper](https://arxiv.org/abs/2608.22854).

## Training Procedure

This model was initialized from Olmo-3-7B-Instruct by copying every other layer and the last 2 layers. It was distilled on 1B tokens of The deduplicated Pile and 1B of the math split from Llama Nemotron Post Training Dataset with cross entropy, KL, and cosine loss to match the activations of Olmo-3-7B-Instruct. We used the following hyperparameters:

- Learning rate: 3e-4
- Learning rate scheduler: cosine
- Warmup ratio: 0.01
- Optimizer: AdamW
- Adam betas: (0.9, 0.95)
- Adam epsilon: 1e-8
- Weight decay: 0.1
- Max. gradient norm: 1.0
- Number of training steps (Pre-training phase): 480
- Effective batch size (Pre-training phase): 2048
- Number of training steps (SFT phase): 585
- Effective batch size (SFT phase): 4096
- Max. sequence length: 1024
- Mixed precision: bf16
- KLDiv weight: 0.1
- Cosine distance weight per layer: 10.0

## Use

To interpolate between this model and Olmo-3-7B-Instruct, please use the `build_intermediate_model` function from [our github repository](https://github.com/dcml-lab/ADAPT):
```python3
import torch
from patching.patch import build_intermediate_model

intermediate_model = build_intermediate_model(
  teacher_name_or_path = "allenai/Olmo-3-7B-Instruct",
  student_name_or_path = "Harvard-DCML/ADAPT-Olmo3-4.3B-Instruct",
  num_layers_to_patch = 2,
  patch_first_k_layers = True,
  dtype = torch.bfloat16,
)
```
Notes:
1. Changing `num_layers_to_patch` changes the size of the intermediate model by patching different numbers of student layers.
2. `patch_first_k_layers` should be set to True for this model for optimal interpolation performance.

## Citation

```
@misc{zhou2026thinkingrightsizeamortized,
      title={Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs}, 
      author={Yan Zhou and Sara Kangaslahti and Jonathan Geuter and Nihal V. Nayak and Marco Fumero and Francesco Locatello and David Alvarez-Melis},
      year={2026},
      eprint={2608.22854},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.22854}, 
}
```