twanghcmut/gr00t-n1d7-so101-left-arm-multitask

GR00T-N1.7 SO101 Left Arm, 11-task

NVIDIA Isaac GR00T N1.7 3B finetuned on the 11 datasets of the SO101_LeftArmOnly_Task collection, merged into one multitask set. One model covers all 11 tasks; the language instruction selects the behaviour.

Trained with the same batch size and step count as twanghcmut/pi05-so101-left-arm-multitask, on the same data.

Tasks

Use the exact wording, including trailing punctuation; the model has not been trained on paraphrases.

#	Instruction	Episodes
0	Pick up the red cube and place it into the steel pot.	50
1	Pick up the blue cube and place it into the steel pot	50
2	Take the banana and place it into the steel pot then close the lid.	50
3	Pick up the blue cube and place it next to the blue cube.	50
4	Take the blue cube out of the steel pot and place it on the blue cube.	51
5	Take the red cube out of the steel pot and place it on the red cube	50
6	Stack the yellow cube on top of the orange cube.	50
7	Pick up one orange block and stack it neatly on top of the other orange block	50
8	Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.	50
9	Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board	50
10	Take out the blue cubes from shortest to tallest and place it on the table	50

551 episodes, 253,745 frames in total.

Embodiment

SO101 follower arm (left), 5 joints plus gripper.

Embodiment tag	NEW_EMBODIMENT
State / action dim	6: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll + gripper
Cameras	top (overhead, 360×640), wrist (480×640) RGB
Action horizon	16
Data fps	30

Action representation: single_arm is RELATIVE (delta from current state), gripper is ABSOLUTE. The policy server unnormalizes both, so get_action returns absolute joint targets either way.

Observation format

Nested, as Gr00tPolicy expects. Do not start the server with --use-sim-policy-wrapper; that switches it to the flat format used by the sim envs.

obs = {
    "video": {                       # (B, T, H, W, C) uint8, T == 1
        "top":   top_rgb[None][None],
        "wrist": wrist_rgb[None][None],
    },
    "state": {                       # (B, T, D) float32, T == 1
        "single_arm": joints[:5].reshape(1, 1, 5).astype("float32"),
        "gripper":    joints[5:6].reshape(1, 1, 1).astype("float32"),
    },
    "language": {"annotation.human.task_description": [[INSTRUCTION]]},
}
action, info = client.get_action(obs)   # action["single_arm"] (1,16,5), action["gripper"] (1,16,1)

Serving

python gr00t/eval/run_gr00t_server.py \
  --model-path twanghcmut/gr00t-n1d7-so101-left-arm-multitask \
  --embodiment-tag NEW_EMBODIMENT --host 0.0.0.0 --port 5555

Training

Base	nvidia/GR00T-N1.7-3B: projector, VL layernorm and diffusion head trained; LLM and vision backbone frozen
Steps	6,000
Global batch size	128 (768k samples)
Learning rate	1e-4, cosine, 5% warmup
Weight decay	1e-5
episode_sampling_rate	0.1
Hardware	1× H100 80GB
Runtime	9.2 h (dataloader-bound, 23 samples/s)
Final train loss	0.0256 (mean over steps 5,760–6,000)

Loss at steps 500 / 1,000 / 2,000 / 3,000 / 4,000 / 5,000 / 6,000: 0.075 → 0.054 → 0.049 → 0.035 → 0.030 → 0.027 → 0.026.

Open-loop evaluation

One trajectory per task, up to 400 steps, action horizon 16, run against the training data with gr00t/eval/open_loop_eval.py. Errors are unnormalized, in the dataset's own action units.

traj	task	MAE
0	0 red cube → pot	2.098
50	1 blue cube → pot	2.327
100	2 banana → pot, close lid	3.258
150	3 blue cube next to blue cube	3.372
200	4 blue cube out of pot → blue cube	2.374
251	5 red cube out of pot → red cube	2.941
301	6 stack yellow on orange	2.604
351	7 stack orange blocks	3.567
401	8 balance red/blue on board	3.717
451	9 red cube via hollow cube	2.760
501	10 blue cubes shortest → tallest	3.385
	average	2.946 (MSE 25.02)

Divided by the absolute action std (mean 28.67 across dims) that is 10.27% of action scale at horizon 16. The π₀.₅ checkpoint trained on the same data and budget scores 9.82% on the same 11 trajectories; GR00T is lower-error on 5 of them (one, traj 100, by only 0.001). The two numbers come from each framework's own open-loop script, so treat a gap this small as a tie rather than a ranking.

Limitations

No held-out split. All 551 episodes were used for training. episode_sampling_rate subsamples timesteps within episodes, not episodes themselves, so nothing was withheld. The open-loop numbers above are therefore measured on training data: they show the policy fits its data and is not degenerate, but they do not measure generalization, and they are not a success rate. Treat real-robot performance as unmeasured until you run it.

One trajectory per task is a small sample; per-task numbers above are single episodes.

Deployment timing

The training data is 30 fps. A client that interpolates between model steps must derive its sub-step count from that ratio: smooth_step = control_hz / 30.