Ryukijano
/

gemma-groot

@@ -1,335 +1,114 @@
 ---
 language:
 - en
-license: mit
-library_name: transformers
-# Using text-to-video as the pipeline tag since the model generates action sequences from vision and language inputs
-pipeline_tag: text-to-video
-datasets:
-- lerobot/robot_sim.PickNPlace
-- lerobot/so100_strawberry_grape
-base_model: NVEagle/eagle_er-qwen3_1_7B-Siglip2_400M_stage1_5_128gpu_er_v7_1mlp_nops
 tags:
 - robotics
-- vision-language-action
-- reinforcement-learning
 - imitation-learning
-- nvidia
-- gr00t
-- gemma
 - diffusion-policy
-- lerobot
-- robot-learning
-- embodied-ai
-- humanoid-robots
-- robot-manipulation
-- computer-vision
-- natural-language-processing
-- deep-learning
-- transformer
-- vision-transformer
-- flow-matching
-- foundation-model
-- multi-modal
-- human-robot-interaction
-- autonomous-robots
-- robot-control
-- robot-perception
-- robot-vision
 ---
-# Gemma-GR00T: A Vision-Language-Action Model for Robotic Control
-This is a fine-tuned version of the NVIDIA GR00T N1.5 model, adapted for robotic control tasks using the LeRobot framework. The model combines vision, language, and action generation capabilities to enable robots to perform complex manipulation tasks based on natural language instructions.
-## Model Description
-Gemma-GR00T is a state-of-the-art multimodal vision-language-action policy that combines Google's Gemma language model with NVIDIA's GR00T robotics framework. This model is specifically designed for advanced robotic manipulation tasks, enabling robots to understand natural language instructions, perceive their environment through vision, and perform precise manipulation actions.
-## Model Details
-- **Model type:** Vision-Language-Action (VLA) model
-- **Base Model:** [NVEagle/eagle_er-qwen3_1_7B-Siglip2_400M_stage1_5_128gpu_er_v7_1mlp_nops](https://huggingface.co/NVEagle/eagle_er-qwen3_1_7B-Siglip2_400M_stage1_5_128gpu_er_v7_1mlp_nops)
-- **Task:** text-to-video (robot action generation from vision and language)
-- **Training Data:** Trained on LeRobot datasets using the `fourier_gr1_arms_only` configuration
-- **Framework:** PyTorch with Hugging Face Transformers
-- **Related Models:** [NVIDIA GR00T-N1.5-3B](https://huggingface.co/nvidia/GR00T-N1.5-3B), [LeRobot Models](https://huggingface.co/lerobot)
-- **Related Datasets:** [LeRobot Datasets](https://huggingface.co/lerobot/datasets)
-### Model Architecture
-The model is built on a sophisticated multimodal architecture that combines state-of-the-art vision and language models for robotic control:
-1. **Backbone**: `Eagle2_5_VLForConditionalGeneration`
-   - A powerful vision-language model that processes both visual and textual inputs
-   - Integrates vision and language representations for multimodal understanding
-2. **Text Encoder**: `Qwen3-1.7B`
-   - Base Model: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)
-   - Type: Causal Language Model
-   - Parameters: 1.7B
-   - Layers: 28
-   - Attention: 16 heads for Q, 8 heads for KV (GQA)
-   - Context Length: 32,768 tokens
-   - Features:
-     - Strong reasoning and instruction-following capabilities
-     - Optimized for long-context understanding
-     - Supports complex language understanding and generation
-3. **Vision Encoder**: `SigLIP` (Sigmoid Loss for Language-Image Pre-training)
-   - Base Model: [google/siglip-base-patch16-224](https://huggingface.co/google/siglip-base-patch16-224)
-   - Type: Vision Transformer (ViT)
-   - Patch Size: 16x16
-   - Image Size: 224x224
-   - Hidden Size: 768
-   - Layers: 12
-   - Attention Heads: 12
-   - Features:
-     - Strong visual representation learning
-     - Excellent zero-shot classification capabilities
-     - Robust to various visual domains
-4. **Action Head**: Diffusion-based Policy
-   - Type: Flow-matching action head
-   - Architecture: 4-layer transformer (ScaledDP)
-   - Hidden Size: 512
-   - Feed-Forward Size: 2,048
-   - Attention Heads: 8
-   - Features:
-     - Generates smooth, continuous actions for robotic control
-     - Uses diffusion process for action generation
-## Training & Evaluation
-### Training Performance
-- **Total Training Steps**: 30,000
-- **Final Epoch**: 114.5
-- **Initial Loss**: 1.27
-- **Final Loss**: 0.11
-- **Learning Rate**: Warmup to 1e-5 with gradual decay
-- **Gradient Norm**: Stabilized around 0.3-1.0 (initial: 11.1)
-### Recommended Evaluation Metrics
-#### Task Performance
-- **Success Rate**: Percentage of successful task completions
-- **Path Length**: Efficiency of movement (shorter paths are better)
-- **Smoothness**: L2 norm of action derivatives (lower is smoother)
-- **Goal Distance**: Final distance to target position
-- **Success Rate at k (SR@k)**: Success rate within k attempts
-#### Model Accuracy
-- **Action MSE**: Mean squared error of predicted vs. ground truth actions
-- **Per-Joint Position Error**: Error for each degree of freedom
-- **Gripper Accuracy**: Binary classification of gripper state
-- **Trajectory Error**: Dynamic Time Warping (DTW) distance from reference
-#### System Efficiency
-- **Inference Time**: Per-step latency (ms)
-- **Memory Usage**: Peak GPU memory consumption (GB)
-- **FLOPS**: Computational requirements
-- **Throughput**: Steps/second during inference
-#### Robustness
-- **Success Rate under Noise**: Performance with added sensor noise
-- **Generalization**: Performance on unseen objects/scenes
-- **Failure Mode Analysis**: Categorization of common failures
-- **Recovery Rate**: Ability to recover from perturbations
-### Evaluation Protocol
-1. **Test Environments**
-   - Fixed initial conditions
-   - Multiple random seeds (recommended: 5+)
-   - Human baseline comparison
-   - Ablation studies
-2. **Visualization**
-   - Trajectory plots (ground truth vs predicted)
-   - Attention heatmaps
-   - Failure case analysis
-   - Action distribution plots
-3. **Reporting**
-   - Mean and standard deviation across seeds
-   - Statistical significance testing
-   - Compute requirements (GPU hours, memory)
-   - Hyperparameter sensitivity analysis
-     - Processes both visual and language conditioning
-5. **Training Configuration**:
-   - Optimizer: AdamW (lr=1e-4, weight_decay=1e-6)
-   - Diffusion Steps: 100
-   - Chunk Size: 16
-   - Action Steps: 8
-   - Observation Steps: 1
-The model processes visual inputs through the SigLIP vision encoder and textual instructions through the Qwen3-1.7B language model, then fuses these representations in the Eagle2.5 backbone to generate precise control actions via the diffusion-based policy head. The architecture is specifically designed for real-time robotic control with low-latency inference.
-## Uses
-### Direct Use
-This model is part of the [Gemma-GR00T](https://github.com/Ryukijano/Gemma-Grook) project and is designed for research and development of robotic manipulation systems. It can be used for:
-- Robotic arm manipulation tasks (pick-and-place, assembly, etc.)
-- Sim-to-real transfer learning in robotics
-- Multimodal robotic control with natural language instructions
-- Research in reinforcement and imitation learning for robotics
-- Integration with the [LeRobot](https://github.com/huggingface/lerobot) ecosystem
-### Related Projects
-- [LeRobot](https://github.com/huggingface/lerobot): The base framework used for training
-- [GR00T](https://developer.nvidia.com/gr00t): NVIDIA's foundation model for humanoid robots
-- [Gemma](https://huggingface.co/google/gemma-7b): The language model backbone
-### Out-of-Scope Use
-This model is not intended for:
-- Critical systems where failure could lead to harm
-- Applications without proper safety measures
-- Real-time control without thorough testing
-- Non-robotic applications
-## How to Use
-### Installation
-```bash
-pip install -r requirements.txt
 ```
-### Loading the Model
 ```python
-from transformers import AutoModelForCausalLM, AutoConfig
-# Load the model
-model = AutoModelForCausalLM.from_pretrained("path/to/exported_weights")
-```
-### Inference Example
-```python
-# Example code for running inference with the model
 import torch
-def run_inference(observation, language_instruction):
-    # Preprocess observation and instruction
-    inputs = preprocess(observation, language_instruction)
-    # Run model inference
-    with torch.no_grad():
-        actions = model(**inputs)
-    return actions
 ```
-## Training Details
-### Training Data
-This model was trained using the [LeRobot](https://github.com/huggingface/lerobot) framework, which provides standardized datasets and tools for robotic learning. The training utilized the following configuration:
-- **Primary Datasets:**
-  - `lerobot/robot_sim.PickNPlace`: Simulated pick and place tasks
-  - `lerobot/so100_strawberry_grape`: Real-world manipulation tasks
-- **Data Configuration:** `fourier_gr1_arms_only`
-- **Dataset Documentation:** [LeRobot Datasets](https://huggingface.co/lerobot/datasets)
-- **Data Processing:** Follows LeRobot's standardized data pipeline for consistency with other models in the ecosystem
-- **Environment:** [Isaac Sim](https://developer.nvidia.com/isaac-sim)
-- **Training Steps:** 30,000
-- **Batch Size:** 32
-- **Learning Rate:** 1e-4
-- **Optimizer:** AdamW
-- **Weight Decay:** 1e-5
-- **Warmup Ratio:** 0.05
-- **Hardware:** 3× NVIDIA L40S GPUs
-- **Framework:** PyTorch with Hugging Face Transformers
-### Data Processing
-The model processes the following modalities from the LeRobot dataset:
-- **Visual Inputs:** Processed through a vision encoder
-- **Proprioception:** Arm joint states and gripper status
-- **Actions:** 32-dimensional continuous action space
-- **Language Instructions:** Natural language task descriptions
-### Training Procedure
-The model was trained using a combination of:
-- Imitation learning from demonstration data
-- Reinforcement learning with PPO
-- Behavior cloning
-## Evaluation
-### Metrics
-- **Success Rate:** 85% on validation tasks
-- **Task Completion:** 90% of tasks completed successfully
-- **Generalization:** 75% success on unseen objects
-### Results
-| Task | Success Rate |
-|------|-------------:|
-| Pick and Place | 88% |
-| Object Stacking | 83% |
-| Tool Use | 79% |
-| Multi-step Tasks | 72% |
-## Limitations and Bias
-- The model's performance is highly dependent on the quality and diversity of the training data.
-- May not generalize well to completely novel objects or environments.
-- Performance may degrade in cluttered or highly dynamic environments.
-- Safety mechanisms should be implemented for real-world deployment.
-## Environmental Impact
-- **Carbon Emissions:** Estimated 120 kg CO2eq
-- **Hardware Type:** NVIDIA L40S GPUs
-- **Hours used:** 240
-- **Cloud Provider:** Private cluster
-- **Compute Region:** UK
-- **Energy Mix:** 40% renewable
-## Technical Specifications
-### Model Architecture
-- **Parameters:** 1.7B
-- **Layers:** 16
-- **Attention Heads:** 32
-- **Hidden Size:** 2048
-- **Context Length:** 2048 tokens
-### Hardware and Software
-- **Training Hardware:** 3× NVIDIA L40S GPUs
-- **Inference Hardware:** NVIDIA L4 or better
-- **Framework:** PyTorch 2.7.1+
-- **CUDA Version:** 12.4
-## Citation
-```bibtex
-@misc{gemmagroot2024,
-  title={Gemma-GR00T: Multimodal Robotic Manipulation with Language Models},
-  author={Your Name},
-  year={2024},
-  publisher={GitHub},
-  howpublished={\url{https://github.com/Ryukijano/Gemma-Grook}},
-}
 ```
-## Model Card Contact
-For questions or comments about this model, please open an issue in the [GitHub repository](https://github.com/Ryukijano/Gemma-Grook/issues).
-## License
-This model is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.

 ---
+license: apache-2.0
 language:
 - en
 tags:
 - robotics
+- vla
+- lerobot
 - imitation-learning
 - diffusion-policy
+- gemma-3
+- siglip
+- scaledp
+- multimodal
 ---
+# Gemma-Le: SigLIP + Gemma 3 + ScaleDP (LeRobot VLA Policy)
+Gemma-Le is a compact Vision-Language-Action policy for robotic manipulation built on top of LeRobot.
+It replaces NV Eagle with standard Hugging Face components:
+- SigLIP `google/siglip-so400m-patch14-384` for vision
+- Gemma 3 `google/gemma-3-4b-it` for language/reasoning (with LoRA PEFT)
+- ScaleDP (Scalable Diffusion Transformer) as the action head
+This repo hosts exported checkpoints trained on LeRobot-format datasets (e.g., `robot_sim.PickNPlace`).
+## Architecture
+- Vision: SigLIP ViT encoder (384px, patch14), pooled embedding
+- Text: Gemma 3 4B-IT, mean-pooled hidden states
+- LoRA: rank=16 on `[q_proj, k_proj, v_proj, o_proj]`
+- Fusion: MLP projects [vision || text] -> `conditioning_dim=768`
+- Action head: ScaleDP Transformer (layers=12, d_model=320, heads=8, ff=1280) predicts diffusion noise
+- Temporal context: `chunk_size=8`; diffusion steps `num_diffusion_steps=50`
+- Mixed precision: AMP auto-selects bf16/fp16; bf16 uses no GradScaler
+## Default config (excerpt)
+```yaml
+vision_model_id: google/siglip-so400m-patch14-384
+text_model_id:   google/gemma-3-4b-it
+image_features:  ["observation.images.ego_view"]
+action_feature:  "action"
+chunk_size: 8
+num_diffusion_steps: 50
+conditioning_dim: 768
+plan_update_interval: 10
+scaledp_num_layers: 12
+scaledp_dim_model: 320
+scaledp_num_heads: 8
+scaledp_dim_feedforward: 1280
+use_lora: true
+lora_rank: 16
+lora_target_modules: ["q_proj","k_proj","v_proj","o_proj"]
+optimizer_lr: 1e-4
+optimizer_weight_decay: 1e-6
 ```
+## Usage (with this repo’s LeRobot fork)
+Install deps and set `PYTHONPATH` to include `lerobot` in this repository.
+Evaluation-style load:
 ```python
 import torch
+from lerobot.common.policies.gemma_le.modeling_gemma_le import GemmaLePolicy
+from huggingface_hub import snapshot_download
+ckpt_dir = snapshot_download(repo_id="Ryukijano/gemma-groot", revision="main")
+policy = GemmaLePolicy.from_pretrained(ckpt_dir, torch_dtype=torch.bfloat16)
+policy.eval()
 ```
+Training entrypoint:
+```bash
+python lerobot/lerobot/scripts/train.py \
+  --policy.type gemma_le \
+  --dataset.repo_id local/robot_sim.PickNPlace \
+  --dataset.root /path/to/robot_sim.PickNPlace \
+  --dataset.episodes "[0,1,2,3,4]" \
+  --batch_size 3 \
+  --steps 200000 \
+  --log_freq 100 \
+  --save_freq 5000 \
+  --policy.vision_model_id google/siglip-so400m-patch14-384 \
+  --policy.text_model_id google/gemma-3-4b-it \
+  --policy.use_amp true \
+  --progress_bar true \
+  --push_to_hub true \
+  --push_repo_id Ryukijano/gemma-groot \
+  --push_branch main \
+  --push_exist_ok true
 ```
+### Slurm (3× L40)
+See `submit_job.sh`. Ensure caches on scratch and set:
+- `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`
+- `HF_HOME`, `HUGGINGFACE_HUB_CACHE`, `TRANSFORMERS_CACHE` to scratch
+## Checkpoints
+- Latest runs uploaded under `runs/<date>/<run>/<step>` in this repo.
+- Example: `runs/2025-08-12/13-06-07_gemma_le/020000/`.
+## Data
+- LeRobotDataset (parquet + mp4 + metadata). Single RGB view: `observation.images.ego_view`. Targets: `action`.
+- Timestamp tolerance is auto-relaxed to `max(tolerance_s, 1/fps + 1e-4)` during training for robust decoding.
+## Notes
+- Base model access: `google/gemma-3-4b-it` may require TOS.
+- Intended for imitation learning; ThinkAct-style planning can be layered on top.
+## Citations
+- LeRobot: https://github.com/huggingface/lerobot
+- Gemma 3: https://ai.google.dev/gemma
+- SigLIP: https://huggingface.co/timm/ViT-SigLIP
+- Diffusion Policy: https://arxiv.org/abs/2303.04137
+```