AutoE2E is designed as three cooperating models:
- a **Reactive model** at 10 Hz that plans the trajectory in real time from multi-view images, a map raster and ego-motion history;
- a **World Action Model** at 1 Hz that learns to predict future visual features (JEPA-style) from the encoded visual history;
- a **Reasoning model** at 1 Hz that detects edge-case scenarios and conditions the planner.
The v1.0 checkpoints contain the **Reactive model**, which is the part that drives. The World Action and Reasoning branches are disabled in these weights and remain under active development.
| Component | v1.0 checkpoint configuration |
|---|---|
| Cameras | 6 surround views at 512 × 512; the front camera is also encoded at 1024 × 1024 |
| Camera history | 8 frames at 0.5 s spacing (3.5 s), BEVFormer V2 temporal fusion |
| Camera BEV encoder | ResNet-50 + BEVFormer V2 (T8), 300 × 200 BEV grid covering 180 m × 120 m (60 m behind to 120 m ahead), frozen from the official BEVFormer V2 R50 T8 initialization |
| Map input (optional) | 14-channel semantic map raster plus a 2-channel route raster, 450 × 300 cells at 0.4 m. It can be left empty; the KITScenes Test results below use cameras only |
| Map/image fusion | Deformable cross-attention from map features into the camera BEV |
| Ego-motion history | 6.4 s at 10 Hz: speed, longitudinal acceleration, yaw rate, curvature |
| Trajectory planner | Deterministic GRU planner with deformable BEV feature lookup |
| Output | 64 steps × (longitudinal acceleration, curvature) at 10 Hz = 6.4 s, integrated into an ego-frame XY trajectory |
| Auxiliary heads | 8-class BEV segmentation and route reconstruction (training signals) |
| Parameters | 79.9 M |
## Results
All values are ADE / FDE in metres for a single predicted trajectory; lower is better. "Lateral" and "longitudinal" are the mean absolute errors across the valid horizon.
### Comparison with other open-weights models (KITScenes-Multimodal Validation, 3 s)
The fine-tuned AutoE2E checkpoint has the second-lowest error of the eight open-weights models compared here on KITScenes-Multimodal validation. Only Qwen Drive 1.0 is ahead, and AutoE2E beats SimForge, Alpamayo 1.5 and METEOR with 79.9 M parameters. All models are scored open-loop: each one predicts the next 3 s from logged camera frames, and the prediction is compared with the path the human driver took. ADE is the average position error over those 3 s and FDE is the error at the 3 s point.
| Model | Checkpoint / setting | Evaluated on | ADE @ 3 s (m) | FDE @ 3 s (m) |
|---|---|---|---:|---:|
| Qwen Drive 1.0 | first of 6 trajectories | 202 keyframes | 0.565 | 1.397 |
| **AutoE2E fine-tuned** | [`kitscenes-epoch-5`](./models/kitscenes-epoch-5) | 202 keyframes | **0.744** | **1.980** |
| SimForge D1.5 | 7 camera slots (2 virtual telephoto), first of 6 trajectories | 202 keyframes | 0.831 | 2.209 |
| **AutoE2E** | [`nuplan-epoch-5`](./models/nuplan-epoch-5), zero-shot | 11,035 samples | **0.915** | **2.423** |
| Alpamayo 1.5 | first of 6 trajectories | 202 keyframes | 1.020 | 2.820 |
| VAD | Base stage-2, oracle driving command | 11,035 samples | 1.242 | 2.821 |
| METEOR | top-1 of 3 paths | 202 keyframes | 1.240 | 3.282 |
| Drive-JEPA | perception-based v2, 20 m route command | 11,035 samples | 2.266 | 4.774 |
The 11,035 samples are every frame of the 117 validation scenes, and the 202 keyframes are non-overlapping anchors taken from the same scenes. Inputs differ by model: AutoE2E also gets an HD map and the route, and VAD and Drive-JEPA get a driving command. [Issue #210](https://github.com/autowarefoundation/auto_e2e/issues/210) has each model's adapter, ground truth and results at the other horizons.
### KITScenes-Multimodal Test (official split, cameras only)
206 scenes, 23,690 samples. The model receives cameras and ego-motion only; map and route inputs are absent. Ground truth comes from the official KITScenes `poses.txt`. Neither checkpoint saw these scenes during training.
| Model | 1 s | 2 s | 3 s | 5 s | Lateral | Longitudinal |
|---|---:|---:|---:|---:|---:|---:|
| nuPlan Epoch 5 (zero-shot) | 0.160 / 0.313 | **0.398 / 0.973** | **0.792 / 2.162** | 2.112 / 6.175 | **1.058** | 1.550 |
| KITScenes Epoch 5 | **0.146 / 0.311** | 0.412 / 1.045 | 0.826 / 2.220 | **2.104 / 5.942** | 1.204 | **1.405** |
### KITScenes-Multimodal Validation (cameras + HD map + route)
117 scenes, 11,035 samples. The route is an oracle reconstructed from the logged future trajectory, so these numbers measure route-conditioned driving rather than online route planning. Ground truth is the `trajectory_xy_m` field of the AutoE2E KITScenes v3.5 shards published under [`datasets/kitscenes-val`](./datasets/kitscenes-val/README.md). That field was produced by an older coordinate conversion; rescoring against the official `poses.txt` is tracked in issue #210.
| Model | 1 s | 2 s | 3 s | 5 s | Lateral | Longitudinal |
|---|---:|---:|---:|---:|---:|---:|
| nuPlan Epoch 5 (zero-shot) | 0.195 / 0.378 | 0.472 / 1.133 | 0.915 / 2.423 | 2.295 / 6.388 | 1.191 | 1.663 |
| KITScenes Epoch 5 | **0.147 / 0.285** | **0.373 / 0.924** | **0.745 / 2.020** | **1.941 / 5.564** | **0.984** | **1.402** |
KITScenes labels stop at 5 s, so the 6.4 s horizon cannot be scored on KITScenes.
### Held-out validation during training (6.4 s)
| Model | Validation set | Samples | ADE @ 6.4 s | FDE @ 6.4 s |
|---|---|---:|---:|---:|
| nuPlan Epoch 5 | nuPlan held-out training logs | 1,024 | 1.267 | 3.898 |
| KITScenes Epoch 5 | KITScenes train-dev scenes | 3,820 | 2.633 | 7.557 |
These sets were used to choose the epoch, so treat them as in-distribution reference values rather than benchmark scores.
The per-run reports, including sample counts and dataset digests, are under [`evaluations/overlay-replay-v1`](./evaluations/overlay-replay-v1). They score the predictions stored for the dashboard. The GRU planner is deterministic, so these stored controls are the model's predictions for every sample. Comparisons with other end-to-end models (METEOR, VAD, UniAD, Drive-JEPA, Qwen-Drive, Alpamayo, SimForge) are coordinated in [issue #210](https://github.com/autowarefoundation/auto_e2e/issues/210), where each protocol difference is documented.
## Predictions
Purple is the logged ground-truth path and green is the AutoE2E prediction, drawn into the cameras and the bird's-eye view. Every frame of these scenes can be played back in the [DataModelConsole dashboard](https://d2itskdqq39tx1.cloudfront.net/).
*KITScenes Epoch 5 entering a roundabout in Karlsruhe (validation, map and route available).*
*nuPlan Epoch 5, never trained on German roads, turning left at a night-time intersection (KITScenes Test, cameras only).*
*nuPlan Epoch 5 following a left curve on a forest road (KITScenes Test, cameras only).*
Camera images: [KITScenes-Multimodal](https://kitscenes.com/), CC BY-NC 4.0.
## Quick start
The checkpoints are PyTorch training checkpoints. They load with the AutoE2E source at the revision used to train them:
```bash
git clone https://github.com/autowarefoundation/auto_e2e.git
cd auto_e2e
git checkout 40a75cf8a26b60f6b094125e3924fe0bdb427b02 # experimental/reactive-bev-learning
pip install -r requirements.txt huggingface_hub
```
```python
import json
import sys
import torch
from huggingface_hub import hf_hub_download
sys.path.insert(0, "Model")
from model_components.auto_e2e import AutoE2E
from training.reactive_multitask import ReactiveTrainingStage, reactive_model_kwargs
repo_id = "AutowareFoundation/auto_e2e"
release = json.load(open(hf_hub_download(repo_id, "config.json")))
entry = release["checkpoints"]["nuplan-epoch-5"] # or "kitscenes-epoch-5"
checkpoint_path = hf_hub_download(repo_id, entry["path"])
payload = torch.load(checkpoint_path, map_location="cpu", weights_only=True)
config = payload["config"]
model = AutoE2E(
backbone=config["backbone"],
embed_dim=config["embed_dim"],
is_pretrained=False,
**reactive_model_kwargs(
ReactiveTrainingStage(config["training_stage"]),
num_views=config["num_views"],
),
)
model.load_state_dict(payload["model_state_dict"], strict=True)
model.eval()
```
Each checkpoint also contains the optimizer, scheduler and training state, so it can resume or seed fine-tuning directly. The evaluation entry point for KITScenes is `evaluate_reactive_kitscenes_checkpoint` in [`Platform/pipelines/distributed_training.py`](https://github.com/autowarefoundation/auto_e2e/blob/40a75cf8a26b60f6b094125e3924fe0bdb427b02/Platform/pipelines/distributed_training.py).
## Files
| Path | Contents |
|---|---|
| `config.json` | Release description: architecture, input/output contract, checkpoint paths and SHA-256 digests |
| `models/