You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2adf07d
Browse filesBrowse the repository at this point in the historyBrowse files
docs(readme): add a Benchmarks section with the KITScenes 3 s ADE/FDE chart and table
The chart and table show ADE and FDE at 3 s on KITScenes-Multimodal
validation for AutoE2E and seven other open-weights models. The figures
are the ones posted in issue autowarefoundation#210, and the table states the setting and
sample set behind each row. The section sits between "See it drive" and
"Get started", so those two later section headers move to 04 and 05.
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Copy file name to clipboardExpand all lines: README.md
+21Lines changed: 21 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -65,6 +65,27 @@ AutoE2E is three cooperating models. The **Reactive model** runs at 10 Hz and dr
65
65
66
66
The read-only [DataModelConsole dashboard](https://d2itskdqq39tx1.cloudfront.net/) replays synchronized camera scenes with ground-truth and predicted trajectories in camera and bird's-eye views, next to dataset versions, MLflow models and Flyte executions.
<imgsrc="./Media/readme/benchmark-kitscenes-3s.svg"alt="ADE and FDE at 3 s on KITScenes-Multimodal validation for eight open-weights driving models. The fine-tuned AutoE2E checkpoint is second, behind Qwen Drive 1.0."width="100%">
72
+
</p>
73
+
74
+
The fine-tuned AutoE2E checkpoint has the second-lowest error of the eight open-weights models we benchmarked on KITScenes-Multimodal validation. Only Qwen Drive 1.0 is ahead, and AutoE2E beats SimForge, Alpamayo 1.5 and METEOR with 79.9 M parameters. All models are scored open-loop: each one predicts the next 3 s from logged camera frames, and the prediction is compared with the path the human driver took. ADE is the average position error over those 3 s and FDE is the error at the 3 s point.
75
+
76
+
| Model | Checkpoint / setting | Evaluated on | ADE @ 3 s (m) | FDE @ 3 s (m) |
77
+
|---|---|---|---:|---:|
78
+
| Qwen Drive 1.0 | first of 6 trajectories | 202 keyframes | 0.565 | 1.397 |
The 11,035 samples are every frame of the 117 validation scenes, and the 202 keyframes are non-overlapping anchors taken from the same scenes. Inputs differ by model: AutoE2E also gets an HD map and the route, and VAD and Drive-JEPA get a driving command. [Issue #210](https://github.com/autowarefoundation/auto_e2e/issues/210) has each model's adapter, ground truth and results at the other horizons.
0 commit comments