Repository navigation
[Feature]: AutoE2E Publication, Benchmarking and Code Cleanup #210
Description
Problem / motivation
We have published our open-loop model artefacts on HuggingFace, we need to finalize our release of the models, and benchmarks - link: https://huggingface.co/AutowareFoundation/auto_e2e
We need to finalize publication of HuggingFace models with their appropriate benchmarks and visualizations as a formal Autoware Robotaxi WG release and we also need to write a paper which summarises our work.
At the same time, our repo has become quite bloated with un-used code and tests, which we should clean-up.
Proposed solution
1 - Visualizations
We need visualizations of the NuPlan trained model on the NuPlan dataset
2 - Benchmarks
We have benchmarks for ADE and FDE for our baseline NuPlan trained model and KITScenes finetuned model only on the KITScenes Val and Test sets.
We should benchmark our model on other public datasets:
We should benchmark other open models, e.g. Alpamayo 1.5, SimForge, DriveJEPA, UniAD, METEOR on KITScenes dataset and NuPlan dataset so that we can have an apples-to-apples comparison.
KIT Scenes Test Set - ADE/FDE metrics
AutoE2E NuPlan, AutoE2E KIT, Alapamayo 1.5, SimForge, DriveJEPA, UniAD, METEOR, VAD, Qwen-Drive-1.0
NuPlan Test Set - ADE/FDE metrics
AutoE2E NuPlan, AutoE2E KIT, Alapamayo 1.5, SimForge, DriveJEPA, UniAD, METEOR, VAD, Qwen-Drive-1.0
Metrics to be reported:
ADE/FDE at 1s, 2s, 3s, 4s and 5s
Request to code contributors - please start the benchmarking and report in the comments what model you are benchmarking and on what dataset and then share your results in the comments. Please READ the comments so you don't double up on a benchmark.
Pipeline for testing/benchmarking is provided, use as is
Official public benchmarking of AutoE2E KITScenes Epoch 5 model
Please submit the AutoE2E KITScenes Epoch 5 model checkpoint trajectory outputs to the KITScenes lab for official evaluation and public benchmarking release
3 - Code cleanup
After 1) and 2) are completed, we need to purge all unnecessary tests and features and parts of the codebase which are not leveraged and update the documentation. We should remove all unnecessary feature branches and merge the core features into the main branch.
4 - Paper publication
We need to summarize our findings as an academic paper and publish on ArXiv.
5 - HuggingFace publication
We need to fix and clean up the HuggingFace publication and make it easy for users to get started with the model
Alternatives considered
No response
Activity
Nishad-Sahu commentedon Sep 23, 2026
Hi, as discussed in the E2E working group meeting on 23rd Sep, I can try running the Benchmarking for UniAD.
Arseni10Lk commentedon Sep 23, 2026
Hey, I will be benchmarking AutoE2E NuPlan on KITScenes test set 3.5.
And while we are waiting for it to be pinpointed, @m-zain-khawaja @riita10069, what version of the NuPlan dataset are we using? There are v1.0 and v1.1. Which of these test splits should be used for benchmarking?
atanasko commentedon Sep 28, 2026
Hi,
I try to evaluate Qwen-Drive-1.0 and Alpamayo-1.5 models on KITScenes-Multimodal/test subset:
NVIDIA GeForce RTX 3090
Driver Version: 580.178.04
CUDA Version: 13.0
Qwen-Drive was tested to predict 6 trajectories per keyframe using camera_ring_front, camera_ring_front_left and camera_ring_front_right
Alpamayo-1.5 was tested to predict 6 trajectories per keyframe using camera_ring_front, camera_ring_front_left and camera_ring_front_right
AutoE2e was tested to predict 1 trajectories per keyframe using camera_base_front_center, camera_ring_front_left, camera_ring_front_right, camera_ring_rear_left, camera_ring_rear, camera_ring_rear_right
nuplan-epoch-4
riita10069 commentedon Sep 30, 2026
@Arseni10Lk
I used nuPlan v1.1 Full dataset
m-zain-khawaja commentedon Sep 30, 2026
@riita10069 - aim to add visualization for NuPlan test set, similar to KITScenes dataset
m-zain-khawaja commentedon Sep 30, 2026
KIT Scenes benchmarking process:
Work on the 'experimental' branch that was used to train the latest AutoE2E model
Data
@riita10069 - share validation shards (map + cameras + egomotion history) on HuggingFace for KIT Scenes dataset on HuggingFace, this is essential since the official test set of the KIT Scenes dataset does not have GPS or HD-map which is a necessary input for the AutoE2E model, so we need to use the same validation data that was used by Ryota-san.
Benchmarking
We need to use the official KIT Scenes Multimodal dataset benchmarking SDK: https://github.com/KIT-MRT/kitscenes/tree/7765cdec5490894266070ab46e23724b58b3da42
We can modify the SDK return values to ensure that the data being fed to each of the models conforms to the required inputs for that model.
We need the same benchmarking script for all models:
def evaluate_reactive_kitscenes_checkpoint(
checkpoint: FlyteFile,
shards: List[FlyteDirectory],
source_split: str = "val",
batch_size: int = 1,
num_loader_workers: int = 4,
expected_test_partition_count: int = 0,
expected_val_partition_count: int = 0,
use_stateful_camera_fpn_cache: bool = False,
) -> ReactiveKITScenesEvaluationOutput:
"""Evaluate a KITScenes fine-tuned checkpoint without checkpoint selection."""
import tempfile
import torch
from data_parsing.pre_extracted import (
discover_split_inventory,
make_multi_dataset_loader,
)
from data_parsing.kit_scenes.temporal_contract import (
KITSCENES_BENCHMARK_FUTURE_STEPS,
kitscenes_temporal_contract,
)
from distributed_training.reactive_data import (
build_reactive_dataset_plan,
)
from model_components.auto_e2e import AutoE2E
from model_components.trajectory_planning.gru_planner import GRUPlanner
from training.reactive_multitask import (
ReactiveTrainingStage,
reactive_model_kwargs,
)
from training.reactive_stage_runner import (
evaluate_reactive_multitask,
inspect_reactive_checkpoint_identity,
)
if not isinstance(batch_size, int) or isinstance(batch_size, bool):
raise ValueError(
"KITScenes evaluation batch size must be an integer"
)
if source_split not in {"val", "overlap_train_val", "test"}:
raise ValueError(
"KITScenes labeled evaluation requires val, overlap_train_val, "
"or test"
)
evaluation_precision_policy = _kitscenes_evaluation_precision_policy()
maximum_batch_size = (
KITSCENES_STATEFUL_CAMERA_FPN_BATCH_SIZE
if use_stateful_camera_fpn_cache
else 4
)
if not 1 <= batch_size <= maximum_batch_size:
raise ValueError(
"KITScenes evaluation batch size exceeds the selected "
"inference policy"
)
if not 0 <= num_loader_workers <= 4:
raise ValueError(
"KITScenes evaluation loader workers must be between zero and four"
)
if not shards:
raise ValueError("KITScenes evaluation shards must not be empty")
mapless_test = source_split == "test"
if use_stateful_camera_fpn_cache and (
source_split not in {"val", "test"}
or batch_size != KITSCENES_STATEFUL_CAMERA_FPN_BATCH_SIZE
):
raise ValueError(
"stateful camera FPN evaluation requires official scene data and "
f"batch size {KITSCENES_STATEFUL_CAMERA_FPN_BATCH_SIZE}"
)
if not mapless_test and expected_test_partition_count:
raise ValueError(
"expected_test_partition_count is only valid for test data"
)
if source_split != "val" and expected_val_partition_count:
raise ValueError(
"expected_val_partition_count is only valid for val data"
)
if (
expected_val_partition_count
== KITSCENES_OFFICIAL_VAL_SCENE_COUNT
and batch_size != KITSCENES_PRIMARY_VAL_BATCH_SIZE
):
raise ValueError(
"official KITScenes val evaluation requires batch size "
f"{KITSCENES_PRIMARY_VAL_BATCH_SIZE}"
)
remote_uris = [_flyte_remote_uri(shard) for shard in shards]
plan = build_reactive_dataset_plan(
remote_uris,
stage=ReactiveTrainingStage.KITSCENES_FINETUNE,
allow_mapless_kitscenes_evaluation=mapless_test,
)
expected_manifest_sha256s: dict[str, str] = {}
expected_shard_sha256s: dict[tuple[str, str], str] = {}
for reference in plan.shards:
previous_manifest_sha256 = expected_manifest_sha256s.setdefault(
reference.partition_id,
reference.manifest_sha256,
)
if previous_manifest_sha256 != reference.manifest_sha256:
raise ValueError(
"KITScenes dataset plan mixes partition manifests"
)
expected_shard_sha256s[
(reference.partition_id, reference.shard_name)
] = reference.shard_sha256
shard_directories: list[str] = []
manifest_identities: list[dict[str, object]] = []
dataset_names: set[str] = set()
dataset_versions: set[str] = set()
source_revisions: set[str] = set()
manifest_group_uids: list[str] = []
scene_uids: list[str] = []
expected_sample_count = 0
for shard in shards:
directory = Path(shard.download())
manifest_path = directory / "manifest.json"
manifest_bytes = manifest_path.read_bytes()
manifest = json.loads(manifest_bytes)
if (
not isinstance(manifest, dict)
or manifest.get("source_split") != source_split
or manifest.get("data_role") != "benchmark"
or manifest.get("temporal_sampling")
!= kitscenes_temporal_contract(benchmark_protocol=True)
):
raise ValueError(
"KITScenes evaluation manifest has the wrong split, role, "
"or temporal contract"
)
partition_id = str(manifest.get("partition_id") or "")
total_samples = int(manifest.get("total_samples", 0))
if total_samples < 0:
raise ValueError(
"KITScenes evaluation manifest has invalid sample count"
)
if source_split == "val" and total_samples > 0:
_validate_kitscenes_val_manifest_contract(manifest)
if mapless_test and total_samples == 0 and any(
bool(manifest.get(field, False))
for field in (
"has_gps",
"has_map",
"has_navigation",
"has_reactive_navigation",
"has_route_reconstruction",
"has_trajectory_xy",
)
):
raise ValueError(
"empty KITScenes test partition advertises target data"
)
if total_samples > 0:
shard_directories.append(str(directory))
actual_manifest_sha256 = hashlib.sha256(
manifest_bytes
).hexdigest()
if (
expected_manifest_sha256s.get(partition_id)
!= actual_manifest_sha256
):
raise ValueError(
"downloaded KITScenes manifest differs from the plan"
)
shard_names = manifest.get("shard_names")
if not isinstance(shard_names, list) or not shard_names:
raise ValueError(
"KITScenes evaluation manifest has no shards"
)
for shard_name_value in shard_names:
shard_name = str(shard_name_value)
shard_path = directory / shard_name
expected_shard_sha256 = expected_shard_sha256s.get(
(partition_id, shard_name)
)
if (
expected_shard_sha256 is None
or not shard_path.is_file()
or _sha256_file(shard_path)
!= expected_shard_sha256
):
raise ValueError(
"downloaded KITScenes shard differs from the plan"
)
expected_sample_count += total_samples
dataset_names.add(str(manifest.get("dataset") or ""))
dataset_versions.add(str(manifest.get("dataset_version") or ""))
source_revisions.add(str(manifest.get("source_revision") or ""))
split_group_uids = manifest.get("split_group_uids")
if (
source_split in {"val", "test"}
and (
not isinstance(split_group_uids, list)
or len(split_group_uids) != 1
or not isinstance(split_group_uids[0], str)
or not split_group_uids[0].startswith("kitscenes-")
)
):
raise ValueError(
f"KITScenes {source_split} partition lacks one scene identity"
)
if source_split in {"val", "test"} and total_samples > 0:
manifest_group_uids.append(split_group_uids[0])
if source_split in {"val", "test"}:
scene_uids.append(split_group_uids[0])
manifest_identities.append({
"manifest_sha256": hashlib.sha256(
manifest_bytes
).hexdigest(),
"partition_id": partition_id,
"split_group_uids": split_group_uids,
"total_samples": int(manifest.get("total_samples", 0)),
})
if (
len(dataset_names) != 1
or "" in dataset_names
or len(dataset_versions) != 1
or "" in dataset_versions
or len(source_revisions) != 1
or "" in source_revisions
):
raise ValueError(
"KITScenes evaluation manifests mix dataset provenance"
)
inventory = (
discover_split_inventory(
shard_directories,
allow_single_group=True,
)
if shard_directories
else None
)
if expected_sample_count != plan.total_samples or (
inventory is not None
and inventory.sample_count != plan.total_samples
):
raise ValueError(
"KITScenes evaluation sample inventory differs from manifests"
)
if inventory is None and (not mapless_test or expected_sample_count != 0):
raise ValueError("KITScenes evaluation has no non-empty shards")
if inventory is not None and source_split in {"val", "test"}:
expected_group_uids = tuple(sorted(manifest_group_uids))
if inventory.group_uids != expected_group_uids:
raise ValueError(
"KITScenes packed scene identities differ from manifests"
)
scene_uid_sha256 = ""
is_official_test = False
is_official_val = False
if mapless_test:
scene_uid_sha256, is_official_test = (
_kitscenes_scene_identity(
scene_uids,
source_split="test",
expected_partition_count=expected_test_partition_count,
)
)
elif source_split == "val":
scene_uid_sha256, is_official_val = (
_kitscenes_scene_identity(
scene_uids,
source_split="val",
expected_partition_count=expected_val_partition_count,
)
)
_validate_kitscenes_official_val_sdk_split(
scene_uids,
expected_partition_count=expected_val_partition_count,
)
if source_split in {"val", "test"}:
_validate_kitscenes_official_sample_inventory(
source_split=source_split,
expected_partition_count=(
expected_test_partition_count
if mapless_test
else expected_val_partition_count
),
empty_partition_count=plan.empty_partition_count,
manifest_sample_count=expected_sample_count,
inventory_sample_count=(
inventory.sample_count if inventory is not None else None
),
)
checkpoint_path = Path(checkpoint.download())
checkpoint_identity = inspect_reactive_checkpoint_identity(
checkpoint_path
)
payload = torch.load(
checkpoint_path,
map_location="cpu",
weights_only=False,
)
config = payload.get("config")
state_dict = payload.get("model_state_dict")
checkpoint_epoch = payload.get("epoch")
if (
not isinstance(config, Mapping)
or not isinstance(state_dict, Mapping)
or isinstance(checkpoint_epoch, bool)
or not isinstance(checkpoint_epoch, int)
or checkpoint_epoch <= 0
or config.get("training_stage")
!= ReactiveTrainingStage.KITSCENES_FINETUNE.value
):
raise ValueError(
"checkpoint is not a reviewed KITScenes fine-tuning checkpoint"
)
route_usage_evaluation_policy = (
_kitscenes_route_usage_evaluation_policy(
mapless_test=mapless_test,
)
)
if inventory is None:
trajectory_metrics: dict[str, float | int | None] = {
"valid_timestep_count": 0,
"total_timestep_count": 0,
"valid_horizon_coverage": None,
"mean_abs_longitudinal_error_m": None,
"mean_abs_lateral_error_m": None,
"nonfinite_prediction_count": 0,
"nonfinite_prediction_rate": None,
}
for horizon in ("1s", "2s", "3s", "5s", "6p4s"):
trajectory_metrics[f"ade_{horizon}_m"] = None
trajectory_metrics[f"fde_{horizon}_m"] = None
trajectory_metrics[f"ade_{horizon}_sample_count"] = 0
trajectory_metrics[f"fde_{horizon}_sample_count"] = 0
metrics = {
"schema_version": "reactive_multitask_evaluation_v1",
"sample_count": 0,
"sample_uid_sha256": hashlib.sha256(b"").hexdigest(),
"inference_precision": {
key: evaluation_precision_policy[key]
for key in (
"autocast_enabled",
"device_type",
"dtype",
)
},
"trajectory": trajectory_metrics,
"bev_segmentation": {
"available": False,
"reason": "targets_unavailable",
},
"route": {
"available": False,
"reason": "input_and_targets_unavailable",
},
}
else:
stage = ReactiveTrainingStage.KITSCENES_FINETUNE
model = AutoE2E(
backbone=str(config["backbone"]),
embed_dim=int(config["embed_dim"]),
is_pretrained=False,
**reactive_model_kwargs(
stage,
num_views=int(config["num_views"]),
),
)
model.load_state_dict(state_dict)
device = torch.device("cuda")
if not torch.cuda.is_available():
raise RuntimeError("KITScenes evaluation requires a GPU")
model.to(device)
planner = model.Reactive_E2E.TrajectoryPlanner
if not isinstance(planner, GRUPlanner):
raise ValueError(
"KITScenes evaluation requires the reviewed deterministic "
"GRU planner"
)
camera_fpn_cache = (
model.create_stateful_camera_fpn_cache()
if use_stateful_camera_fpn_cache
else None
)
loader = make_multi_dataset_loader(
shard_directories,
batch_size=batch_size,
num_workers=num_loader_workers,
split="all",
val_fraction=0.0,
shuffle=0,
pin_memory=True,
prefetch_factor=1,
max_active_loaders=1,
decode_future_frames=False,
decode_front_camera_fpn=use_stateful_camera_fpn_cache,
)
metrics = evaluate_reactive_multitask(
model,
loader,
stage=stage,
device=device,
include_counterfactuals=bool(
route_usage_evaluation_policy[
"counterfactuals_enabled"
]
),
include_route_swap_counterfactual=bool(
route_usage_evaluation_policy[
"route_swap_counterfactual_enabled"
]
),
reuse_precomputed_image_bev=bool(
route_usage_evaluation_policy[
"reuse_precomputed_image_bev"
]
),
include_route_gradient=bool(
route_usage_evaluation_policy[
"input_gradient_enabled"
]
),
camera_fpn_cache=camera_fpn_cache,
inference_autocast_dtype=torch.bfloat16,
)
_validate_kitscenes_runtime_precision(
metrics,
evaluation_precision_policy,
)
trajectory_metrics = metrics.get("trajectory")
if not isinstance(trajectory_metrics, dict):
raise ValueError("KITScenes evaluation omitted trajectory metrics")
trajectory_metrics["ade_6p4s_m"] = None
trajectory_metrics["fde_6p4s_m"] = None
trajectory_metrics["ade_6p4s_sample_count"] = 0
trajectory_metrics["fde_6p4s_sample_count"] = 0
if inventory is not None and (
metrics.get("sample_count") != inventory.sample_count
or metrics.get("sample_uid_sha256")
!= inventory.sample_uid_digest
):
raise ValueError(
"KITScenes evaluation did not cover the exact packed inventory"
)
report = {
"schema_version": "reactive_kitscenes_labeled_evaluation_v1",
"checkpoint_epoch": checkpoint_epoch,
"checkpoint_sha256": checkpoint_identity["checkpoint_sha256"],
"dataset": next(iter(dataset_names)),
"dataset_version": next(iter(dataset_versions)),
"evaluation_role": (
"official_test_camera_only_missing_map_route"
if is_official_test
else "official_val_camera_map_route"
if is_official_val
else "test_subset_camera_only_missing_map_route"
if mapless_test
else "labeled_external_benchmark"
),
"input_track": (
"camera_only_missing_map_route"
if mapless_test
else "camera_map_route"
),
"inference_cache_policy": (
KITSCENES_STATEFUL_CAMERA_FPN_POLICY
if use_stateful_camera_fpn_cache
else "stateless"
),
"evaluation_batch_size": batch_size,
"evaluation_precision_policy": evaluation_precision_policy,
"trajectory_inference_policy": {
"planner": "gru",
"stochastic_noise": False,
"version": KITSCENES_TRAJECTORY_INFERENCE_POLICY,
},
"route_usage_evaluation_policy": (
route_usage_evaluation_policy
),
"maximum_labeled_horizon_steps": (
KITSCENES_BENCHMARK_FUTURE_STEPS
),
"partition_count": len(manifest_identities),
"expected_partition_count": (
expected_test_partition_count
if mapless_test
else expected_val_partition_count
if source_split == "val"
else 0
),
"expected_sample_count": (
KITSCENES_OFFICIAL_TEST_SAMPLE_COUNT
if is_official_test
else KITSCENES_OFFICIAL_VAL_SAMPLE_COUNT
if is_official_val
else 0
),
"scene_uid_sha256": (
scene_uid_sha256
if source_split in {"val", "test"}
else None
),
"manifest_identities": sorted(
manifest_identities,
key=lambda item: str(item["partition_id"]),
),
"metrics": metrics,
"source_revision": next(iter(source_revisions)),
"source_split": source_split,
}
report_bytes = (
json.dumps(
report,
allow_nan=False,
indent=2,
sort_keys=True,
)
+ "\n"
).encode("ascii")
report_path = (
Path(tempfile.mkdtemp(prefix="reactive-kitscenes-eval-"))
/ "report.json"
)
report_path.write_bytes(report_bytes)
return ReactiveKITScenesEvaluationOutput(
report=FlyteFile(str(report_path)),
report_sha256=hashlib.sha256(report_bytes).hexdigest(),
checkpoint_sha256=checkpoint_identity["checkpoint_sha256"],
checkpoint_epoch=checkpoint_epoch,
)
auto_e2e/Platform/pipelines/distributed_training.py
Lines 3257 to 3784 in 40a75cf
KIT Benchmark scores - We need ADE/FDE @ 1s, 2s, 3s and 5s:
m-zain-khawaja commentedon Sep 30, 2026
@atanasko - Qwen Drive 1.0 and Alapamyo 1.5
@Arseni10Lk - METEOR
@Nishad-Sahu - UniAD
Hassan Ansari - Drive JEPA
@riita10069 - VAD
@CodeName-Detective SimForge
Please share results ASAP
riita10069 commentedon Sep 30, 2026
experimental/reactive-bev-learningm-zain-khawaja commentedon Sep 30, 2026
Next steps:
riita10069 commentedon Sep 30, 2026
@m-zain-khawaja @gcordova10 @Arseni10Lk @atanasko @Nishad-Sahu
Here is the KIT Val data
https://huggingface.co/AutowareFoundation/auto_e2e/tree/main/datasets/kitscenes-val
21 remaining items
CodeName-Detective commentedon Oct 8, 2026
@riita10069 @ZaynabEM @m-zain-khawaja
Before I proceed with the SimForge evaluation on nuPlan, could you please confirm that we are still using nuPlan v1.1 TEST for the external-model benchmark, and clarify the intended input/output and evaluation-population contract?
I have now inspected the official nuPlan v1.1 TEST release in some detail so that the SimForge adapter can follow the dataset correctly.
What I found from nuPlan v1.1 TEST
public_set_test_sensor.txt.CAM_F0,CAM_B0,CAM_L0/L1/L2,CAM_R0/R1/R2.Proposed SimForge input contract
For the common 3-camera comparison, the most natural mapping appears to be:
CAM_L0→ SimForge front-leftCAM_F0→ SimForge frontCAM_R0→ SimForge front-rightSimForge would receive its native temporal inputs:
t0-0.3, t0-0.2, t0-0.1, t0t0-1.5 ... t0For clarity, although I am proposing a nuPlan
lidar_pctoken/timestamp as the common identifier fort0, the LiDAR point cloud itself is not an input to SimForge. Thelidar_pctimestamp would only provide a model-independent way to define the exact evaluation moment shared across different models.Proposed output/scoring contract
For ground truth, I plan to use the official nuPlan ego-pose trajectory, interpolated to:
t0+0.1 ... t0+6.4 sand transformed directly into the
t0ego frame.The scoring would then match the KITScenes evaluation:
For models producing multiple trajectories, I would report the same:
Definitions used in the KITScenes benchmark.
The nuPlan rear-axle frame currently appears to be the most likely physical trajectory reference point for SimForge as well, based on the available DriveWorks / Physical-AI coordinate conventions and the Alpamayo action-space behaviour, although I am still keeping that assumption explicitly documented.
Could you please confirm the following?
Are we still using nuPlan v1.1 TEST for the external-model benchmark?
Should each external model use its native input requirements, as we did for KITScenes?
For SimForge/Qwen-style models, should the common comparison use the three forward cameras (
CAM_L0,CAM_F0,CAM_R0), or should each model use its full native camera configuration?Should the output/scoring contract remain:
ADE/FDE @ 1 / 2 / 3 / 4 / 5 s, with first / mean / minimum metrics where applicable?
Most importantly, what is the intended nuPlan TEST benchmark population?
For the population, could you provide either:
lidar_pctokens/timestamps from the 147 sensor-enabled TEST logs, orThis is the main remaining blocker. There are hundreds of thousands of technically valid timestamps across the 147 sensor-enabled TEST logs, and a substantial fraction correspond to stationary periods, so independently choosing a stride could produce a population different from what the other models are evaluated on.
I want to make sure SimForge is evaluated on the same nuPlan moments as the other external models.
riita10069 commentedon Oct 9, 2026
@m-zain-khawaja
I updated DataModelConsole for NuPlan test set.
https://d2itskdqq39tx1.cloudfront.net/datasets/nuplan-test?version=v1.2
and I cleaned up HuggingFace Model Card.
https://huggingface.co/AutowareFoundation/auto_e2e
m-zain-khawaja commentedon Oct 10, 2026
Hi @CodeName-Detective - thanks for sharing the SimForge results, we are skipping NuPlan benchmarking for now
m-zain-khawaja commentedon Oct 10, 2026
Hi @riita10069 - thank you, HF looks good, please also add an icon in our main README file below the hero image which points to our HF. DataConsole looks great, I will use the clips to make our teaser video.
docs(readme): add a community section with social links and the footer
ZaynabEM commentedon Oct 12, 2026
Recap: KITScenes benchmark - where the data comes from, ground truth, and how runs are done
Due to the discussion within the thread and the many results reported, I wanted to create a recap so that we can all agree on what has been done and how we got the results. Please correct anything that doesn't match what you ran.
1. Data origin
KIT-MRT/KITScenes-Multimodal). Canonical ground truth is pinned to dataset revision6fde0034.data/val/, 117 episodes, 11,035 samples. The 23 episodes ofdata/overlap_train_val/(2,490 samples) are excluded. The v3.5 mirror holds 140 episodes (117 + 23).datasets/kitscenes-val/v3.5inAutowareFoundation/auto_e2e, with squashed/square images (512×512 / 1024×1024), HD map, ego-motion history and route.2. Evaluation populations
Two populations are reported in this thread.
a) Dense validation track: all 117 scenes, 11,035 samples.
b) 202 keyframes:
Model/navigation/lanelet2_matcher.py).3. Ground truth
poses.txt(@6fde0034) using the current direct local-pose transform, expressed in the ego frame (x forward, y left). Implemented by_local_future_to_ego_xy()+pose_yaws()inModel/data_parsing/kit_scenes/(commitc44be718).trajectory_xy_m. It comes from the sameposes.txtstream but through the older path (local XY → UTM → WGS84 →wgs84_future_to_ego_xy()), which applied the UTM-grid yaw to true-north lat/lon deltas. Result: a heading offset of about 0.17 to 0.36° that is nearly constant within a scene, and about 0.1 to 0.2 m of sideways error at 3 s.trajectory_xy_m.poses.txtandtrajectory_xy_mblocks, each labelled.poses.txt.4. How benchmarks are run
experimental/reactive-bev-learningbranch (the one used to train the latest AutoE2E), the official KITScenes benchmarking SDK, and a common evaluation script (evaluate_reactive_kitscenes_checkpointwithsource_split="val",Platform/pipelines/distributed_training.py). Each model gets its own adapter script.camera_base_front_centerplus the ring cameras), HD map and an oracle route. The 41 keyframes without a valid route are map-only.m-zain-khawaja commentedon Oct 14, 2026
Code Cleanup
Step 1
Review and merge or close all PRs - we currently have 17 PRs (except closed loop sim PRs)
Step 2
Respond to, and close out and resolve all open issues - we currently have 27 Issues (except closed loop sim issues)
Step 3
Resolve, merge and rebase our existing branches into 2 branches only, dev and main - we currently have 34 branches
Step 4
Review docs, diagrams and instructions
Step 5
Test it out, make sure we haven't introduced any bugs in our code cleanup process - test the data parsing, training, evaluation, visualization etc
Step 6
We should publish the AutoE2E v0.5 release - open loop trained E2E model
Nishad-Sahu commentedon Oct 14, 2026
@ZaynabEM This is the updated UniAD results. The results improved significantly by using the original KIT Scenes images without any image processing. Although in the overall picture it is still the worst performing model.
Updated report - UniAD (trained on nuScenes) zero shot evaluation on KITscenes.: https://drive.google.com/file/d/19KTiIsYhA2qP6-IIMeozkc_1KJO0K2kY/view?usp=sharing.
A final results summary table from the report is attached here :
riita10069 commentedon Oct 19, 2026
@ZaynabEM Thank you for the recap! It matches what I ran, with a few small notes:
data/val/(11,035 samples) plus the 23 indata/overlap_train_val/(2,490 samples).camera_base_front_centerplus the five ring cameras other thancamera_ring_front.camera_base_front_center).I've rescored both checkpoints against the canonical
poses.txtGT (117 scenes, 11,035 samples, ADE / FDE in m):The predictions are the same as in my earlier post, and the legacy rows reproduce it exactly. Please use the canonical rows for the paper table.
m-zain-khawaja commentedon Oct 21, 2026
@riita10069 - could you please add this benchmarks table to the main README and also our HuggingFace listing
Title: Error (m) compared to human driving (smaller is better) - KIT Multimodal dataset
Note: AutoE2E Finetuned means KIT trained epoch 5 and AutoE2E means NuPlan trained epoch 5
docs(readme): add a Benchmarks section with the KITScenes 3 s ADE/FDE…
docs(readme): animate the benchmark chart and match its labels to the…
docs(readme): add the KITScenes accuracy benchmark chart
atanasko commentedon Oct 28, 2026
@riita10069
Note from @m-zain-khawaja can you please clean the branches and merge changes so we have one main and one develop branch?
ZaynabEM commentedon Nov 4, 2026
@riita10069 @m-zain-khawaja @gcordova10 @atanasko @Arseni10Lk I have uploaded the paper here
Let me know if you find anything that needs to be modified. As for my part, as discussed I will rework the figures to unify the content. I have invited you as reviewers to the paper on overleaf, so you can comment directly on the part that you believe needs modification. Here is the link for viewing the paper on overleaf: https://www.overleaf.com/read/tdwstzxxdnjd#fa4528