Request for official evaluation code for Waymo reproduction

[xixixix-cyber]

Hi, thanks for open-sourcing VectorWorld! I am currently trying to reproduce the Waymo evaluation results, but I noticed that the repository does not seem to contain the official eval pipeline used in the paper.

Right now, my eval code is self-implemented based on the Scenario Dreamer evaluation code and adapted to VectorWorld, so I am not fully confident that my protocol matches the one used in the paper. This makes it difficult to determine whether the metric gap comes from the checkpoint/configuration or from evaluation differences.

Would it be possible to also release the official evaluation code used for the paper?

I believe releasing the eval code would greatly improve reproducibility for the community. It would also help clarify whether the discrepancy comes from the model/checkpoint/configuration or simply from differences in evaluation implementation.

Thanks again for the great work!

[jiangchaokang]

Hi @xixixix-cyber,

Thank you for your interest in VectorWorld and for the detailed feedback!

We apologize for the inconvenience. Currently, we have only released a portion of the codebase, and several components — including the official evaluation pipeline — are still being organized. As the paper is currently under review, our bandwidth is largely occupied by the rebuttal process, which has slowed down our progress on cleaning up and releasing the remaining code.

We plan to release the complete evaluation code as soon as the review process concludes. We appreciate your patience.

In the meantime, we'd like to clarify that most of the evaluation metrics used in our paper are largely consistent with, or identical to, those in Scenario Dreamer. Therefore, your current self-implemented evaluation pipeline adapted from Scenario Dreamer should be a reasonable approximation. We believe any discrepancy you observe is more likely attributable to model/checkpoint/configuration differences rather than evaluation protocol differences.

Sorry again for the incomplete release at this stage, and thanks for raising this — it's exactly the kind of feedback that helps us improve reproducibility for the community.

Best regards