Privetely Owned Vehicle 02/16/2025 Working Group Meeting Slot 1 & 2 weekly updates

[pranavreddy23]

VisionPilot Weekly Update

Agenda

• Milestones
• Full pipeline integration
• QNX Integration
• ZENOH integration
• SceneSeg transfer-learning off-road WG
• AutoSteer Network (EgoPath + EgoLanes)
• CARLA integration steering pipeline
• AutoSpeed
• Object Finder
• E2E Research

---

Milestones

• End of October – VisionPilot stack ready for testing in simulator (on track)
• End of November – VisionPilot stack deployed on edge hardware and benchmarked
  • First target hardware: Nvidia AGX ORIN
• End of December – VisionPilot stack deployed on actual vehicle and able to demonstrate autonomous driving

---

Full Pipeline Integration

@pranavreddy23

Pranav has begun work on integrating the full VisionPilot pipeline for version 0.9 release. Currently both pipelines run parallelly and asynchronously.

Some Synchronization and cleaning work is pending.

AutoSteer Network (EgoPath + EgoLanes)

@m-zain-khawaja @TranHuuNhatHuy

• EgoLanes model successfully trained by @TranHuuNhatHuy with strong results.
• Post-processing pipeline implemented to ensure seamless ego-lane and driving corridor detection throughout OEM highway autonomous drive.
• Latest results are promising and ready for demo pipeline integration.

Next Steps

• Match predicted curve parameters:
  • Lane offset
  • Yaw offset
  • Curvature
• Compare against corresponding ground truth parameters.

---

QNX Integration

The Garraio team (Mohammad Wagdy and Mahmoud) are working on QNX integration of the VisionPilot stack.

• PathFinder QNX implementation & testing: ✅ Completed
• AutoSpeed integration: 🚧 In progress

---

ZENOH Integration

@evshary @m-zain-khawaja

CY has completed the Zenoh pipeline for:

• SceneSeg
• Scene3D
• DomainSeg

Using VisionPilot/common folder libraries.

Capabilities

• ONNX Runtime backend
• TensorRT backend
• Same shared framework as ROS2

Next Steps

• Add new models (AutoSpeed / AutoSteer)
• Full pipeline integration

---

SceneSeg Transfer-Learning Off-Road WG

@mhmdrz-mirdamadi

Muhammad Reza is evaluating a "stixels"-inspired approach to SceneSeg for off-road environments.

• Network design: ✅ Completed
• Semantic branch pre-training: ✅ Completed

---

CARLA Integration – Steering Pipeline

@JITERN @atanasko

• CARLA 0.10.0 and 0.9.16 support: ✅ Complete (Python API)
• Controller nodes (Lateral & Longitudinal): ✅ Completed
• PathFinder ROS2 node: ✅ Completed
• CARLA integration: ✅ Completed

Ongoing Work

• Improve controller feedforward terms:
  • Curvature for steering
  • Nominal speed demand for longitudinal control

Atanasko:

• Integrating CARLA Scenario Runner with CARLA 0.9.16
• Running NCAP scenarios in OpenScenario format for simulation testing

---

Rerun & Camera SDK Integration

@pranavreddy23

• Rerun visualization and logging: ✅ Implemented
• Memory allocation issues: 🚧 Being investigated

---

AutoSpeed

@pranavreddy23 @atanasko

• First version of AutoSpeed network: ✅ Trained
• ONNX model: ✅ Created
• TensorRT FP16 model: ✅ Created
• ROS2 node + standalone version: ✅ Implemented

Additional Updates

• CPU support for AutoSpeed: ✅ Ready
• Sent for OpenAD kit integration
• PR merged into main branch

---

Object Finder

@pranavreddy23

Work has begun on the C++ version of the ObjectFinder module.

Features

• Tracks:
  • Closest-in-path object (CIPO)
  • Other moving objects
• Uses OpenLane dataset ground truth data

Completed Components

• Calibration module (homography using Waymo Open Dataset)
• Position-only measurement system + Kalman filter
• ORB + BF feature mapping:
  • Detect cut-in scenarios
  • Maintain main CIPO tracking

Code cleaning is in progress. PR will be raised soon.

---

E2E Research

@auo355 – Zaynab El Mawas

Augustine and Zaynab are conducting a literature review of full monolithic E2E models and experimental technologies.

Kevin has started a PoC test of Nvidia's COSMOS network using CARLA simulator to evaluate E2E simulation pipelines.

Literature Review

https://docs.google.com/document/d/1ls-7LhYuCL1SbgiJBoCRg62UZ9FdcgLv5uA2zSYTIkY/edit?usp=sharing

---

Zoom Meeting Recording

Please contact the work group lead (@m-zain-khawaja) to request access to the meeting recording.