CUDA error causes the micp_localization node to die

[Mh-Magdy]

First of all thank you for this great package and the amazing work.
I'm running micp node with combining unit = cpu and backed optix and everything is OK. I changed the combining unit to gpu i got the following error:

I edited it back to cpu and i got the same error, also changed the backend to embree same error :(

could you help me please to skip this error?

[Mh-Magdy]

The complete configuration file for referencing:

base_frame: base_link
map_frame: map
odom_frame: odom
tf_rate: 50
micp:
combining_unit: cpu
corr_rate_max: 500
adaptive_max_dist: True #
viz_corr: True
print_corr_rate: False
disable_corr: False
trans: [0.0, 0.0, 0.0]
rot: [0.0, 0.0, 0.0] # euler angles (3) or quaternion (4)

sensors:
velodyne:
topic: mid/points
type: spherical
model:
range_min: 0.5
range_max: 90.0
phi_min: -0.261799067259
phi_inc: 0.03490658503988659
phi_N: 16
theta_min: -3.14159011841
theta_inc: 0.01431249500496489
theta_N: 440
micp:
max_dist: 2.0
adaptive_max_dist_min: 0.15
backend: optix

[Mh-Magdy]

I changed the CUDA-toolkit version from 12.6 to 11.8 and the issue dissappeard.

[amock]

Hi @Mh-Magdy,

thanks for testing. However, that's weird. Normally, it should run with any cuda version. So I would say it's still an issue. So I will reopen it as a reminder for me to check this.

Could you give me some more info about your setup that you used?

• operating system
• cuda version (nvcc --version)
• gpu driver version (nvidia-smi)
• OptiX version
• ROS version
• rmcl version or branch
• rmagine version or branch
• did it also fail when running the https://github.com/amock/rmcl_example with your GPU config file?

With this I think I could reproduce the error and hopefully fix it soon. (Or someone else)

[Mh-Magdy]

Hiii @amock 👋

• OS: Ubuntu 20.04
• CUDA: 12.6
• GPU driver: nvidia-driver-560 - third-party non-free recommended
• Optix: 7.7
• ROS: ROS1 noetic
• RMCL: noetic branch
• rmagine: latest main branch on GitHub
• Actually it worked just fine with the example configuration for the CPU version and embree backend, but with the gpu it give me the error i mentioned above and when i changed the cuda version and gpu driver version it worked (The example config).

There are another minor issues that faced me recently after the update, i will report them to you in more details but i will give you a hint about them now:

When i perform some edits on the sensor parameters for example changing the number of horizontal samples to match my real sensor (theta_inc, theta_N) if backed is optix the package fails to run and if i change it to embree it works fine. I will capture any issues like these and give you details on the issue and my environment/setup as well as the config to help you reproduce the errors.

Thank you Alexander

[amock]

Hi @Mh-Magdy,

I have finally found some time to deal with your issue. First I tried to resemble your setup:

• Ubuntu 20.04.6 LTS
• ROS1 noetic
• GPU: RTX 2060 Super, Driver 560.35.03, CUDA V12.6.77
• OptiX 7.7
• rmagine: (main branch) 2.2.7
• rmcl: noetic branch
• rmcl_msgs: noetic branch
• rmcl_example: noetic branch

In the first terminal I started the example simulation by executing:

roslaunch rmcl_example start_robot.launch

then I changed the rmcl_example/launch/rmcl_micp.launch to load the parameters from rmcl_example/config/micp_gpu.yaml.
This config uses the GPU for both computing the correspondences and combining the covariances.
So I assumed this setup would cause the error you described.
Then I executed:

roslaunch rmcl_example rmcl_micp.launch

In the RViz window I set an initial pose guess and everything went fine. So unfortunately, I could not reproduce the error you described. Could you maybe try the exact same procedure on your system? Otherwise I am not sure what is wrong on your system :/ Perhaps you could also check if rmagine alone is working. There are some benchmark executables in it. Or perhaps you could check if CUDA is working for other projects.

Best
Alex

[Mh-Magdy]

Hi Alexander @amock ,

I hope you are doing well. I’d like to share some observations from a recent experiment related to this issue.

In the initial experiments that led to opening this issue, I ran the entire software suite (rmcl, mesh_nav, and Gazebo) inside a Docker container that utilized the host machine’s GPU driver. However, due to my limited experience with Docker, I occasionally misconfigured environment variables and mismanaged Docker layers. As a result, the container did not properly utilize the NVIDIA GPU driver libraries—particularly liboptixnv.so, which is required by rmagine beside the downloaded optix headers to generate the rmagine-optix executable.

Although the rmagine-optix executable appeared to build successfully without errors, it actually caused multiple runtime issues, including the errors discussed above.

I hope these observations help clarify the challenges and contribute to finding a solution. Thank you for your collaboration and support.

Best regards,
Muhammad

[amock]

Hi Muhammad @Mh-Magdy,

The last months, we were working on putting everything into Docker images as well -- partly because our situation forced us to do so. But as nice side effect, we are planning to upload preconfigured Docker files and provide intructions how to use them. It comes with the next update which brings features like object pose tracking and convenience tools for scan filtering. Preview: https://www.youtube.com/watch?v=9i3B1ayvMn4 .

Thanks for sharing your insights; they might help us with our Docker setup! And yes, this libnvoptix thing is quite important. (If anyone from NVIDIA is reading this, please consider integrating this library to Jetpack.)

Best
Alex

[kk4hvd]

@amock Hallo amock Thank you for nice work.

Actually I have a same issue what he mentioned about.
I'm contacting you because I think I can give you a little more detail on the situation.

OS: Ubuntu 22.04
GPU: Nvidia 2060 mobile(NUC 11 Enthusiast Kit )
CUDA: 11.8
GPU driver: NVIDIA-SMI 550.127.05
Optix: 7.6
ROS: ROS2 humble
RMCL: Main branch

/--------------------Yaml config file ---------------------/

rmcl_lidar3d_conversion:
ros__parameters:

sensor_frame: base_scan

debug_cloud: True
# filter / pre-processing:
# skip_begin: 0
# skip_end: 0
# increment: 1



use_sim_time: True
debug_cloud: True

model:
  phi_n: 128
  phi_min: -0.7853
  phi_inc: 0.0123 # vertical deg2rad ( 90deg / 128ch )
  theta_min: -3.14159
  theta_inc: 0.0123 # horizontal deg2rad ( 360deg / 512 )
  theta_n: 512
  range_min: 0.3
  range_max: 81.0


# filter / pre-processing:
width:
  skip_begin: 0 # skip elements from begin  
  skip_end:   0 # skip elements from end
  increment:  8 # increment to skip data. Simple filter to reduce scan density
height:
  skip_begin: 0 # cutting height from begin
  skip_end:   0 # cutting height from end
  increment:  1 # increment to skip data. Simple filter to reduce scan density 

Parameters for the MICP-L node

rmcl_micpl:
ros__parameters:

# required frames
#base_frame: base_footprint
base_frame: base_scan
map_frame: map
odom_frame: odom

# maximum number of correction steps per second
# lower this to decrease the correction speed but save energy 
disable_correction: False
correction_rate_max: 20.0
optimization_iterations: 5
tf_time_source: 1 # 0: measurement_latest, 1: correction_latest
tf_rate: 100.0
broadcast_tf: True
publish_pose: True
pose_noise: 0.01 # minimum noise of pose. can be set from sensor noise

# adjust max distance dependent of the state of localization
adaptive_max_dist: True # enable adaptive max dist

# initial offset applied to every incoming pose guess
pose_guess_offset: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0] # translation + euler angles (6) or translation + quaternion (7)

# very first initial pose guess
initial_pose_guess: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0] # translation + euler angles (6) or translation + quaternion (7)

# describe your sensor setup here

sensors: # list of range sensors - at least one is required
  lidar3d:
    data_source: topic
    model_type: spherical #spherical    o1dn
    topic_name: /rmcl_inputs/lidar3d



    correspondences:
      backend: optix
      type: RC
      metric: P2L
      adaptive_max_dist_min: 0.5
      max_dist: 2.0
      visualize: True

1. Works fine
   model_type: o1dn with "conv_pc2_to_o1dn_node"

2. Not working -> PROBLEM
   model_type: spherical with "conv_pc2_to_scan_node"
   

I went to where the problem was occurring.

RCCOptix.cpp -> void RCCOptixSpherical::find(const rm::Transform& Tbm_est)

void RCCOptixSpherical::find(const rm::Transform& Tbm_est)
{
std::cout << "FIND IN!" << std::endl;

//if(model_buffers_.points.empty())
//{
    //  std::cout << "model_buffers_.points empty!!!!!!!!!!!!!!!!!!!!!" << std::endl;
    


  size_t n_old_measurements = model_buffers_.points.size();
  size_t n_new_measurements = m_model->size();

  std::cout << "Optix - n_old_measurements" << n_old_measurements<< std::endl;
  std::cout << "Optix - n_new_measurements" << n_new_measurements<< std::endl;


  std::cout << "Optix - m_model->getHeight" << m_model->getHeight() << std::endl;
  std::cout << "Optix - m_model->getWidth" << m_model->getWidth()<< std::endl;
  if(n_new_measurements > n_old_measurements)
  {
    rm::resize_memory_bundle<rm::VRAM_CUDA>(model_buffers_, m_model->getHeight(), m_model->getWidth(), 1);
  }

  simulate(Tbm_est, model_buffers_);
//}

}

model_buffers_.points.size() <- when I access to model_buffer, process has died.

What is the problem? How can I fix this?

[amock]

[NOTE!]
From here on I guess we are talking about a problem of rmcl v2.2.1 . Some parameters have changed their names since previous versions. Therefore there was an increase of the minor version number.

First of all: It shouldn't break like this, so thanks for stress-testing this a bit more.

The problem seems to originate from the change from conv_pc2_to_o1dn_node to conv_pc2_to_scan_node. Both are nodes that convert a PointCloud2 message into one of the internal representations. Converting the cloud to the o1dn representation works, but converting it to the spherical representation somehow crashes the executable.

You could start by disabling the MICP-L node, so that only the conversion is done. Then you may want to check whether the debug_cloud is being published, which visualizes the final result of the conversions node for debugging purposes. It should look similar to the input cloud. Or have you already checked that?

If the debug cloud looks good, then I’m not sure what’s going wrong just by looking at isolated pieces of code. The safest way for me to help would be a repository, such as a fork of the rmcl_examples, where you try to reproduce the error. Push the changes there, and I can download it and reproduce the same issue. Then I could start digging deeper.

[kk4hvd]

I think it is different issue. Please let me open new issue and I will describe more specific.

[amock]

Sure, feel free to open another issue!