ur_robot_driver recover from errors due to late robot startup

[s-ohl-ostfalia-de]

I would like to have the ur_robot_driver running as a system service all the time. This works fine if it is started while the robot (ur5e) is started and motors are enabled. If it is starting late and the driver is already running or the robot is shutdown and then restarted, it first complains about communication errors and then about the motors not beeing enabled.

If I now start sending movement commands, it will not move until I restart the robot-driver (not the robot or the motors).

Is there a clean way have the driver recover from errors and continue working (without restarting it)?

Startup command:
ros2 launch ur_robot_driver ur_control.launch.py ur_type:=ur5e robot_ip:=192.168.1.160 headless_mode:=true launch_rviz:=false initial_joint_controller:=scaled_joint_trajectory_controller kinematics_params_file:=/calibration/ur5e-calibration-arm3.yaml launch_dashboard_client:=false

OS: Ubuntu 22.04
ROS2 release: humble

The following logs occur if the robot is starting while the driver is already running:
{Logs}
[ur_ros2_control_node-1] [WARN] [1770642040.461906545] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.464046525] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.466176073] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.468301814] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.470430338] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.472569070] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.474718233] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.476887139] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.479197123] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.481374540] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.483558261] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.485759261] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.487915079] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.490098608] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.492321942] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.494465537] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.496664713] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.498801525] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.501200079] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.503352762] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.505500581] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.507644656] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.509782396] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.511918248] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.514085330] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.516255324] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.518754159] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.520949655] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [WARN] [1770642040.523168285] [UR_Client_Library:]: Unable to get data package while reconnecting to the RTDE interface
[ur_ros2_control_node-1] [ERROR] [1770642040.523645404] [UR_Client_Library:]: Failed to initialize RTDE client after 3 attempts, unable to reconnect
[robot_state_helper-2] [INFO] [1770642045.374841447] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A2
[robot_state_helper-2] [INFO] [1770642045.375079862] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A0
[robot_state_helper-2] [INFO] [1770642045.375113044] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A1
[robot_state_helper-2] [INFO] [1770642045.375128947] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A3
[ur_ros2_control_node-1] [INFO] [1770642045.375123667] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A2
[ur_ros2_control_node-1] [INFO] [1770642045.375283208] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A0
[ur_ros2_control_node-1] [INFO] [1770642045.375304166] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A1
[ur_ros2_control_node-1] [INFO] [1770642045.375319397] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A3
[ur_ros2_control_node-1] [INFO] [1770642046.077571743] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A4
[robot_state_helper-2] [INFO] [1770642046.077834925] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A4
[ur_ros2_control_node-1] [INFO] [1770642046.579454192] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A2
[robot_state_helper-2] [INFO] [1770642046.579529771] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A2
[ur_ros2_control_node-1] [INFO] [1770642049.995686497] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A4
[robot_state_helper-2] [INFO] [1770642049.995686305] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A4
[ur_ros2_control_node-1] [INFO] [1770642050.797213426] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A5
[robot_state_helper-2] [INFO] [1770642050.797353800] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A5
[ur_ros2_control_node-1] [INFO] [1770642057.723891422] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A7
[robot_state_helper-2] [INFO] [1770642057.724097937] [UR_Client_Library:]: Logging an ErrorCodeMessage from the UR Controller Box: C100A7

{/Logs}

[urfeex]

Thank you for reporting this. We are currently in the process of massively refactoring the RTDE communication which will touch this significantly. I would first try try reproduce this and then make a quick test with the refactored version and then come back to this.

[urfeex]

Looking a bit deeper into this I see two main issues:

• When the robot disconnects, reading data from the robot fails. This triggers an RTDE reconnection that works just fine as long as you restart the robot early enough. If the robot doesn't reconnect before the timeout for that hits (your log output states "Failed to initialize RTDE client after 3 attempts, unable to reconnect"), the RTDE client will remain stopped. The proper way would probably be to bring the hardware interface to its lifecycle state unconfigured which we currently don't. You can do that manually by using the respective ros2_control service calls / the ros2control-cli. Once you bring back the hardware interface to state active afterwards, it should be operational again (except for the next point)
• The robot runs a program reading external commands coming from the ROS driver. This has to be explicitly started and you probably do that in your driver startup by passing headless_mode:=true to the launch file. If the program has been interrupted, it can be sent to the robot for execution again (given that it's in remote control) by using the resend_robot_program service.

TLDR:

You can bring the robot driver back to life using

ros2 control set_hardware_component_state ur5e unconfigured
ros2 control set_hardware_component_state ur5e active
ros2 service call /io_and_status_controller/resend_robot_program std_srvs/srv/Trigger