-
Notifications
You must be signed in to change notification settings - Fork 57
Description
For any retest of C++ test suite, the gtest output xml file gets overwritten. This is problematic, for example, if the test suite contains a flaky test. It could fail on the first run, but pass on the second run (given that --retest-until-pass is passed to colcon test). The generated XML output file contains only test results of the last test execution, any previous test exeuction gets overwritten. The only way to detect such cases would be to manually parse the output of colcon test, line by line.
I would suggest to generate separate XML files for each run, e.g.:
build/<pkg>/test_results/<pkg>/<test_suite_name>_run_1.gtest.xml
build/<pkg>/test_results/<pkg>/<test_suite_name>_run_2.gtest.xml
build/<pkg>/test_results/<pkg>/<test_suite_name>_run_3.gtest.xml
...
The problem is that all previous files must be deleted before calling the test suite, as there might be only one test run, while the previous one had multiple ones, and those previous XML files are then being copied to the Testing folder.
Ideally, the output files should be merged, but since gtest does not allow to merge output files, that operation requires manual XML parsing and generation.