ENH: Add generate-lcov-html flag

[ganesh-k13]

Changes

• Added generate-lcov-html flag in test

Example

~/os/devpy/example_pkg (main) » spin test --gcov --coverage
Invoking `build` prior to running tests:                                                                                                                                            
Removing previous GCOV .gcda files...                                                                                                                                               
$ meson setup build --prefix=/usr -Db_coverage=true
The Meson build system        
Version: 0.64.0                            
.
.
.
.
.
---------- coverage: platform linux, python 3.10.12-final-0 ----------
Name                                            Stmts   Miss  Cover
-------------------------------------------------------------------
example_pkg/__init__.py                             3      0   100%
example_pkg/conftest.py                             0      0   100%
example_pkg/submodule/__init__.py                   0      0   100%
example_pkg/submodule/tests/__init__.py             0      0   100%
example_pkg/submodule/tests/test_submodule.py       2      0   100%
example_pkg/tests/__init__.py                       0      0   100%
example_pkg/tests/test_core.py                      4      0   100%
-------------------------------------------------------------------
TOTAL                                               9      0   100%
Coverage HTML written to dir /home/ganesh/os/devpy/example_pkg/build/coverage/

~/os/devpy/example_pkg (main) » spin test --generate-lcov-html
Deleting old HTML coverage report...
Generating HTML coverage report...
$ ninja coverage-html -C build
Coverage report generated successfully and written to /home/ganesh/os/devpy/example_pkg/build/meson-logs/coveragereport/index.html

---

Other error handling

» spin build  
» spin test --generate-lcov-html
Error: GCOV files missing... Cannot generate coverage reports. Coverage reports can be generated by `spin test --coverage --gcov`

---

Related

• Based on ENH: Added coverage, gcov and generate-lcov-html flags in spin test numpy/numpy#24992
• key missing spin commands for developer workflow numpy/numpy#24080
• ENH: Added coverage option for test command #100
• Add --gcov-report flag to spin test #159
• Add gcov flags for build #146

ToDo

• Add UT

[czgdp1807]

Adding one point here. I think its ninja which calls lcov when we pass --generate-lcov-html to spin (trying on this PR) ninja doesn't call lcov with --no-external flag. So the end result will include headers from dependencies as well. See details in scipy/scipy#21919 (comment) and scipy/scipy#21919 (comment). If there's a way to pass options to lcov via ninja CLI then we should pass --no-external flag.

[stefanv]

This will depend on the build path the user specifies, so has to be set inside the build fn.

[stefanv]

Instead of adding a new flag, what do you think of adding an html-lcov option to --gcov-format?

[ganesh-k13]

Yeah, it makes sense to club it. It would need a bit of rework I have used the enum to denote the coverage format directly. Let me see how we can make it neater and more generic.

[stefanv]

Thanks @ganesh-k13!

[ganesh-k13]

Adding one point here. I think its ninja which calls lcov when we pass --generate-lcov-html to spin (trying on this PR) ninja doesn't call lcov with --no-external flag. So the end result will include headers from dependencies as well. See details in scipy/scipy#21919 (comment) and scipy/scipy#21919 (comment). If there's a way to pass options to lcov via ninja CLI then we should pass --no-external flag.

@czgdp1807 , I searched quite a bit but I don't see any way to pass args to lcov via ninja. That being said, I'm not seeing external libraries listed for NumPy (example: numpy/numpy#24992). I'm trying to see how that's happening.

[czgdp1807]

Hi. So you can check my uploaded coverage reports for SciPy - https://github.com/czgdp1807/scipy-lcov-genhtml-coverage-output.

On main branch of above repository,

On no_external branch of above repository (passed --no-external flag here)

So, you can see, with --no-external results for only SciPy files are shown and without it, dependencies are also covered (noisy).

I tried to search for an environment variable through which we can pass CLI options to lcov. But there isn't such any. Google search results gave LCOV_HOME but that doesn't seem to be correct.

[ganesh-k13]

Bit of an oversight from me, but seems like --gcov-format html for Python and --gcov for C/CPP gives the exact same result as --generate-lcov-html in this PR. So I assume I can close this now?

@czgdp1807 , regarding the external library coverage being included, can you see if you get the same results for NumPy in your machine. If not, I assume it's something to do with the way SciPy is built that is different that we need to deep dive on.

[czgdp1807]

I verified. https://github.com/czgdp1807/scipy-lcov-genhtml-coverage-output contains the latest reports. I think we are good. spin test --gcov --coverage --gcov-format=html generates both Python coverage and C/C++/Fortran coverage. It calls "lcov" only for those files which are within SciPy. Go ahead and close this PR for now.