Tests dependency/ordering

[vid512]

One feature that some people including me are lacking in catch2 and/or snitch, is the ability to make test dependent on success of other tests.

For example, if you have a component A, and a component B which uses A internally, then you want A tests to run before B tests. If A tests are failing, then B tests are basically invalid, and there's no reason to run them. Currently, B tests might produce a lot of complicated errors and undefined states, which can confuse the programmer and steal his focus from cleanly caught errors in A tests. Especially if they appear first in a long list of failed tests.

See: catchorg/Catch2#60

My preferred solution would be to add a third parameter TEST_CASE, which makes the test case run only after specified test cases (tags) were successful.

[vid512]

Apparently, some people in Python dev, consider this a misfeature? (See:
https://stackoverflow.com/questions/3885489/test-framework-allowing-tests-to-depend-on-other-tests ).

Some suggest you should always mock the A in B tests. That seem to me to just add so much more work to achieve the same practical results, as long as A is opaque or simple enough in B's interface. I could understand doing that if tests take very long time to run, and you really want ability to test B separately from whether A is working, without having to re-run tests. That is not my case though, I am perfectly fine running B tests again after fixing A, much rather than having to mock everything used by other components. I haven't really used TDD or mocking in practice, so maybe I have some wrong idea about it?

Some people suggested to order/structure only test reports, so that you clearly see the cascading error from A to B, but still run all tests. OK but how? It seems to me as, at best, the more verbose of ordering/structuring tests themselves. I still need to specify (somehow somewhere) that test/result B depends on test/result A.

It still seems to me, there are cases when having this as an (opt-in) option would be useful.