Mark sklearn tests (#1202)

* Add sklearn marker

* Mark tests that use scikit-learn

* Only run scikit-learn tests multiple times

The generic tests that don't use scikit-learn should only be tested once
(per platform).

* Rename for correct variable

* Add sklearn mark for filesystem test

* Remove quotes around sklearn

* Instead include sklearn in the matrix definition

* Update jobnames

* Add explicit false to jobname

* Remove space

* Add function inside of expression?

* Do string testing instead

* Add missing ${{

* Add explicit true to old sklearn tests

* Add instruction to add pytest marker for sklearn tests

Author: PGijsbers

.github/workflows/test.yml

@@ -4,13 +4,14 @@ on: [push, pull_request]
 
 jobs:
   test:
-    name: (${{ matrix.os }}, Py${{ matrix.python-version }}, sk${{ matrix.scikit-learn }})
+    name: (${{ matrix.os }}, Py${{ matrix.python-version }}, sk${{ matrix.scikit-learn }}, sk-only:${{ matrix.sklearn-only }})
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
         python-version: [3.6, 3.7, 3.8]
         scikit-learn: [0.21.2, 0.22.2, 0.23.1, 0.24]
         os: [ubuntu-latest]
+        sklearn-only: ['true']
         exclude:  # no scikit-learn 0.21.2 release for Python 3.8
           - python-version: 3.8
             scikit-learn: 0.21.2
@@ -19,17 +20,22 @@ jobs:
             scikit-learn: 0.18.2
             scipy: 1.2.0
             os: ubuntu-latest
+            sklearn-only: 'true'
           - python-version: 3.6
             scikit-learn: 0.19.2
             os: ubuntu-latest
+            sklearn-only: 'true'
           - python-version: 3.6
             scikit-learn: 0.20.2
             os: ubuntu-latest
+            sklearn-only: 'true'
           - python-version: 3.8
             scikit-learn: 0.23.1
             code-cov: true
+            sklearn-only: 'false'
             os: ubuntu-latest
           - os: windows-latest
+            sklearn-only: 'false'
             scikit-learn: 0.24.*
       fail-fast:  false
       max-parallel: 4
@@ -62,7 +68,10 @@ jobs:
       if: matrix.os == 'ubuntu-latest'
       run: |
         if [ ${{ matrix.code-cov }} ]; then codecov='--cov=openml --long  --cov-report=xml'; fi
-        pytest -n 4 --durations=20 --timeout=600 --timeout-method=thread --dist load -sv $codecov --reruns 5 --reruns-delay 1
+        # Most of the time, running only the scikit-learn tests is sufficient
+        if [ ${{ matrix.sklearn-only }} = 'true' ]; then sklearn='-m sklearn'; fi
+        echo pytest -n 4 --durations=20 --timeout=600 --timeout-method=thread --dist load -sv $codecov $sklearn --reruns 5 --reruns-delay 1
+        pytest -n 4 --durations=20 --timeout=600 --timeout-method=thread --dist load -sv $codecov $sklearn --reruns 5 --reruns-delay 1
     - name: Run tests on Windows
       if: matrix.os == 'windows-latest'
       run: |  # we need a separate step because of the bash-specific if-statement in the previous one.

CONTRIBUTING.md

@@ -153,7 +153,8 @@ following rules before you submit a pull request:
 
 - Add [unit tests](https://github.com/openml/openml-python/tree/develop/tests) and [examples](https://github.com/openml/openml-python/tree/develop/examples) for any new functionality being introduced. 
     - If an unit test contains an upload to the test server, please ensure that it is followed by a file collection for deletion, to prevent the test server from bulking up. For example, `TestBase._mark_entity_for_removal('data', dataset.dataset_id)`, `TestBase._mark_entity_for_removal('flow', (flow.flow_id, flow.name))`.
-    - Please ensure that the example is run on the test server by beginning with the call to `openml.config.start_using_configuration_for_example()`.      
+    - Please ensure that the example is run on the test server by beginning with the call to `openml.config.start_using_configuration_for_example()`.
+    - Add the `@pytest.mark.sklearn` marker to your unit tests if they have a dependency on scikit-learn.
 
 -  All tests pass when running `pytest`. On
    Unix-like systems, check with (from the toplevel source folder):

tests/conftest.py

@@ -174,6 +174,10 @@ def pytest_sessionfinish() -> None:
     logger.info("{} is killed".format(worker))
 
 
+def pytest_configure(config):
+    config.addinivalue_line("markers", "sklearn: marks tests that use scikit-learn")
+
+
 def pytest_addoption(parser):
     parser.addoption(
         "--long",