[WIP] Add 781 (#922)

* Add Flake8 configuration

Uses the configuration from ci_scripts

* Add mypy configuration file

Based on the ci_scripts parameters.

* Pre-commit mypy flake8, add flake8 excludes

Any venv folder does not need flake8.
The example directory got flake8 warnings so I assumed it should be
excluded.

* Add Black to pre-commit

Add ignore E203 as Black will observe PEPs specification for white space
around a colon it is next to an expression.

* Set max line length to 100

* Blacken code

There are a few places where big indentation is introduced that may
warrant refactoring so it looks better.
I did not refactor anything yet, but did exlude three (?) lists (of ids)
to not be formatted.

* Add unit tests to flake8 and mypy pre-commit

* Use pre-commit for flake8, mypy and black checks

This ensures it runs with the same versions and settings as developers.

* Update docs, add 'test' dependencies

Add two other developer dependencies not strictly required for unit
tests, but required for development.
I think the overlap between people who want to execute unit tests and
perform commits is (close to) 100% anyway.

* Uninstall pytest-cov on appveyor ci

It seems to cause an error on import due to a missing sqlite3 dll.
As we don't check coverage anyway, hopefully just uninstalling is
sufficient.

* Add -y to uninstall

* Sphinx issue fix (#923)

* Sphinx issue fix

* Removing comment

* More robust handling of openml_url (#921)

I ran into issues when the openml server config is not exactly 'https://www.openml.org/api/v1/xml', e.g. I had 'https://www.openml.org/api/v1'.
I only noticed when getting a bad dataset url.

This edit makes the API more robust against how exactly the server URL is set in the config.

* format for black artifacts

* Add Flake8 configuration

Uses the configuration from ci_scripts

* Add mypy configuration file

Based on the ci_scripts parameters.

* Pre-commit mypy flake8, add flake8 excludes

Any venv folder does not need flake8.
The example directory got flake8 warnings so I assumed it should be
excluded.

* Add Black to pre-commit

Add ignore E203 as Black will observe PEPs specification for white space
around a colon it is next to an expression.

* Set max line length to 100

* Blacken code

There are a few places where big indentation is introduced that may
warrant refactoring so it looks better.
I did not refactor anything yet, but did exlude three (?) lists (of ids)
to not be formatted.

* Add unit tests to flake8 and mypy pre-commit

* Use pre-commit for flake8, mypy and black checks

This ensures it runs with the same versions and settings as developers.

* Update docs, add 'test' dependencies

Add two other developer dependencies not strictly required for unit
tests, but required for development.
I think the overlap between people who want to execute unit tests and
perform commits is (close to) 100% anyway.

* Uninstall pytest-cov on appveyor ci

It seems to cause an error on import due to a missing sqlite3 dll.
As we don't check coverage anyway, hopefully just uninstalling is
sufficient.

* Add -y to uninstall

* format for black artifacts

Co-authored-by: Neeratyoy Mallik <neeratyoy@gmail.com>
Co-authored-by: Joaquin Vanschoren <joaquin.vanschoren@gmail.com>

Author: 3 people

.flake8

@@ -0,0 +1,10 @@
+[flake8]
+max-line-length = 100
+show-source = True
+select = C,E,F,W,B
+ignore = E203, E402, W503
+per-file-ignores =
+    *__init__.py:F401
+exclude =
+    venv
+    examples

.pre-commit-config.yaml

@@ -0,0 +1,24 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 19.10b0
+    hooks:
+      - id: black
+        args: [--line-length=100]
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v0.761
+    hooks:
+      - id: mypy
+        name: mypy openml
+        files: openml/*
+      - id: mypy
+        name: mypy tests
+        files: tests/*
+  - repo: https://gitlab.com/pycqa/flake8
+    rev: 3.8.3
+    hooks:
+      - id: flake8
+        name: flake8 openml
+        files: openml/*
+      - id: flake8
+        name: flake8 tests
+        files: tests/*

CONTRIBUTING.md

@@ -109,29 +109,37 @@ following rules before you submit a pull request:
  - If any source file is being added to the repository, please add the BSD 3-Clause license to it.
 
 
-You can also check for common programming errors with the following
-tools:
-
--  Code with good unittest **coverage** (at least 80%), check with:
-
+First install openml with its test dependencies by running
   ```bash
-  $ pip install pytest pytest-cov
-  $ pytest --cov=. path/to/tests_for_package
+  $ pip install -e .[test]
   ```
-
--  No style warnings, check with:
-
+from the repository folder.
+This will install dependencies to run unit tests, as well as [pre-commit](https://pre-commit.com/).
+To run the unit tests, and check their code coverage, run:
   ```bash
-  $ pip install flake8
-  $ flake8 --ignore E402,W503 --show-source --max-line-length 100
+  $ pytest --cov=. path/to/tests_for_package
   ```
-
--  No mypy (typing) issues, check with:
-
+Make sure your code has good unittest **coverage** (at least 80%).
+
+Pre-commit is used for various style checking and code formatting.
+Before each commit, it will automatically run:
+ - [black](https://black.readthedocs.io/en/stable/) a code formatter.
+   This will automatically format your code.
+   Make sure to take a second look after any formatting takes place,
+   if the resulting code is very bloated, consider a (small) refactor.
+   *note*: If Black reformats your code, the commit will automatically be aborted.
+   Make sure to add the formatted files (back) to your commit after checking them.
+ - [mypy](https://mypy.readthedocs.io/en/stable/) a static type checker.
+   In particular, make sure each function you work on has type hints.
+ - [flake8](https://flake8.pycqa.org/en/latest/index.html) style guide enforcement.
+   Almost all of the black-formatted code should automatically pass this check,
+   but make sure to make adjustments if it does fail.
+    
+If you want to run the pre-commit tests without doing a commit, run:
   ```bash
-  $ pip install mypy
-  $ mypy openml --ignore-missing-imports --follow-imports skip
+  $ pre-commit run --all-files
   ```
+Make sure to do this at least once before your first commit to check your setup works.
 
 Filing bugs
 -----------

appveyor.yml

@@ -36,6 +36,8 @@ install:
   - "cd C:\\projects\\openml-python"
   - "pip install .[examples,test]"
   - "pip install scikit-learn==0.21"
+  # Uninstall coverage, as it leads to an error on appveyor
+  - "pip uninstall -y pytest-cov"
 
 
 # Not a .NET project, we build scikit-learn in the install step instead

ci_scripts/flake8_diff.sh

@@ -58,7 +58,8 @@ if [[ "$COVERAGE" == "true" ]]; then
     pip install codecov pytest-cov
 fi
 if [[ "$RUN_FLAKE8" == "true" ]]; then
-    pip install flake8 mypy
+    pip install pre-commit
+    pre-commit install
 fi
 
 # Install scikit-learn last to make sure the openml package installation works

ci_scripts/install.sh

@@ -28,7 +28,7 @@ run_tests() {
 }
 
 if [[ "$RUN_FLAKE8" == "true" ]]; then
-    source ci_scripts/flake8_diff.sh
+    pre-commit run --all-files
 fi
 
 if [[ "$SKIP_TESTS" != "true" ]]; then

ci_scripts/test.sh

@@ -23,8 +23,8 @@
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 # sys.path.insert(0, os.path.abspath('.')# )
 
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
 
 # -- General configuration ------------------------------------------------
 
@@ -35,38 +35,38 @@
 #  extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 #  ones.
 extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.autosummary',
-    'sphinx.ext.doctest',
-    'sphinx.ext.coverage',
-    'sphinx.ext.mathjax',
-    'sphinx.ext.ifconfig',
-    'sphinx.ext.autosectionlabel',
-    'sphinx_gallery.gen_gallery',
-    'numpydoc'
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.doctest",
+    "sphinx.ext.coverage",
+    "sphinx.ext.mathjax",
+    "sphinx.ext.ifconfig",
+    "sphinx.ext.autosectionlabel",
+    "sphinx_gallery.gen_gallery",
+    "numpydoc",
 ]
 
 autosummary_generate = True
 numpydoc_show_class_members = False
 
-autodoc_default_flags = ['members', 'inherited-members']
+autodoc_default_flags = ["members", "inherited-members"]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # The suffix of source filenames.
-source_suffix = '.rst'
+source_suffix = ".rst"
 
 # The encoding of source files.
 # source_encoding = 'utf-8-sig'
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = "index"
 
 # General information about the project.
-project = u'OpenML'
-copyright = (
-    u'2014-{}, the OpenML-Python team.'.format(time.strftime("%Y,%m,%d,%H,%M,%S").split(',')[0])
+project = u"OpenML"
+copyright = u"2014-{}, the OpenML-Python team.".format(
+    time.strftime("%Y,%m,%d,%H,%M,%S").split(",")[0]
 )
 
 # The version info for the project you're documenting, acts as replacement for
@@ -90,7 +90,7 @@
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = ['_build', '_templates', '_static']
+exclude_patterns = ["_build", "_templates", "_static"]
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
@@ -108,7 +108,7 @@
 # show_authors = False
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = "sphinx"
 
 # A list of ignored prefixes for module index sorting.
 # modindex_common_prefix = []
@@ -121,69 +121,57 @@
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'bootstrap'
+html_theme = "bootstrap"
 
 html_theme_options = {
     # Navigation bar title. (Default: ``project`` value)
-    'navbar_title': "OpenML",
-
+    "navbar_title": "OpenML",
     # Tab name for entire site. (Default: "Site")
     # 'navbar_site_name': "Site",
-
     # A list of tuples containting pages to link to.  The value should
     # be in the form [(name, page), ..]
-    'navbar_links': [
-        ('Start', 'index'),
-        ('User Guide', 'usage'),
-        ('API', 'api'),
-        ('Examples', 'examples/index'),
-        ('Contributing', 'contributing'),
-        ('Changelog', 'progress'),
+    "navbar_links": [
+        ("Start", "index"),
+        ("User Guide", "usage"),
+        ("API", "api"),
+        ("Examples", "examples/index"),
+        ("Contributing", "contributing"),
+        ("Changelog", "progress"),
     ],
-
     # Render the next and previous page links in navbar. (Default: true)
-    'navbar_sidebarrel': False,
-
+    "navbar_sidebarrel": False,
     # Render the current pages TOC in the navbar. (Default: true)
-    'navbar_pagenav': False,
-
+    "navbar_pagenav": False,
     # Tab name for the current pages TOC. (Default: "Page")
-    'navbar_pagenav_name': "On this page",
-
+    "navbar_pagenav_name": "On this page",
     # Global TOC depth for "site" navbar tab. (Default: 1)
     # Switching to -1 shows all levels.
-    'globaltoc_depth': 1,
-
+    "globaltoc_depth": 1,
     # Include hidden TOCs in Site navbar?
     #
     # Note: If this is "false", you cannot have mixed ``:hidden:`` and
     # non-hidden ``toctree`` directives in the same page, or else the build
     # will break.
     #
     # Values: "true" (default) or "false"
-    'globaltoc_includehidden': "false",
-
+    "globaltoc_includehidden": "false",
     # HTML navbar class (Default: "navbar") to attach to <div> element.
     # For black navbar, do "navbar navbar-inverse"
-    'navbar_class': "navbar",
-
+    "navbar_class": "navbar",
     # Fix navigation bar to top of page?
     # Values: "true" (default) or "false"
-    'navbar_fixed_top': "true",
-
+    "navbar_fixed_top": "true",
     # Location of link to source.
     # Options are "nav" (default), "footer" or anything else to exclude.
-    'source_link_position': "None",
-
+    "source_link_position": "None",
     # Bootswatch (http://bootswatch.com/) theme.
     #
     # Options are nothing with "" (default) or the name of a valid theme
     # such as "amelia" or "cosmo".
-    'bootswatch_theme': "flatly",
-
+    "bootswatch_theme": "flatly",
     # Choose Bootstrap version.
     # Values: "3" (default) or "2" (in quotes)
-    'bootstrap_version': "3",
+    "bootstrap_version": "3",
 }
 
 # Add any paths that contain custom themes here, relative to this directory.
@@ -224,7 +212,7 @@
 # html_use_smartypants = True
 
 # Custom sidebar templates, maps document names to template names.
-html_sidebars = {'**': ['localtoc.html']}
+html_sidebars = {"**": ["localtoc.html"]}
 
 # Additional templates that should be rendered to pages, maps page names to
 # template names.
@@ -257,27 +245,26 @@
 # html_file_suffix = None
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = 'OpenMLdoc'
+htmlhelp_basename = "OpenMLdoc"
 
 
 # -- Options for LaTeX output ---------------------------------------------
 
 latex_elements = {
     # The paper size ('letterpaper' or 'a4paper').
     # 'papersize': 'letterpaper',
-
     # The font size ('10pt', '11pt' or '12pt').
     # 'pointsize': '10pt',
-
     # Additional stuff for the LaTeX preamble.
     # 'preamble': '',
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
-latex_documents = [('index', 'OpenML.tex', u'OpenML Documentation',
-                    u'Matthias Feurer', 'manual'), ]
+latex_documents = [
+    ("index", "OpenML.tex", u"OpenML Documentation", u"Matthias Feurer", "manual"),
+]
 
 # The name of an image file (relative to this directory) to place at the top of
 # the title page.
@@ -304,10 +291,7 @@
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [
-    ('index', 'openml', u'OpenML Documentation',
-     [u'Matthias Feurer'], 1)
-]
+man_pages = [("index", "openml", u"OpenML Documentation", [u"Matthias Feurer"], 1)]
 
 # If true, show URL addresses after external links.
 # man_show_urls = False
@@ -319,9 +303,15 @@
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    ('index', 'OpenML', u'OpenML Documentation',
-     u'Matthias Feurer', 'OpenML', 'One line description of project.',
-     'Miscellaneous'),
+    (
+        "index",
+        "OpenML",
+        u"OpenML Documentation",
+        u"Matthias Feurer",
+        "OpenML",
+        "One line description of project.",
+        "Miscellaneous",
+    ),
 ]
 
 # Documents to append as an appendix to all manuals.
@@ -343,13 +333,13 @@
 # Sphinx-gallery configuration.
 sphinx_gallery_conf = {
     # disable mini galleries clustered by the used functions
-    'backreferences_dir': None,
+    "backreferences_dir": None,
     # path to the examples
-    'examples_dirs': '../examples',
+    "examples_dirs": "../examples",
     # path where to save gallery generated examples
-    'gallery_dirs': 'examples',
+    "gallery_dirs": "examples",
     # compile execute examples in the examples dir
-    'filename_pattern': '.*example.py$|.*tutorial.py$',
+    "filename_pattern": ".*example.py$|.*tutorial.py$",
     # TODO: fix back/forward references for the examples.
 }