|
| 1 | +Common XenServer/XCP-ng Python classes |
| 2 | +====================================== |
| 3 | + |
| 4 | +The `xcp` directory contains the Common XenServer and XCP-ng Python packages. |
| 5 | +They are intented for use in XenServer and XCP-ng Dom0 only and deal with logging, |
| 6 | +Hardware/PCI, networking, and other Dom0 tasks. |
| 7 | + |
| 8 | +The pip package name is `python-libs` which is also the rpm package name in XenServer. |
| 9 | +XCP-ng packages it as [xcp-python-libs](https://github.com/xcp-ng-rpms/xcp-python-libs) |
| 10 | +([koji](https://koji.xcp-ng.org/packageinfo?packageID=400)). |
| 11 | + |
| 12 | +It supports Python 2.7 and is currently in progress to get further fixes for >= 3.6. |
| 13 | +It depends on `six`, and on Python 2.7, also `configparser` and `pyliblzma`. |
| 14 | + |
| 15 | +Test-driven Development (TDD) Model |
| 16 | +----------------------------------- |
| 17 | +This package has CI which can be run locally but is also run in GitHub CI to ensure |
| 18 | +Test-driven development. |
| 19 | + |
| 20 | +The Continous Integration Tests feature: |
| 21 | +- Combined coverage testing of Python 2.7 and Python 3.8 code branches |
| 22 | +- Automatic Upload of the combined coverage to CodeCov (from the GitHub Workflow) |
| 23 | +- Checking of the combined coverage against the diff to master: Fails if changes are not covered! |
| 24 | +- Pylint report in the GitHub Action Summary page, with Warning and Error annotatios, even in the code review. |
| 25 | +- Check that changes don't generate pylint warnings (if warning classes which are enabled in pylintrc) |
| 26 | +- Static analysis using mypy, pyre and pytype |
| 27 | + |
| 28 | +This enforces that any change (besides whitespace): |
| 29 | +- has code coverage and |
| 30 | +- does not introduce a pylint warning which is not disabled in `pylintrc` |
| 31 | +- does not introduce a type of static analysis warning which currently suppressed. |
| 32 | + |
| 33 | +Status Summary |
| 34 | +-------------- |
| 35 | +- The warnings shown (in the GitHub Actions Summary Page) are reminders that the |
| 36 | + PRs 22, 23, 24 are needed, before other projects should try to use it with Python3! |
| 37 | + |
| 38 | +Pylint results from GitHub CI in GitHub Actions page |
| 39 | +---------------------------------------------------- |
| 40 | +A step of the GitHub workflow produces a browser-friendly `pylint` report: |
| 41 | +From the [Actions tab](https://github.com/xenserver/python-libs/actions), |
| 42 | +open a recent workflow run the latest and scroll down until you see the tables! |
| 43 | + |
| 44 | +Testing locally and in GitHub CI using tox |
| 45 | +------------------------------------------ |
| 46 | + |
| 47 | +`pytest` runs tests, checks by `pylint` and `mypy`. With `tox`, developers can |
| 48 | +run the full test suite for Python 2.7 and 3.x. Unit tests are passing, but there are |
| 49 | + many Python3 issues which it does not uncover yet. |
| 50 | + |
| 51 | +> Intro: Managing a Project's Virtualenvs with tox - |
| 52 | +> A comprehensive beginner's introduction to tox. |
| 53 | +> https://www.seanh.cc/2018/09/01/tox-tutorial/ |
| 54 | +
|
| 55 | +To run the tests for all supported and installed python versions, run: |
| 56 | +```yaml |
| 57 | +pip3 install --user --upgrade 'py>=1.11.0' 'virtualenv<20.22' 'tox>=4.5.1'; hash -r; tox |
| 58 | +``` |
| 59 | +- `tox>=4` is needed in order to fix reading the python2.7 deps from `pyproject.toml` |
| 60 | +- The latest versions of `tox` need `'py>=1.11.0'`. Ensure that it is at least 1.11. |
| 61 | +- `virtualenv-20.22` breaks using python2.7 for the `py27` virtualenv with tox, |
| 62 | + therefore it has to be downgraded thus `'virtualenv<20.22'`. |
| 63 | + |
| 64 | +You can run tox with just the python versions you have using `tox -e py27-test -e py3.11-mypy`. |
| 65 | +The syntax is `-e py<pvthon-version>-<factor1>[-factor2]` The currently supported factors |
| 66 | +are: |
| 67 | +- `test`: runs pytest |
| 68 | +- `cov`: runs pytest --cov and generate XML and HTML reports in `.tox/py<ver>-cov/logs/` |
| 69 | +- `mypy`: runs mypy |
| 70 | +- `fox`: runs like `cov` but then opens the HTML reports in Firefox! |
| 71 | + |
| 72 | +The list of `virtualenvs` can be shown using this command: `tox -av` |
| 73 | +```yaml |
| 74 | +$ tox -av |
| 75 | +default environments: |
| 76 | +py311-pyre -> Run in a py311 virtualenv: Run pyre for static analyis, only passes using: tox -e py311-pyre |
| 77 | +py310-pytype -> Run in a py310 virtualenv: Run pytype for static analyis, intro: https://youtu.be/abvW0mOrDiY |
| 78 | +py39-check -> Run in a py39 virtualenv: Run mypy for static analyis |
| 79 | +py38-covcombine -> Run in a py38 virtualenv: Generate combined coverage reports with py27-test coverage merged |
| 80 | +py37-mdreport -> Run in a py37 virtualenv: Make a test report (which is shown in the GitHub Actions Summary Page) |
| 81 | +py36-lint -> Run in a py36 virtualenv: Run pylint and fail on warnings remaining on lines in the diff to master |
| 82 | + |
| 83 | +additional environments: |
| 84 | +cov -> Run in a /usr/bin/python3 virtualenv: Generate coverage html reports (incl. diff-cover) for this environment |
| 85 | +covcp -> Run in a /usr/bin/python3 virtualenv: Copy the generated .converage and coverage.xml to the UPLOAD_DIR dir |
| 86 | +fox -> Run in a /usr/bin/python3 virtualenv: Generate combined coverage html reports and open them in firefox |
| 87 | +test -> Run in a /usr/bin/python3 virtualenv: Run pytest in this environment with --cov for use in other stages |
| 88 | +``` |
| 89 | +If you have only one version of Python3, that works too. Use: `tox -e py<ver>-test` |
| 90 | + |
| 91 | +Installation of additional python versions for testing different versions: |
| 92 | +- Fedora 37: `sudo dnf install tox` installs all Python versions, even 3.12a7. |
| 93 | +- On Ubuntu, the deadsnakes/ppa is broken(except for 3.12), so conda or pyenv has to be used. |
| 94 | + For full instructions, see https://realpython.com/intro-to-pyenv/, E.g install on Ubuntu: |
| 95 | + ```yaml |
| 96 | + sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev |
| 97 | + libreadline-dev libsqlite3-dev xz-utils libffi-dev liblzma-dev |
| 98 | + curl https://pyenv.run | bash # and add the displayed commands to .bashrc |
| 99 | + pyenv install 3.{6,7,8,9} && pyenv local 3.{6,7,8,9} # builds and adds them |
| 100 | + ``` |
| 101 | +- For testing on newer Ubuntu hosts which have `python2-dev`, but not `pip2`, install `pip2` this way: |
| 102 | + ```yml |
| 103 | + curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py;sudo python2 get-pip.py |
| 104 | + ``` |
| 105 | + |
| 106 | +Static analysis using mypy, pyre, pyright and pytype |
| 107 | +---------------------------------------------------- |
| 108 | +The preconditions for using static analysis with `mypy` (which passes now but has |
| 109 | +only a few type comments) and `pyright` are present now and `mypy` is enabled in `tox` |
| 110 | +which runs the tests in GitHub CI as well. But of course, because they code is largely |
| 111 | +still not yet typed, no strict checks can be enabled so far. However, every checker |
| 112 | +which is possible now, is enabled. |
| 113 | + |
| 114 | +Checking the contents of untyped functions is enabled for all but four modules which |
| 115 | +would need more work. Look for `check_untyped_defs = false` in `pytproject.toml`. |
| 116 | + |
| 117 | +The goal or final benefit would be to have it to ensure internal type correctness |
| 118 | +and code quality but also to use static analysis to check the interoperability with |
| 119 | +the calling code. |
| 120 | + |
| 121 | +Type annotations: Use Type comments for now! |
| 122 | +-------------------------------------------- |
| 123 | +Python2.7 can't support the type annotation syntax, but until all users are migrated, |
| 124 | +annotations in comments (type comments) can be used. They are supported by |
| 125 | +tools like `mypy` and `pyright` (VS Code): |
| 126 | + |
| 127 | +Quoting from https://stackoverflow.com/questions/53306458/python-3-type-hints-in-python-2: |
| 128 | + |
| 129 | +> Function annotations were introduced in [PEP 3107](https://www.python.org/dev/peps/pep-3107/) for Python 3.0. The usage of annotations as type hints was formalized in in [PEP 484](https://www.python.org/dev/peps/pep-0484/) for Python 3.5+. |
| 130 | +> |
| 131 | +> Any version before 3.0 then will not support the syntax you are using for type hints at all. However, PEP 484 [offers a workaround](https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code), which some editors may choose to honor. In your case, the hints would look like this: |
| 132 | +```py |
| 133 | +def get_default_device(use_gpu=True): |
| 134 | + # type: (bool) -> cl.Device |
| 135 | + ... |
| 136 | +``` |
| 137 | +Many type checkers support this syntax: mypy, pyright/pylance, pytype |
| 138 | + |
| 139 | +As proof, these examples show how the comment below triggers the checks: |
| 140 | +```diff |
| 141 | +--- a/xcp/xmlunwrap.py |
| 142 | ++++ b/xcp/xmlunwrap.py |
| 143 | +@@ -29,1 +29,2 @@ class XmlUnwrapError(Exception): |
| 144 | + def getText(nodelist): |
| 145 | ++ # type:(Element) -> str |
| 146 | +``` |
| 147 | +mypy: |
| 148 | +```py |
| 149 | +$ mypy xcp/xmlunwrap.py |
| 150 | +xcp/xmlunwrap.py:31: error: Name "Element" is not defined |
| 151 | +xcp/xmlunwrap.py:38: error: Incompatible return value type (got "bytes", expected "str") |
| 152 | +``` |
| 153 | +pyright (used by VS Code by default): |
| 154 | +```py |
| 155 | +$ pyright xcp/xmlunwrap.py|sed "s|$PWD/||" |
| 156 | +... |
| 157 | +pyright 1.1.295 |
| 158 | +xcp/xmlunwrap.py |
| 159 | + xcp/xmlunwrap.py:32:13 - error: "Element" is not defined (reportUndefinedVariable) |
| 160 | + xcp/xmlunwrap.py:38:12 - error: Expression of type "Unknown | bytes" cannot be assigned to return type "str" |
| 161 | + Type "Unknown | bytes" cannot be assigned to type "str" |
| 162 | + "bytes" is incompatible with "str" (reportGeneralTypeIssues) |
| 163 | + xcp/xmlunwrap.py:81:38 - error: Argument of type "Unknown | None" cannot be assigned to parameter "default" of type "str" in function "getStrAttribute" |
| 164 | + Type "Unknown | None" cannot be assigned to type "str" |
| 165 | + Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues) |
| 166 | +3 errors, 0 warnings, 0 informations |
| 167 | +Completed in 0.604sec |
| 168 | +``` |
| 169 | +See https://github.com/xenserver/python-libs/pull/23 for the context of this example. |
| 170 | + |
| 171 | +Special open TODOs: |
| 172 | +------------------- |
| 173 | + |
| 174 | +Charset encoding/string handling: |
| 175 | +* With Python3, `read()` on files `open()`ed without specifying binary mode will attempt |
| 176 | + to decode the data into the Python3 Unicode string type, which will fail for all |
| 177 | + binary data. Thus all `open()` calls which might open binary files have to be converted |
| 178 | + to binary mode by default unless the caller is sure he is opening an ASCII file, |
| 179 | + even then, enabling an error handle to handle decoding errors is recommended. |
| 180 | +* With Python3, the `stdin`, `stdout` and `stderr` pipes for `Popen()` default to |
| 181 | + `bytes`(binary mode.) Binary mode is much safer because it foregoes the encode/decode. The existing users need to be able to enable text mode (when safe, it will attempt |
| 182 | + to decode and encode!) or preferably be able to use bytes (which is the type behind Python2 strings too) instead. See these PRs for details: |
| 183 | + * https://github.com/xenserver/python-libs/pull/22 |
| 184 | + * https://github.com/xenserver/python-libs/pull/23 |
| 185 | + * https://github.com/xenserver/python-libs/pull/24 |
| 186 | + * What's more: When code is called from a xapi plugin (such as ACK), when such code |
| 187 | + attempts to read text files like the `pciids` file, and there is a Unicode char |
| 188 | + it int, and the locale is not set up to be UTF-8 (because xapi plugins are started |
| 189 | + from xapi), the UTF-8 decoder has to be explicitly enabled for these files, |
| 190 | + bese by adding `encoding="utf-8"` to the arguments of these specific `open()` calls, |
| 191 | + to have valid Unicode text strings, e.g. `xcp.pci`, for regular text processing. |
| 192 | + * TODO: More to be opened for all remaining `open()` and `Popen()` users, |
| 193 | + as well as ensuring that users of `urllib` are able to work with they bytes |
| 194 | + it returns (there is no option to use text mode, data may be gzip-encoded!) |
| 195 | + |
| 196 | +Users |
| 197 | +----- |
| 198 | + |
| 199 | +* https://github.com/xenserver/host-installer |
| 200 | + * /opt/xensource/installer/ (has copies of `cpiofile.py`, `repository.py` (with `accessor.py`) |
| 201 | +* https://github.com/xcp-ng-rpms/host-upgrade-plugin ([koji](https://koji.xcp-ng.org/packageinfo?packageID=104)): |
| 202 | + * /etc/xapi.d/plugins/prepare_host_upgrade.py |
| 203 | +* https://github.com/xapi-project/xen-api (`xapi-core.rpm` and `xenopsd.rpm`) |
| 204 | + * /etc/xapi.d/extensions/pool_update.apply |
| 205 | + * /etc/xapi.d/extensions/pool_update.precheck |
| 206 | + * /etc/xapi.d/plugins/disk-space |
| 207 | + * /etc/xapi.d/plugins/install-supp-pack |
| 208 | + * /opt/xensource/libexec/host-display |
| 209 | + * /opt/xensource/libexec/mail-alarm |
| 210 | + * /opt/xensource/libexec/usb_reset.py |
| 211 | + * /opt/xensource/libexec/usb_scan.py |
| 212 | + * /usr/libexec/xenopsd/igmp_query_injector.py |
| 213 | +* xenserver-release-config/[xcp-ng-release-config](https://koji.xcp-ng.org/rpminfo?rpmID=10250) |
| 214 | + * /opt/xensource/libexec/fcoe_driver |
| 215 | + * /opt/xensource/libexec/xen-cmdline |
| 216 | +* https://github.com/xcp-ng-rpms/interface-rename: |
| 217 | + * /etc/sysconfig/network-scripts/interface-rename.py |
| 218 | + * /opt/xensource/bin/interface-rename |
| 219 | +* pvsproxy (Proprietary) |
| 220 | + * /usr/libexec/xapi-storage-script/volume/org.xen.xapi.storage.tmpfs/memoryhelper.py |
| 221 | +* https://github.com/xenserver/linux-guest-loader (not installed by default anymore) |
| 222 | + * /opt/xensource/libexec/eliloader.py |
| 223 | +* https://github.com/xcp-ng-rpms/vcputune |
| 224 | + * /opt/xensource/bin/host-cpu-tune |
| 225 | +* The ACK xenapi plugin see: https://github.com/xenserver/python-libs/pull/21 |
| 226 | + |
| 227 | +Verification: |
| 228 | +```ps |
| 229 | +# rpm -qf $(grep -r import /usr/libexec/ /usr/bin /etc/xapi.d/ /opt/xensource/|grep xcp|cut -d: -f1|grep -v Binary) --qf '%{name}\n'|sort -u|tee xcp-python-libs-importers.txt |
| 230 | +host-upgrade-plugin |
| 231 | +interface-rename |
| 232 | +pvsproxy |
| 233 | +vcputune |
| 234 | +xapi-core |
| 235 | +xenopsd |
| 236 | +xenserver-release-config |
| 237 | +# grep -s import $(rpm -ql xapi-core)|grep xcp|cut -d: -f1 |
| 238 | +/etc/xapi.d/extensions/pool_update.apply |
| 239 | +/etc/xapi.d/extensions/pool_update.precheck |
| 240 | +/etc/xapi.d/plugins/disk-space |
| 241 | +/etc/xapi.d/plugins/disk-space |
| 242 | +/etc/xapi.d/plugins/install-supp-pack |
| 243 | +/opt/xensource/libexec/host-display |
| 244 | +/opt/xensource/libexec/mail-alarm |
| 245 | +/opt/xensource/libexec/usb_reset.py |
| 246 | +/opt/xensource/libexec/usb_scan.py |
| 247 | +``` |
| 248 | + |
0 commit comments