|
| 1 | +Common XenServer/XCP-ng Python classes |
| 2 | +====================================== |
| 3 | + |
| 4 | +The `xcp` directory contains the Common XenServer and XCP-ng Python packages. |
| 5 | +They are intented for use in XenServer and XCP-ng Dom0 only and deal with logging, |
| 6 | +Hardware/PCI, networking, and other Dom0 tasks. |
| 7 | + |
| 8 | +The pip package name is `python-libs` which is also the rpm package name in XenServer. |
| 9 | +XCP-ng packages it as [xcp-python-libs](https://github.com/xcp-ng-rpms/xcp-python-libs) |
| 10 | +([koji](https://koji.xcp-ng.org/packageinfo?packageID=400)). |
| 11 | + |
| 12 | +It supports Python 2.7 and is currently in progress to get further fixes for >= 3.6. |
| 13 | +It depends on `six`, and on Python 2.7, also `configparser` and `pyliblzma`. |
| 14 | + |
| 15 | +Pylint results from GitHub CI in GitHub Actions page |
| 16 | +---------------------------------------------------- |
| 17 | +A step of the GitHub workflow produces a browser-friendly `pylint` report: |
| 18 | +From the [Actions tab](https://github.com/xenserver/python-libs/actions), |
| 19 | +open a recent workflow run the latest and scroll down until you see the tables! |
| 20 | + |
| 21 | +Testing locally and in GitHub CI using tox |
| 22 | +------------------------------------------ |
| 23 | + |
| 24 | +`pytest` runs tests, checks by `pylint` and `mypy`. With `tox`, developers can |
| 25 | +run the full test suite for Python 2.7 and 3.x. Unit tests are passing, but there are |
| 26 | + many Python3 issues which it does not uncover yet. |
| 27 | + |
| 28 | +> Intro: Managing a Project's Virtualenvs with tox - |
| 29 | +> A comprehensive beginner's introduction to tox. |
| 30 | +> https://www.seanh.cc/2018/09/01/tox-tutorial/ |
| 31 | +
|
| 32 | +To run the tests for all supported and installed python versions, run: |
| 33 | +```yaml |
| 34 | +pip3 install --user --upgrade 'py>=1.11.0' 'virtualenv<20.22' 'tox>=4.5.1'; hash -r; tox |
| 35 | +``` |
| 36 | +- `tox>=4` is needed in order to fix reading the python2.7 deps from `pyproject.toml` |
| 37 | +- The latest versions of `tox` need `'py>=1.11.0'`. Ensure that it is at least 1.11. |
| 38 | +- `virtualenv-20.22` breaks using python2.7 for the `py27` virtualenv with tox, |
| 39 | + therefore it has to be downgraded thus `'virtualenv<20.22'`. |
| 40 | + |
| 41 | +You can run tox with just the python versions you have using `tox -e py27-test -e py3.11-mypy`. |
| 42 | +The syntax is `-e py<pvthon-version>-<factor1>[-factor2]` The currently supported factors |
| 43 | +are: |
| 44 | +- `test`: runs pytest |
| 45 | +- `cov`: runs pytest --cov and generate XML and HTML reports in `.tox/py<ver>-cov/logs/` |
| 46 | +- `mypy`: runs mypy |
| 47 | +- `fox`: runs like `cov` but then opens the HTML reports in Firefox! |
| 48 | + |
| 49 | +The list of `virtualenvs` can be shown using this command: `tox -av` |
| 50 | +```yaml |
| 51 | +tox -av |
| 52 | +default environments: |
| 53 | +py27-test -> Run in a py27 virtualenv: Run pytest in this environment with --cov for use in other stages |
| 54 | +py36-test -> Run in a py36 virtualenv: Run pytest in this environment with --cov for use in other stages |
| 55 | +py37-test -> Run in a py37 virtualenv: Run pytest in this environment with --cov for use in other stages |
| 56 | +py38-test -> Run in a py38 virtualenv: Run pytest in this environment with --cov for use in other stages |
| 57 | +py39-test -> Run in a py39 virtualenv: Run pytest in this environment with --cov for use in other stages |
| 58 | +py310-cov -> Run in a py310 virtualenv: Generate coverage html reports (incl. diff-cover) for this environment |
| 59 | +py311-mypy -> Run in a py311 virtualenv: Run mypy for static analyis (in the future, add pyre, pytype, etc) |
| 60 | +py311-covcombine -> Run in a py311 virtualenv: Generate combined coverage reports with py27-test coverage merged |
| 61 | + |
| 62 | +additional environments: |
| 63 | +covcp -> Run in a /usr/bin/python3 virtualenv: Copy the generated .converage and coverage.xml to the UPLOAD_DIR dir |
| 64 | +fox -> Run in a /usr/bin/python3 virtualenv: Generate combined coverage html reports and open them in firefox |
| 65 | +lint -> Run in a /usr/bin/python3 virtualenv: Run pylint and fail on warnings remaining on lines in the diff to master |
| 66 | +``` |
| 67 | +If you have only one version of Python3, that works too. Use: `tox -e py<ver>-test` |
| 68 | + |
| 69 | +Installation of additional python versions for testing different versions: |
| 70 | +- Fedora 37: `sudo dnf install tox` installs all Python versions, even 3.12a7. |
| 71 | +- On Ubuntu, the deadsnakes/ppa is broken(except for 3.12), so conda or pyenv has to be used. |
| 72 | + For full instructions, see https://realpython.com/intro-to-pyenv/, E.g install on Ubuntu: |
| 73 | + ```yaml |
| 74 | + sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev |
| 75 | + libreadline-dev libsqlite3-dev xz-utils libffi-dev liblzma-dev |
| 76 | + curl https://pyenv.run | bash # and add the displayed commands to .bashrc |
| 77 | + pyenv install 3.{6,7,8,9} && pyenv local 3.{6,7,8,9} # builds and adds them |
| 78 | + ``` |
| 79 | +- For testing on newer Ubuntu hosts which have `python2-dev`, but not `pip2`, install `pip2` this way: |
| 80 | + ```yml |
| 81 | + curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py;sudo python2 get-pip.py |
| 82 | + ``` |
| 83 | + |
| 84 | +Static analysis using mypy, pyre, pyright and pytype |
| 85 | +---------------------------------------------------- |
| 86 | +The preconditions for using static analysis with `mypy` (which passes now but has |
| 87 | +only a few type comments) and `pyright` are present now and `mypy` is enabled in `tox` |
| 88 | +which runs the tests in GitHub CI as well. But of course, because they code is largely |
| 89 | +still not yet typed, no strict checks can be enabled so far. However, every checker |
| 90 | +which is possible now, is enabled. |
| 91 | + |
| 92 | +Checking the contents of untyped functions is enabled for all but four modules which |
| 93 | +would need more work. Look for `check_untyped_defs = false` in `pytproject.toml`. |
| 94 | + |
| 95 | +The goal or final benefit would be to have it to ensure internal type correctness |
| 96 | +and code quality but also to use static analysis to check the interoperability with |
| 97 | +the calling code. |
| 98 | + |
| 99 | +Type annotations: Use Type comments for now! |
| 100 | +-------------------------------------------- |
| 101 | +Python2.7 can't support the type annotation syntax, but until all users are migrated, |
| 102 | +annotations in comments (type comments) can be used. They are supported by |
| 103 | +tools like `mypy` and `pyright` (VS Code): |
| 104 | + |
| 105 | +Quoting from https://stackoverflow.com/questions/53306458/python-3-type-hints-in-python-2: |
| 106 | + |
| 107 | +> Function annotations were introduced in [PEP 3107](https://www.python.org/dev/peps/pep-3107/) for Python 3.0. The usage of annotations as type hints was formalized in in [PEP 484](https://www.python.org/dev/peps/pep-0484/) for Python 3.5+. |
| 108 | +> |
| 109 | +> Any version before 3.0 then will not support the syntax you are using for type hints at all. However, PEP 484 [offers a workaround](https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code), which some editors may choose to honor. In your case, the hints would look like this: |
| 110 | +```py |
| 111 | +def get_default_device(use_gpu=True): |
| 112 | + # type: (bool) -> cl.Device |
| 113 | + ... |
| 114 | +``` |
| 115 | +Many type checkers support this syntax: mypy, pyright/pylance, pytype |
| 116 | + |
| 117 | +As proof, these examples show how the comment below triggers the checks: |
| 118 | +```diff |
| 119 | +--- a/xcp/xmlunwrap.py |
| 120 | ++++ b/xcp/xmlunwrap.py |
| 121 | +@@ -29,1 +29,2 @@ class XmlUnwrapError(Exception): |
| 122 | + def getText(nodelist): |
| 123 | ++ # type:(Element) -> str |
| 124 | +``` |
| 125 | +mypy: |
| 126 | +```py |
| 127 | +$ mypy xcp/xmlunwrap.py |
| 128 | +xcp/xmlunwrap.py:31: error: Name "Element" is not defined |
| 129 | +xcp/xmlunwrap.py:38: error: Incompatible return value type (got "bytes", expected "str") |
| 130 | +``` |
| 131 | +pyright (used by VS Code by default): |
| 132 | +```py |
| 133 | +$ pyright xcp/xmlunwrap.py|sed "s|$PWD/||" |
| 134 | +... |
| 135 | +pyright 1.1.295 |
| 136 | +xcp/xmlunwrap.py |
| 137 | + xcp/xmlunwrap.py:32:13 - error: "Element" is not defined (reportUndefinedVariable) |
| 138 | + xcp/xmlunwrap.py:38:12 - error: Expression of type "Unknown | bytes" cannot be assigned to return type "str" |
| 139 | + Type "Unknown | bytes" cannot be assigned to type "str" |
| 140 | + "bytes" is incompatible with "str" (reportGeneralTypeIssues) |
| 141 | + xcp/xmlunwrap.py:81:38 - error: Argument of type "Unknown | None" cannot be assigned to parameter "default" of type "str" in function "getStrAttribute" |
| 142 | + Type "Unknown | None" cannot be assigned to type "str" |
| 143 | + Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues) |
| 144 | +3 errors, 0 warnings, 0 informations |
| 145 | +Completed in 0.604sec |
| 146 | +``` |
| 147 | +See https://github.com/xenserver/python-libs/pull/23 for the context of this example. |
| 148 | + |
| 149 | +Special open TODOs: |
| 150 | +------------------- |
| 151 | + |
| 152 | +Charset encoding/string handling: |
| 153 | +* With Python3, `read()` on files `open()`ed without specifying binary mode will attempt |
| 154 | + to decode the data into the Python3 Unicode string type, which will fail for all |
| 155 | + binary data. Thus all `open()` calls which might open binary files have to be converted |
| 156 | + to binary mode by default unless the caller is sure he is opening an ASCII file, |
| 157 | + even then, enabling an error handle to handle decoding errors is recommended. |
| 158 | +* With Python3, the `stdin`, `stdout` and `stderr` pipes for `Popen()` default to |
| 159 | + `bytes`(binary mode.) Binary mode is much safer because it foregoes the encode/decode. The existing users need to be able to enable text mode (when safe, it will attempt |
| 160 | + to decode and encode!) or preferably be able to use bytes (which is the type behind Python2 strings too) instead. See these PRs for details: |
| 161 | + * https://github.com/xenserver/python-libs/pull/22 |
| 162 | + * https://github.com/xenserver/python-libs/pull/23 |
| 163 | + * https://github.com/xenserver/python-libs/pull/24 |
| 164 | + * What's more: When code is called from a xapi plugin (such as ACK), when such code |
| 165 | + attempts to read text files like the `pciids` file, and there is a Unicode char |
| 166 | + it int, and the locale is not set up to be UTF-8 (because xapi plugins are started |
| 167 | + from xapi), the UTF-8 decoder has to be explicitly enabled for these files, |
| 168 | + bese by adding `encoding="utf-8"` to the arguments of these specific `open()` calls, |
| 169 | + to have valid Unicode text strings, e.g. `xcp.pci`, for regular text processing. |
| 170 | + * TODO: More to be opened for all remaining `open()` and `Popen()` users, |
| 171 | + as well as ensuring that users of `urllib` are able to work with they bytes |
| 172 | + it returns (there is no option to use text mode, data may be gzip-encoded!) |
| 173 | + |
| 174 | +Users |
| 175 | +----- |
| 176 | + |
| 177 | +* https://github.com/xenserver/host-installer |
| 178 | + * /opt/xensource/installer/ (has copies of `cpiofile.py`, `repository.py` (with `accessor.py`) |
| 179 | +* https://github.com/xcp-ng-rpms/host-upgrade-plugin ([koji](https://koji.xcp-ng.org/packageinfo?packageID=104)): |
| 180 | + * /etc/xapi.d/plugins/prepare_host_upgrade.py |
| 181 | +* https://github.com/xapi-project/xen-api (`xapi-core.rpm` and `xenopsd.rpm`) |
| 182 | + * /etc/xapi.d/extensions/pool_update.apply |
| 183 | + * /etc/xapi.d/extensions/pool_update.precheck |
| 184 | + * /etc/xapi.d/plugins/disk-space |
| 185 | + * /etc/xapi.d/plugins/install-supp-pack |
| 186 | + * /opt/xensource/libexec/host-display |
| 187 | + * /opt/xensource/libexec/mail-alarm |
| 188 | + * /opt/xensource/libexec/usb_reset.py |
| 189 | + * /opt/xensource/libexec/usb_scan.py |
| 190 | + * /usr/libexec/xenopsd/igmp_query_injector.py |
| 191 | +* xenserver-release-config/[xcp-ng-release-config](https://koji.xcp-ng.org/rpminfo?rpmID=10250) |
| 192 | + * /opt/xensource/libexec/fcoe_driver |
| 193 | + * /opt/xensource/libexec/xen-cmdline |
| 194 | +* https://github.com/xcp-ng-rpms/interface-rename: |
| 195 | + * /etc/sysconfig/network-scripts/interface-rename.py |
| 196 | + * /opt/xensource/bin/interface-rename |
| 197 | +* pvsproxy (Proprietary) |
| 198 | + * /usr/libexec/xapi-storage-script/volume/org.xen.xapi.storage.tmpfs/memoryhelper.py |
| 199 | +* https://github.com/xenserver/linux-guest-loader (not installed by default anymore) |
| 200 | + * /opt/xensource/libexec/eliloader.py |
| 201 | +* https://github.com/xcp-ng-rpms/vcputune |
| 202 | + * /opt/xensource/bin/host-cpu-tune |
| 203 | +* The ACK xenapi plugin see: https://github.com/xenserver/python-libs/pull/21 |
| 204 | + |
| 205 | +Verification: |
| 206 | +```ps |
| 207 | +# rpm -qf $(grep -r import /usr/libexec/ /usr/bin /etc/xapi.d/ /opt/xensource/|grep xcp|cut -d: -f1|grep -v Binary) --qf '%{name}\n'|sort -u|tee xcp-python-libs-importers.txt |
| 208 | +host-upgrade-plugin |
| 209 | +interface-rename |
| 210 | +pvsproxy |
| 211 | +vcputune |
| 212 | +xapi-core |
| 213 | +xenopsd |
| 214 | +xenserver-release-config |
| 215 | +# grep -s import $(rpm -ql xapi-core)|grep xcp|cut -d: -f1 |
| 216 | +/etc/xapi.d/extensions/pool_update.apply |
| 217 | +/etc/xapi.d/extensions/pool_update.precheck |
| 218 | +/etc/xapi.d/plugins/disk-space |
| 219 | +/etc/xapi.d/plugins/disk-space |
| 220 | +/etc/xapi.d/plugins/install-supp-pack |
| 221 | +/opt/xensource/libexec/host-display |
| 222 | +/opt/xensource/libexec/mail-alarm |
| 223 | +/opt/xensource/libexec/usb_reset.py |
| 224 | +/opt/xensource/libexec/usb_scan.py |
| 225 | +``` |
| 226 | + |
0 commit comments