Cryptography 36.0.2 (and rust support)

[emanuele-f]

Here are some notes I took when building the cryptography 36.0.2 python module in Chaquopy, which was needed to run mitmproxy 8.0.0. I hope this will be useful to integrate it in the officially supported packages.

Overview

mitmproxy 8.0.0 depends on the new cryptography 36.x module, for which rust is now mandatory.
cryptography uses setuptools_rust to build native shared objects and pyo3 to invoke them from python.

Here the relevant part of the dependency tree:

mitmproxy
  cryptography
    setuptools_rust
      rust
    pyo3
      python

Other than rust setup for cross-compilation, the problematic part is that cryptography uses pyo3 v0.15.1, which requires a python interpreter to be cross-compiled for the target architecture (and it looks for _sysconfigdata* in the python lib folder). In theory, both pyo3 and cryptography support building abi3 modules, which work regardless of the specific python version and could
be built without an existing python interpreter. However, this possibility is only added in pyo3 0.16.4, which is not currently supported by cryptography. See PyO3/pyo3#2310 for more details.

So, in order to build the cryptography module, we currently need:

• Install rust and configure it for the cross compilation
• Cross compile python

With some tricks, it's possible to compile only the core of python without the need to also cross-compile its dependencies. In fact, the pyo3 shared modules of cryptography will be linked against the soname used by python library shipped with chaquopy.

Preparing the build environment

The following instructions are meant for building on an archlinux host without docker. Install requirements (most requirements not listed here, see target/Dockerfile)

pacman -S patchelf

Copy the Android toolchains used by chaquopy:

docker build -t chaquopy-base -f base.dockerfile .
docker build -t chaquopy-target target

# replace 62b7d2f98872 with the chaquopy-target container ID
cd target
docker cp 62b7d2f98872:/root/target/toolchains toolchains

To avoid polluting the build machine, use an overlayfs to install all the stuff:

cd /home/emanuele/src/build-wheel
mkdir build
cd build
rm -rf sysroot
mkdir -p sysroot overlay workdir

# NOTE: to refresh the lowerdir when mounted, run "sudo mount -o remount overlay"
sudo mount -t overlay overlay -o lowerdir=/,upperdir=./sysroot,workdir=./workdir ./overlay
sudo arch-chroot ./overlay

Build and install the exact python version required by Chaquopy:

version=3.8.7

cd /home/emanuele/src/build-wheel/build
wget https://www.python.org/ftp/python/$version/Python-$version.tgz

tar -xf Python-$version.tgz
cd Python-$version
./configure --prefix=/usr
make -j $(nproc)
make install

# verify
python3.8 --version

Install rust and libraries required for cross-compilation in rust

pacman -Rs rustup rust
curl https://sh.rustup.rs -sSf | sh -s -- -y
source "$HOME/.cargo/env"

# required for cross-compilation
rustup target add aarch64-linux-android
rustup target add arm-linux-androideabi
rustup target add i686-linux-android
rustup target add x86_64-linux-android

# Verify installed cross-compilation libraries
rustc --print target-list | grep android

# install requirements, they will be needed to build cryptography
cd /home/emanuele/src/build-wheel/server/pypi
pip3.8 install -r requirements.txt

Building for an ABI

These steps must be performed for each Android ABI.

First select the target ABI:

# only pick the target ABI of choice
export ARCH="armeabi-v7a" CPU="arm"     TOOL_PREFIX="arm-linux-androideabi" TOOL_CFLAGS="-march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb" TOOL_LDFLAGS="-march=armv7-a -Wl,--fix-cortex-a8"
export ARCH="arm64-v8a"   CPU="aarch64" TOOL_PREFIX="aarch64-linux-android" TOOL_CFLAGS="" TOOL_LDFLAGS=""
export ARCH="x86"         CPU="x86"     TOOL_PREFIX="i686-linux-android"    TOOL_CFLAGS="" TOOL_LDFLAGS=""
export ARCH="x86_64"      CPU="x86_64"  TOOL_PREFIX="x86_64-linux-android"  TOOL_CFLAGS="" TOOL_LDFLAGS=""

Setup the environment for cross compilation:

version=3.8.7
cd /home/emanuele/src/build-wheel/build

# cleanup and setup
rm -rf $ARCH/sysroot/usr $ARCH/Python-$version
mkdir -p $ARCH/sysroot/usr/lib $ARCH/sysroot/usr/include
tar -C $ARCH -xf Python-$version.tgz

TOOLCHAIN=`readlink -f ../target/toolchains`
SYSROOT=`readlink -f $ARCH/sysroot`

# copy headers from the toolchain to the sysroot
cp -r $TOOLCHAIN/$ARCH/sysroot/usr/* $SYSROOT/usr

# (OPTIONAL) extend sysroot with python deps, not actually needed by pyo3
#cp -r deps/OpenSSL-for-Android-Prebuilt/openssl-1.1.1k-clang/include/* $ARCH/sysroot/usr/include
#cp -r deps/libcrypt_0.2-5_aarch64/usr/* $ARCH/sysroot/usr
#...

# Needed to cross compile python (some may not be needed)
export CC="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-gcc"
export CXX="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-g++"
export AR="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-ar"
export AS="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-as"
export LD="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-ld"
export LDSHARED="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-gcc -shared"
export RANLIB="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-ranlib"
export STRIP="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-strip --strip-unneeded"
export NM="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-nm"
export READELF="$TOOLCHAIN/$ARCH/$TOOL_PREFIX/bin/readelf"
export CROSS_COMPILE_TARGET=yes
export CFLAGS="-fPIC -DANDROID --sysroot=$SYSROOT $TOOL_CFLAGS"
export CXXFLAGS="$CFLAGS"

# Flags taken from build-wheel.py
export LDFLAGS="--sysroot=$SYSROOT -Wl,--no-undefined -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libunwind.a $TOOL_LDFLAGS"

Cross compile python

cd $ARCH/Python-$version
./configure --build=x86_64-unknown-linux-gnu --host=$CPU-linux-android --enable-shared --prefix=`readlink -f ../sysroot/usr` \
    ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no ac_cv_have_long_long_format=yes ac_cv_buggy_getaddrinfo=no

# patch to support Android API 19
# chaquopy targets API 19 on armv7a/x86, which is not fully supported by python https://bugs.python.org/issue36162
# the API level is specified via the "--target=" arg inside the $TOOL_PREFIX-gcc script
sed -i 's/^#define HAVE_SENDFILE 1$/\/* #undef HAVE_SENDFILE *\//g' pyconfig.h
sed -i 's/^#define HAVE_TRUNCATE 1$/\/* #undef HAVE_TRUNCATE *\//g' pyconfig.h
sed -i 's/^#define HAVE_WCSFTIME 1$/\/* #undef HAVE_WCSFTIME *\//g' pyconfig.h

# build (ignore errors, they occur because of missing dependencies, but we are only interested in libpython and related _sysconfigdata)
make -j $(nproc)
make install

# Patch SONAME to correspond to the chaquopy "libpython3.8.so"
# check with x86_64-linux-android-readelf -Wa $SYSROOT/usr/lib/libpython3.8.so | grep SONAME
mv $SYSROOT/usr/lib/libpython3.8.so{.1.0,}
patchelf --set-soname libpython3.8.so $SYSROOT/usr/lib/libpython3.8.so

Finally, build cryptography (assume rust_cross_compile.patch is applied (see below), which sets PYO3_CROSS_LIB_DIR):

cd /home/emanuele/src/build-wheel/server/pypi

python3.8 ./build-wheel.py --toolchain ../../target/toolchains/$ARCH cryptography

If build successful, you can retrieve the built wheel from outside the overlayfs in the build-wheel/sysroot folder.

build-wheel patches

rust_cross_compile.patch:

--- src-original/setup.py
+++ src/setup.py
@@ -10,6 +10,16 @@ import sys
 
 from setuptools import setup

+# https://doc.rust-lang.org/rustc/codegen-options/index.html
+os.environ["RUSTFLAGS"] = f"-C linker={os.environ['CC']}"
+os.environ["CARGO_BUILD_TARGET"] = os.environ['CHAQUOPY_TRIPLET']
+
+# https://pyo3.rs/v0.15.2/building_and_distribution.html
+os.environ["PYO3_PYTHON"] = f"python{os.environ['CHAQUOPY_PYTHON']}"
+os.environ["PYO3_CROSS_PYTHON_VERSION"] = os.environ['CHAQUOPY_PYTHON']
+os.environ["PYO3_CROSS_LIB_DIR"] = f"{os.environ['RECIPE_DIR']}/../../../../build/{os.environ['CHAQUOPY_ABI']}/sysroot/usr/lib"
+#print(os.environ)
+
 try:
     from setuptools_rust import RustExtension
 except ImportError:

cryptography meta.yaml:

package:
  name: cryptography
  version: 36.0.2

requirements:
  build:
    - cffi 1.13.2
    - setuptools-rust 1.2.0
  host:
    - openssl

Add rust to the docker build script

# Install rust and rustup
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y

# Install crates needed for cross compilation (note you need to re-run all the build commands)
# https://rust-lang.github.io/rustup/cross-compilation.html
Run source $HOME/.cargo/env && \
    rustup target add x86_64-linux-android && \
    rustup target add i686-linux-android && \
    rustup target add aarch64-linux-android && \
    rustup target add armv7-linux-androideabi

[mhsmith]

Thanks very much. I'm currently working on updating Chaquopy to Python 3.10 (#624), so I'll have a look at updating Cryptography after that. The cleanest solution would probably be to take the existing Rust support from the tokenizers package and merge that into the main build-wheel tool, along with any necessary additions from this issue.

If anyone else needs a new version of this package, please click the thumbs up button above, and post a comment explaining why you need it.

[newfix]

First of all, I would like to thank you for creating and sharing Chaquopy. I am truly impressed how easy it is to run Python code on Android using this SDK.
I am currently using Chaquopy (via BeeWare / Briefcase) to develop an Android app in Python that needs to encrypt and decrypt data using cryptographic primitives which are only available in recent versions of the Cryptography package. Therefore, I am interested in getting new versions of the Cryptography package (preferably the latest version) to work with Chaquopy. So far, all my attempts to install version 39 have been without success.
Please let me know if there is an (easy) way to make it work or if a new version of Cryptography for Chaquopy is to be expected soon.

[mhsmith]

Would version 36.0.2 be new enough for you? If so, you can try using the wheels built by @emanuele-f at https://github.com/emanuele-f/chaquopy-wheels.

If you need a newer version than that, please let me know exactly which features you need it for.

[newfix]

Unfortunately, version 36.0.2 is not suitable for me, given that I have to use AESSIV which was introduced with version 37 (see https://cryptography.io/en/latest/hazmat/primitives/aead/#cryptography.hazmat.primitives.ciphers.aead.AESSIV). The performance improvement of ChaCha20Poly1305 introduced with version 39 would be a bonus, but is not strictly necessary.

[emanuele-f]

I will probably look into building cryptography 38 soon, but I provide no guarantee. In the long term, it would be great to have it as an official supported package.

[newfix]

Having Cryptography version 38 work with Chaquopy would be great.

Just out of curiosity: Is there any particular reason for choosing version 38 rather than the latest version?

[emanuele-f]

I'm trying to run the latest mitmproxy release, which strictly dependends on version 38 https://github.com/mitmproxy/mitmproxy/blob/9.0.1/setup.py#L79

[emanuele-f]

Hi @mhsmith, following the new instructions at https://github.com/chaquo/chaquopy/tree/master/server/pypi to build for python 3.10, I got the same error as before (e.g. see below for PyYAML). I think the problem is that python 3.8 must be used for the conda env (conda create -n build-wheel python=3.8) not python 3.10.

Building wheels for collected packages: PyYAML
  Building wheel for PyYAML (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [565 lines of output]
      In file included from ext/_yaml.c:271:
      ext/_yaml.h:10: warning: "PyString_CheckExact" redefined
         10 | #define PyString_CheckExact PyBytes_CheckExact
            |
      ext/_yaml.c:139: note: this is the location of the previous definition
        139 |   #define PyString_CheckExact          PyUnicode_CheckExact

After switching to python 3.8 in conda, pip install -r server/pypi/requirements.txt still fails. The problem seem to be in the PyYAML version and pip version. I've changed the requirements as follows:

-pip==19.2.3
-setuptools==46.4.0
-wheel==0.33.6
+#pip==19.2.3
+#setuptools==46.4.0
+#wheel==0.33.6
 
 # Other direct requirements
 Jinja2==2.10
 jsonschema==2.6.0
 pyelftools==0.24
-PyYAML==3.12
+PyYAML==3.13

and now installing them works good. I've then tried to build the brotli package just to see if things are set up properly, but it fails:

./build-wheel.py --python 3.10 --abi x86_64 brotli

build-wheel.py: mkdir -p /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel
build-wheel.py: unzip -d /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel -q /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/src/Brotli-1.0.7-cp310-cp310-linux_x86_64.whl
build-wheel.py: Processing native binaries
build-wheel.py: mv /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.cpython-310-x86_64-linux-gnu.so /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: chmod +w /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: /home/emanuele/Android/Sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip --strip-unneeded /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: Error: /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so is linked against unknown library 'libm.so.6'

Kindly request your help to proceed

[mhsmith]

Please post the full logs for both of those failures: you can attach them as files if they're too large to paste.

And please clarify the following:

• The first failure, with PyYAML: did that happen after the following sequence of commands? Or something different?

  conda create -n build-wheel python=3.10
  conda activate build-wheel
  pip install -r server/pypi/requirements.txt
  

• The second failure, with brotli: did you create that environment with Python 3.8 or 3.10?
  • If 3.8, it's apparently still managed to find a python3.10 executable somewhere to do the build, so where did that come from (which python3.10)?
  • And does the brotli build work with Python 3.8?

[emanuele-f]

I will redo on a clean environment and post the full logs. Do you confirm that to build for python 3.10 I have to use conda create -n build-wheel python=3.10 as of the docs?

[mhsmith]

Any python3.10 executable on the PATH should be fine: conda is just a convenient way of getting it.