Skip to content

packaging_compiled.md

Latest commit

 

History

History
465 lines (364 loc) · 12.6 KB

packaging_compiled.md

File metadata and controls

465 lines (364 loc) · 12.6 KB
layout page
title Compiled packaging
permalink /guides/packaging-compiled/
nav_order 6
parent Topical Guides

{% include toc.html %}

Packaging Compiled Projects

There are a variety of ways to package compiled projects. In the past, the only way to do it was to use setuptools/distutils, which required using lots of fragile internals - distutils was intended primarily to compile CPython, and setuptools tried to stay away from changing the compile portions except as required. Now, however, we have several very nice options for compiled packages!

The most exciting developments have been new native build backends:

  • scikit-build-core: Builds C/C++/Fortran using CMake.
  • meson-python: Builds C/C++/Fortran using Meson.
  • maturin: Builds Rust using Cargo. Written entirely in Rust!
  • enscons: Builds C/C++ using SCONs. (Aging now, but this was the first native backend!)

{: .note }

You should be familiar with [packing a pure Python project]({% link pages/guides/packaging_compiled.md %}) - the metadata configuration is the same.

There are also classic setuptools plugins:

{: .important }

Selecting a backend: If you are using Rust, use maturin. If you are using CUDA, use scikit-build-core. If you are using a classic language (C, C++, Fortran), then you can use either scikit-build-core or meson-python, depending on whether you prefer writing CMake or Meson. Meson is a lot more opinionated; it requires you use version control, it requires a README.md and LICENSE file. It requires your compiler be properly set up. Etc. While CMake can be as elegant as Meson, there are a lot of historical examples of poorly written CMake.

pyproject.toml: build-system

{% rr PY001 %} Packages must have a pyproject.toml file {% rr PP001 %} that selects the backend:

{% tabs %} {% tab skbc Scikit-build-core %}

[build-system]
requires = ["pybind11", "scikit-build-core>=0.11"]
build-backend = "scikit_build_core.build"

{% endtab %} {% tab meson Meson-python %}

[build-system]
requires = ["meson-python>=0.18", "pybind11"]
build-backend = "mesonpy"

{% endtab %} {% tab maturin Maturin %}

[build-system]
requires = ["maturin>=1.9,<2"]
build-backend = "maturin"

{% endtab %} {% endtabs %}

{% include pyproject.md %}

Tool section in pyproject.toml

These tools all read the project table. They also have extra configuration options in tool.* settings.

{% tabs %} {% tab skbc Scikit-build-core %}

[tool.scikit-build]
minimum-version = "build-system.requires"
build-dir = "build/{wheel_tag}"

These options are not required, but can improve your experience.

{% endtab %} {% tab meson Meson-python %}

No tool.meson-python configuration required for this example.

{% endtab %} {% tab maturin Maturin %}

[tool.maturin]
module-name = "package._core"
python-source = "src"
sdist-generator = "git"  # default is cargo

Maturin assumes you follow Rust's package structure, so we need a little bit of configuration here to follow the convention of the other tools here.

{% endtab %} {% endtabs %}

Backend specific files

{% tabs %} {% tab skbc Scikit-build-core %}

Example CMakeLists.txt file (using pybind11, so include pybind11 in build-system.requires too):

cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)

set(PYBIND11_FINDPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(_core MODULE src/main.cpp)
install(TARGETS _core DESTINATION ${SKBUILD_PROJECT_NAME})

Scikit-build-core will use your .gitignore to help it avoid adding ignored files to your distributions; it also has a default ignore for common cache files, so you can get started without one, but it's recommended.

{% endtab %} {% tab meson Meson-python %}

Example meson.build file (using pybind11, so include pybind11 in build-system.requires too):

project(
    'package',
    'cpp',
    version: '0.1.0',
    license: 'BSD',
    meson_version: '>= 1.1.0',
    default_options: [
        'cpp_std=c++11',
    ],
)

py = import('python').find_installation(pure: false)
pybind11_dep = dependency('pybind11')

py.extension_module('_core',
    'src/main.cpp',
    subdir: 'package',
    install: true,
    dependencies : [pybind11_dep],
)

install_subdir('src/package', install_dir: py.get_install_dir() / 'package', strip_directory: true)

Meson requires that your source be tracked by version control. In a real project, you will likely be doing this, but when trying out a build backend you might not think to set up a git repo to build it.

{% endtab %} {% tab maturin Maturin %}

Example Cargo.toml file:

[package]
name = "package"
version = "0.1.0"
edition = "2021"

[lib]
name = "_core"
# "cdylib" is necessary to produce a shared library for Python to import from.
crate-type = ["cdylib"]

[dependencies]
rand = "0.9.2"

[dependencies.pyo3]
version = "0.27.2"
# "extension-module" tells pyo3 we want to build an extension module (skips linking against libpython.so)
# "abi3-py310" tells pyo3 (and maturin) to build using the stable ABI with minimum Python version 3.10
features = ["extension-module", "abi3-py310"]

{% endtab %} {% endtabs %}

Example compiled file

This example will make a _core extension inside your package; this pattern allows you to easily provide both Python files and compiled extensions, and keeping the details of your compiled extension private. You can select whatever name you wish, though, or even make your compiled extension a top level module.

{% tabs %} {% tab skbc Scikit-build-core %}

Example src/main.cpp file:

#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
  m.doc() = R"pbdoc(
      Pybind11 example plugin
      -----------------------
      .. currentmodule:: python_example
      .. autosummary::
         :toctree: _generate
         add
         subtract
  )pbdoc";

  m.def("add", &add, R"pbdoc(
      Add two numbers
      Some other explanation about the add function.
  )pbdoc");

  m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
      Subtract two numbers
      Some other explanation about the subtract function.
  )pbdoc");
}

{% endtab %} {% tab meson Meson-python %}

Example src/main.cpp file:

#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
  m.doc() = R"pbdoc(
      Pybind11 example plugin
      -----------------------
      .. currentmodule:: python_example
      .. autosummary::
         :toctree: _generate
         add
         subtract
  )pbdoc";

  m.def("add", &add, R"pbdoc(
      Add two numbers
      Some other explanation about the add function.
  )pbdoc");

  m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
      Subtract two numbers
      Some other explanation about the subtract function.
  )pbdoc");
}

{% endtab %} {% tab maturin Maturin %}

Example src/lib.rs file:

use pyo3::prelude::*;

/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
mod _core {
    use super::*;

    #[pyfunction]
    fn add(x: i64, y: i64) -> i64 {
        x + y
    }

    #[pyfunction]
    fn subtract(x: i64, y: i64) -> i64 {
        x - y
    }


    #[pymodule_init]
    fn pymodule_init(m: &Bound<'_, PyModule>) -> PyResult<()> {
        m.add("__version__", env!("CARGO_PKG_VERSION"))?;
        Ok(())
    }
}

{% endtab %} {% endtabs %}

Package structure

The recommendation (followed above) is to have source code in /src, and the Python package files in /src/<package>. The compiled files also can go in /src.

Versioning

Check the documentation for the tools above to see what forms of dynamic versioning the tool supports.

Including/excluding files in the SDist

Each tool uses a different mechanism to include or remove files from the SDist, though the defaults are reasonable.

Distributing

Unlike pure Python, you'll need to build redistributable wheels for each platform and supported Python version if you want to avoid compilation on the user's system using cibuildwheel. See [the CI page on wheels][gha_wheels] for a suggested workflow.

Special considerations

NumPy

Modern versions of NumPy (1.25+) allow you to target older versions when building, which is highly recommended, and this became required in NumPy 2.0. Now you add:

#define NPY_TARGET_VERSION NPY_1_22_API_VERSION

(Where that number is whatever version you support as a minimum) then make sure you build with NumPy 1.25+ (or 2.0+). Before 1.25, it was necessary to actually pin the oldest NumPy you supported (the oldest-supported-numpy package is the easiest method). If you support Python < 3.9, you'll have to use the old method for those versions.

If using pybind11, you don't need NumPy at build-time in the first place.

{: .important }

Python 3.13.4 is broken on Windows for compiling code - it always reports that it is free-threaded. 3.13.5 was rushed out to fix it.

[gha_wheels]: {% link pages/guides/gha_wheels.md %}

<script src="{% link assets/js/tabs.js %}"></script>