Resolving rosdep pip keys on Jazzy (PEP 668): use a venv for rosdep-installed Python dependencies

[hakuturu583]

Summary

On Jazzy (Ubuntu 24.04), rosdep install fails for any package that declares a pip-based rosdep key (*-pip), because the system Python is marked externally managed (PEP 668). Autoware has no pip keys today, so this has not come up yet, but the first one (autowarefoundation/autoware_universe#13557, python3-typesafe-carla-pip) had to be limited to Humble with condition="$ROS_DISTRO == humble".

I'd like to agree on how Autoware should resolve pip keys on Jazzy and later. This post proposes a shared virtual environment that rosdep installs into as the first step toward per-package virtual environments with colcon-uv (#7117), and lists PIP_BREAK_SYSTEM_PACKAGES=1 as the simpler alternative. I'd like to avoid the status quo, which keeps pip-only libraries out of rosdep on Jazzy (option C below), for the reasons given there.

Background

• Ubuntu 24.04 ships /usr/lib/python3.12/EXTERNALLY-MANAGED, so sudo pip install into the system Python is refused.
• rosdep's pip installer runs sudo -H pip3 install -U <pkg>. On Python >= 3.11, rosdep refuses to run it unless one of these is true (see rosdep2/platforms/pip.py):
  • rosdep itself runs inside a virtual environment (sys.prefix != sys.base_prefix), or
  • PIP_BREAK_SYSTEM_PACKAGES=1 is set, or break-system-packages = true is set in pip.conf.
• Humble (Ubuntu 22.04, Python 3.10) is not affected.
• Upstream report: Installation instructions invalid for Ubuntu 23.04 ros-infrastructure/rosdep#922 (duplicate of Add velocity visualizer with 3D graph. #978), and the official docs page Pip installation after PEP 668.

Observed in autowarefoundation/autoware_universe#13557, rosdep-check (jazzy):

ERROR: the following rosdeps failed to install
  pip:
rosdep installation of pip packages requires installing packages globally as root.
When using Python >= 3.11, PEP 668 compliance requires you to allow pip to install alongside
externally managed packages using the 'break-system-packages' option.

Where Autoware runs rosdep

Location	Repository	Installs exec_depend?	Affected by pip keys on Jazzy
.github/workflows/rosdep-check-pr.yaml	autoware_universe	yes (all dependency types)	yes
colcon-build action used by build-and-test-*	autoware-github-actions	no (remove-exec-depend runs first)	only if a pip key is a build/test depend
docker/universe.Dockerfile (universe-devel and universe stages, --dependency-types=exec)	autoware	yes	yes: the Jazzy image build would fail
.webauto-ci/main/autoware-build/run.sh	autoware	yes	yes
Developers running rosdep install with the apt-installed rosdep (set up by the ros2_dev_tools ansible role)	-	yes	yes

None of these set PIP_BREAK_SYSTEM_PACKAGES or use a venv today. The only existing venv in the setup is acados' /opt/acados/.venv (ansible role acados), which is installed with pip directly, not through rosdep.

Verified: rosdep can install pip keys into a venv

On Ubuntu 24.04, this installs a pip key into the venv with no PEP 668 error:

python3 -m venv --system-site-packages ~/autoware_venv
source ~/autoware_venv/bin/activate
pip install rosdep            # rosdep must run from the venv's interpreter
rosdep install --from-paths src --ignore-src -y --rosdistro jazzy --as-root pip:false
# -> Successfully installed typesafe-carla-0.8.5 ... / All required rosdeps installed successfully

Notes:

• rosdep must be the venv's copy. The apt-installed /usr/bin/rosdep runs on the system Python, so its venv check fails.
• --as-root pip:false is needed. Otherwise rosdep runs sudo -H pip3, and sudo resets PATH to the system pip. apt keys are still installed with sudo.
• --system-site-packages keeps the apt-installed ROS Python packages (rclpy, cv_bridge, ...) visible.

Options

A. Shared venv for rosdep (proposal)

1. Create a shared venv with --system-site-packages (e.g. /opt/autoware/venv in Docker images, a user-level path via ansible for source builds), and install rosdep into it.
2. Run rosdep install ... --as-root pip:false from that venv in the Dockerfiles, rosdep-check-pr.yaml, colcon-build, and webauto CI.
3. Put the venv's bin first on PATH at build time and at run time (ENV PATH=... in the images, an activation step in the docs and setup script).

Pros: the system Python stays untouched, in line with PEP 668 and the Ubuntu/ROS guidance, and pip keys work on every distro.

Cons and open points:

• Point 3 matters. ament_python executables get the build interpreter's absolute path as their shebang, so colcon must run with the venv active. ament_cmake_python scripts with #!/usr/bin/env python3 follow PATH.
• Every place that builds or launches Autoware must use the venv: Docker, ansible, CI, and the docs.
• Interaction with the apt-installed Python packages: pip may install newer versions of packages that apt already provides into the venv, and these shadow the apt ones.

B. PIP_BREAK_SYSTEM_PACKAGES=1

Set it in the same places (or break-system-packages = true in /etc/pip.conf in the images and the ansible role).

Pros: a small change, with nothing to change at run time.
Cons: it installs into the system Python, which is the conflict PEP 668 exists to prevent.

C. Avoid pip keys (not recommended)

Allow only apt or ROS-buildfarm (bloom) packages, and keep pip-only libraries out of rosdep (Humble-only conditions, manual install docs). This is the current de facto state, and I'd like to move away from it:

• Many libraries Autoware needs are published only on PyPI: ML frameworks, simulator clients such as typesafe_carla, and so on. Packaging each of them as an apt or bloom package is not realistic, especially those that ship prebuilt native binaries, because the ROS buildfarm builds without network access.
• Dependencies installed by hand are invisible to rosdep: rosdep-check cannot verify them and the Docker images do not include them, so environments drift apart.
• A Humble-only condition is only a stopgap. Humble reaches end of life in May 2027, and from then on these dependencies would not be resolved anywhere.

Relation to colcon-uv (#7117) and the proposed steps

colcon-uv (#7117) goes further than option A: it gives each package its own virtual environment, so dependencies are isolated per package and can be pinned with a lockfile. In the long run, this is the ideal setup.

Moving there in one go, however, changes how every Python package is built, installed, and launched. I propose getting there in steps:

1. First step: a shared venv that rosdep can install into (option A). Keep declaring dependencies in package.xml and resolving them with rosdep, but install pip keys into a shared venv instead of the system Python. Confirm that Autoware builds, tests, and runs reliably with the venv in Docker, CI, and source builds.
2. Next step: consider moving to colcon-uv. Once running on a venv has proven stable, consider moving packages to colcon-uv and per-package venvs, starting with those that benefit most (heavy ML stacks, packages that need pinned versions).

The first step also prepares for the second: once every build and launch path already runs inside a venv, moving to per-package venvs is a smaller change.

Questions

1. I propose that Autoware allows pip-based rosdep keys (A or B) instead of staying with C. Are there objections to this direction?
2. Is a shared venv (A) acceptable, or should we start with B and move to A later?
3. Where should the venv live (/opt/autoware/venv, a workspace-local venv, ...), and who owns the changes in autoware (Docker, ansible), autoware-github-actions, and autoware_universe CI?
4. Do you agree with the steps above: a shared venv for rosdep first, then colcon-uv once the venv setup has proven stable? What should "stable" mean before we start the colcon-uv migration, for example a release cycle without venv-related issues in Docker, CI, and source builds?

Related

• feat(autoware_carla_interface): support typesafe_carla as an optional CARLA client autoware_universe#13557: the first pip key in Autoware (python3-typesafe-carla-pip), currently limited to Humble.
• Proposal: Adopt uv as the standard Python package manager for Autoware via colcon-uv #7117 "Proposal: Adopt uv as the standard Python package manager for Autoware via colcon-uv": per-package venvs managed by colcon-uv. In that thread, @xmfcx suggested shipping colcon-uv through apt "to avoid the destructive --break-system-packages", which is the same concern as option B here.
• Dependency Version Pinning for Reproducible Builds #6861 "Dependency Version Pinning for Reproducible Builds": rosdep lock files currently pin apt packages only, so pip keys would need to be considered there as well.