Skip to content

fix: workaround podman stop/rm timeout in rootless mode with --pid host#1982

Open
xz-dev wants to merge 6 commits into89luca89:mainfrom
xz-dev:fix/podman-rootless-stop-timeout
Open

fix: workaround podman stop/rm timeout in rootless mode with --pid host#1982
xz-dev wants to merge 6 commits into89luca89:mainfrom
xz-dev:fix/podman-rootless-stop-timeout

Conversation

@xz-dev
Copy link
Contributor

@xz-dev xz-dev commented Jan 24, 2026

Summary

podman stop / podman rm --force times out when crun's cgroup-path is empty. This commonly happens on non-systemd systems (OpenRC/elogind) where cgroup delegation is not automatically configured. The issue can affect both rootless and rootful modes when cgroup is unavailable.

Background: cgroup delegation

In cgroup v2, only root can create sub-cgroups by default. "Delegation" means granting a non-root user write access to a cgroup subtree, allowing them to create child cgroups.

  • systemd systems: systemd-logind automatically delegates user slices (e.g., /sys/fs/cgroup/user.slice/user-1000.slice/) to users.
  • non-systemd systems (OpenRC/elogind): Delegation is NOT automatic - users lack write permission to /sys/fs/cgroup, so podman cannot create cgroups for containers.
  • cgroup disabled: If cgroup support is disabled entirely, even root cannot create cgroups.

To fix this properly, configure cgroup delegation:
See: #1939

Why --pid host is problematic but --unshare-process is not

This is due to Linux kernel's PID namespace behavior:

With PID namespace isolation (default / --unshare-process)

When a container has its own PID namespace, the Linux kernel provides automatic process cleanup. According to pid_namespaces(7):

If the "init" process of a PID namespace terminates, the kernel terminates all of the processes in the namespace via a SIGKILL signal.

This is a kernel-level guarantee that doesn't depend on cgroup:

  • Container's entry process is PID 1 in its namespace
  • When PID 1 exits (or is killed), kernel automatically SIGKILLs all processes in that namespace
  • This cleanup is mandatory and cannot be bypassed

With --pid host

Container processes share the host's PID namespace:

  • Container's entry process is NOT PID 1 (e.g., PID 12345 on host)
  • No automatic kernel cleanup when the main process exits
  • Child processes become orphans, adopted by host's init (systemd)
  • Must rely on cgroup to track and kill all container processes

Process cleanup mechanisms comparison

Mode Cleanup Mechanism Depends on cgroup Reliable
PID namespace (default) Kernel zap_pid_ns_processes() No ✅ Always
--pid host + cgroup available cgroup.kill / cgroup.procs Yes ✅ Yes
--pid host + no cgroup Signal to main process only Yes (but broken) ❌ No

This PR provides a workaround for the last case (no reliable cleanup method exists).

Root Cause

Without available cgroup, crun cannot create a cgroup for the container. When cgroup-path is empty (or null), crun kill --all cannot enumerate processes via cgroup, causing podman stop to timeout.

# Verification: cgroup-path is empty (no cgroup available)
$ podman create --name test --pid host alpine sleep infinity
$ cat /run/user/$(id -u)/crun/$(podman inspect test --format '{{.Id}}')/status | grep cgroup-path
    "cgroup-path": "",

# With proper cgroup delegation or on systemd systems, cgroup-path is NOT empty
    "cgroup-path": "/user.slice/user-1000.slice/user@1000.service/user.slice/libpod-xxx.scope/container",

Solution

Check if crun's cgroup-path is actually empty. If so, call podman kill before stop/rm --force, which sends signals directly to the container's init process PID, bypassing the cgroup lookup issue.

The workaround:

  • Applies to both rootless (/run/user/UID/crun/) and rootful (/run/crun/) modes
  • Is applied proactively (before stop/rm) rather than as a fallback, to avoid masking other potential failures
  • Handles cgroup-path being empty string or null in JSON

Limitation

This workaround only kills the init process, not orphaned child processes:

This is a known limitation without proper cgroup (see containers/podman#11888). To ensure all processes are cleaned up, use --unshare-process when creating the container.

A warning is displayed when the workaround is applied, with a link to the fix documentation.

Changes

  • distrobox-stop: Check crun cgroup-path; if empty, call podman kill first and show warning
  • distrobox-rm: Check crun cgroup-path; if empty, call podman kill first and show warning
  • Support both rootless and rootful crun status paths

Test Results

Environment cgroup-path Workaround Result
Gentoo rootless (no delegation) empty ✅ applied 0.25s ✅
Gentoo rootless (with delegation) valid ❌ not needed 0.21s ✅
Gentoo rootful valid ❌ not needed 0.29s ✅
Fedora (systemd) valid ❌ not needed 0.23s ✅

Closes #1939
See also: chimera-linux/cports#1718

@xz-dev xz-dev force-pushed the fix/podman-rootless-stop-timeout branch from 231511f to 832522a Compare January 24, 2026 03:12
In rootless mode with --pid host (distrobox default), podman stop/rm --force
times out because "crun kill --all" fails when the container's cgroup-path
is empty.

Root cause: When using --pid host, the container shares the host's PID
namespace and crun doesn't create a dedicated cgroup. The "crun kill --all"
command relies on cgroup to find processes, but with empty cgroup-path,
no processes are found and killed.

Solution: Call "podman kill" before stop/rm, which uses "crun kill" (without
--all flag) that sends signals directly to the container's init process PID,
bypassing the cgroup lookup issue.

Closes: 89luca89#1939
See also: chimera-linux/cports#1718

Signed-off-by: xz-dev <xiangzhedev@gmail.com>
Add documentation and warning about a limitation of --pid host mode:
orphaned child processes may remain after container stop/rm.

This is a known podman limitation (containers/podman#11888), not something
introduced by our workaround. The workaround only kills the init process;
child processes that daemonized or backgrounded will persist.

Changes:
- Add warning in distrobox-create for rootless podman without --unshare-process
- Document the limitation in distrobox-stop and distrobox-rm comments
- Recommend --unshare-process for full process cleanup

Signed-off-by: xz-dev <xiangzhedev@gmail.com>
@xz-dev xz-dev force-pushed the fix/podman-rootless-stop-timeout branch from 832522a to 17a0ba9 Compare January 24, 2026 03:22
Document that processes started via "podman exec" (e.g., distrobox-enter)
run in separate process groups and will become orphaned when the container
stops. This is in addition to the existing daemonized process limitation.

The workaround only kills the container's init process, not exec'd processes.

Also only apply the workaround when PidMode is "host", since this is
the specific condition that causes the cgroup-path to be empty.

Signed-off-by: xz-dev <xiangzhedev@gmail.com>
@xz-dev xz-dev force-pushed the fix/podman-rootless-stop-timeout branch 2 times, most recently from 0751fdd to 900531f Compare January 24, 2026 10:19
…etection

Instead of checking PidMode=host, now check if crun's cgroup-path is
actually empty. This is more accurate because:
- With systemd cgroup manager (e.g., Fedora), cgroup delegation works
  even with --pid host, so the workaround is not needed
- With cgroupfs manager, cgroup-path may be empty regardless of PidMode

Changes:
- Check crun status file for empty cgroup-path instead of PidMode
- Move warning from distrobox-create to distrobox-stop/rm (more accurate)
- Show warning only when workaround is actually applied
- Simplify sed command (single sed instead of grep|sed)

The workaround is applied proactively (before stop/rm) rather than as a
fallback, to avoid masking other potential failures.

Signed-off-by: xz-dev <xiangzhedev@gmail.com>
@xz-dev xz-dev force-pushed the fix/podman-rootless-stop-timeout branch 3 times, most recently from d66dd6a to c7f9472 Compare January 24, 2026 11:10
…nation

- Add background explanation of cgroup delegation in code comments
- Explain why non-systemd systems (OpenRC/elogind) need manual configuration
- Update user warnings to be more helpful with actionable fix reference
- Add link to issue 89luca89#1939 for troubleshooting

Signed-off-by: xz-dev <xiangzhedev@gmail.com>
@xz-dev xz-dev force-pushed the fix/podman-rootless-stop-timeout branch from c7f9472 to 4091335 Compare January 24, 2026 11:11
The core issue is whether cgroup is available, not whether running as
rootless. This change:

- Remove rootless-only restriction from the workaround
- Support both rootless (/run/user/UID/crun/) and rootful (/run/crun/)
  crun status paths
- Update comments to clarify that cgroup unavailability can occur in
  both rootless (without delegation) and rootful (cgroup disabled) modes

Signed-off-by: xz-dev <xiangzhedev@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Suggestion] podman kill on closing distrobox ephemeral

1 participant