Run a language SDK runtime in its own process group by jason810496 · Pull Request #75707 · apache/airflow

jason810496 · 2026-10-27T04:09:45Z

closes: Child processes of language SDK tasks survive when the task is stopped #75533

Why

Stopping a task that a language SDK runs signalled the runtime's pid alone, so a CLI the task had started was reparented to PID 1 and kept running. That child also holds the runtime open, so the runtime itself survived until the SIGKILL escalation killed_task_cleanup_time (60 s by default) later.

How

Python tasks have not had this problem since #65738: the task runner gets its own process group and the supervisor signals the group with os.killpg. _PopenActivitySubprocess never opted in, so the same killpg path in _signal_subprocess was never used for a language SDK task.

It now does, which covers every way the supervisor stops a task (a cleared task, execution_timeout, repeated heartbeat failures, task-success overtime).

proc = subprocess.Popen(
    [*command, f"--comm={...}", f"--logs={...}"],
    ...
    process_group=0,
)

WatchedSubprocess._signal_subprocess is pulled out to signal_process_tree in supervisor.py, so the supervisor and the coordinator share one implementation.

A startup that fails signals the group too. A runtime that exits before connecting is already reaped, so its group can no longer be resolved and is reached by the leader's pid instead.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code Opus 5 following the guidelines

Stopping a task that a language SDK runs signalled the runtime's pid alone, so anything the task had started was reparented to PID 1 and kept running. While such a child is alive it also holds the runtime open, so the runtime itself survived until the SIGKILL escalation `killed_task_cleanup_time` later. The runtime now starts with `process_group=0` and the supervisor is told the child has its own group, so the existing `os.killpg` path in `_signal_subprocess` reaches the whole tree, as it has for Python tasks since apache#65505. A startup that fails before the supervisor takes over signals the group too. This covers every way the supervisor stops a task -- a cleared task, `execution_timeout`, repeated heartbeat failures and task-success overtime -- for the TypeScript, Java and executable runtimes alike. closes: apache#75533 Co-Authored-By: Claude <noreply@anthropic.com>

… group - Pull `WatchedSubprocess._signal_subprocess` out to `signal_process_tree` in `supervisor.py` and call it from both places, instead of a near-copy in the coordinator with looser error handling. - A runtime that exits before connecting is reaped by `_accept_connections`, so its group can no longer be resolved and `Popen.send_signal` is a no-op on an exited process. Sweep the group by the leader's pid, guarded against pid reuse. - Make the `process_group=0` and `new_process_group=True` sites reference each other, since they must agree. - Drop the tests the shared helper's own tests in `test_supervisor.py` already cover, give the remaining mocked tests one style through a fixture, and add a real-process test for the startup-failure path. Co-Authored-By: Claude <noreply@anthropic.com>

jason810496

Thanks for the review.

jason810496 · 2026-10-27T06:31:04Z

requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/apache-airflow-providers-apprise/json (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1016)')))

Both CI failure so far are flaky, just triggered re-run.

github-actions · 2026-10-27T07:59:22Z

Backport failed to create: v3-4-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-4-test

You can attempt to backport this manually by running:

cherry_picker 750f849 v3-4-test

This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…) (#75731) Stopping a task that a language SDK runs signalled the runtime's pid alone, so anything the task had started was reparented to PID 1 and kept running. While such a child is alive it also holds the runtime open, so the runtime itself survived until the SIGKILL escalation `killed_task_cleanup_time` later. The runtime now starts with `process_group=0` and the supervisor is told the child has its own group, so the existing `os.killpg` path reaches the whole tree, as it has for Python tasks since #65505. A startup that fails signals the group too, and a runtime that exits before connecting is already reaped, so its group is reached by the leader's pid. This covers every way the supervisor stops a task -- a cleared task, `execution_timeout`, repeated heartbeat failures and task-success overtime -- for the TypeScript, Java and executable runtimes alike. (cherry picked from commit 750f849) closes: #75533 Co-authored-by: Claude <noreply@anthropic.com>

boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk labels Oct 27, 2026

jason810496 marked this pull request as ready for review October 27, 2026 04:55

jason810496 requested a review from uranusjr as a code owner October 27, 2026 04:55

jason810496 self-assigned this Oct 27, 2026

jason810496 added the backport-to-v3-4-test Backport to v3-4-test label Oct 27, 2026

jason810496 added this to the Airflow 3.4.0 milestone Oct 27, 2026

uranusjr mentioned this pull request Oct 27, 2026

Stop a language SDK task's children when the task is stopped #75545

Closed

1 task done