Skip to content

Child processes of language SDK tasks survive when the task is stopped #75533

Closed

Description

@shivaam

Under which category would you file this issue?

Task SDK

Apache Airflow version

main (b43705c)

What happened and how to reproduce it?

When the supervisor stops a task that a language SDK runs (TypeScript through NodeCoordinator, and the same for JavaCoordinator and ExecutableCoordinator), it signals only the runtime process. Processes the task started keep running.

The common case is clearing a running task. The heartbeat gets a 409, the supervisor sends SIGTERM, and the try ends with server_terminated, so the task state is handled correctly. But a CLI the task started (for example with child_process.spawn) is re-parented to PID 1 and keeps running alongside the next try. While that child is alive it also keeps Node's event loop open, so once the handler has returned, the runtime exits only when the SIGKILL escalation fires after [core] killed_task_cleanup_time (60 s by default).

A TypeScript handler that shows it:

import { spawn } from "node:child_process";
import { Bundle, TaskHandler, getContext } from "apache-airflow-ts-sdk";

async function runCli() {
  const { signal } = getContext();
  spawn("sleep", ["300"], { stdio: "ignore" }); // stands in for any CLI the task runs
  await new Promise((_, reject) => signal.addEventListener("abort", () => reject(signal.reason)));
}

await new Bundle(new TaskHandler("my_dag", "run_cli", runCli)).serve();

Reproduced on main with a local test. It packs this handler with airflow-ts-pack and runs it through NodeCoordinator.execute_task with the real Node runtime. The Execution API is mocked and answers the heartbeat with 409 once the handler is running, as the API server does after the task is cleared. The supervisor reported only server_terminated. But sleep 300 was still running with PPID 1, and the runtime was ended by SIGKILL (exit code -9) after the escalation delay. This was not tried in a full deployment.

Python tasks have not had this problem since #65738 (for #65505): the task process runs in its own process group, and the supervisor signals the group with os.killpg. _PopenActivitySubprocess.start overrides ActivitySubprocess.start. It calls subprocess.Popen without a process group and does not set new_process_group, so _signal_subprocess signals the runtime's pid alone:

  • proc = subprocess.Popen(
    [
    *command,
    "--comm={0[0]}:{0[1]}".format(comm_server.getsockname()),
    "--logs={0[0]}:{0[1]}".format(logs_server.getsockname()),
    ],
    stdout=stdout_w.fileno(),
    stderr=stderr_w.fileno(),
    env=_build_runtime_env(),
    )
    tracker.track(proc)
    for soc in tracker.untrack(stdout_w, stderr_w):
    soc.close()
    log.info("Starting subprocess", pid=proc.pid)
    socks, drained = _accept_connections(
    {"comm": comm_server, "logs": logs_server},
    {"stdout": stdout_r, "stderr": stderr_r},
    proc,
    max_wait=startup_timeout,
    )
    tracker.track(*socks.values())
    self = cls(
    id=what.id,
    pid=proc.pid,
    process=PopenTracker(proc),
    process_log=logger or structlog.get_logger(logger_name="task").bind(),
    start_time=time.monotonic(),
    stdin=socks[comm_server],
    subprocess_schema_version=subprocess_schema_version,
    comm_server=comm_server,
    logs_server=logs_server,
    **kwargs,
    )
  • def _signal_subprocess(self, sig: signal.Signals) -> None:
    """
    Deliver ``sig`` to the child process, or to its whole process group when it has its own.
    When ``new_process_group`` was set at ``start()`` time, the signal is sent with
    ``os.killpg`` so subprocesses spawned by the child (venv children, bash shells, etc.)
    are reached too (see issue #65505). Falls back to signalling the child PID alone when
    the group cannot be resolved or signalled -- and, critically, when the child still
    shares the supervisor's own process group (``setpgid`` failed), because ``killpg`` on
    our own group would signal the supervisor itself and its siblings.
    """
    if self._new_process_group:
    try:
    pgid = os.getpgid(self._process.pid)
    except (ProcessLookupError, PermissionError):
    pgid = None
    if pgid is not None and pgid != os.getpgid(0):
    try:
    os.killpg(pgid, sig)
    return
    except (ProcessLookupError, PermissionError):
    pass
    self._process.send_signal(sig)

The same applies to the other ways the supervisor stops a language SDK task: execution_timeout, repeated heartbeat failures and task-success overtime.

What you think should happen instead?

Stopping a language SDK task should not leave the processes it started running. Two broad directions, each worth researching:

  1. In the supervisor: run the language runtime in its own process group and signal the whole group, as is done for Python tasks. Emulating this in the same test stopped the child.
  2. In the language runtime or the task: have the SDK, or the task author, stop the child processes when the task is aborted (for example, the TS SDK's ctx.signal can be passed to spawn).

Operating System

Reproduced on macOS; the launch code is the same on any POSIX worker.

Deployment

Not applicable: reproduced with a test against main.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct


Drafted-by: Claude Code (Opus 5.5); reviewed by @shivaam before posting

Activity

  1. self-assigned this
    on Oct 27, 2026
  2. jason810496 commented on Oct 27, 2026

    @jason810496
    Member

    Yes, it's a real issue that we should at supervisor level not in each SDK runtime.
    #75387 will solve it partially.

  3. removed
    needs-triagelabel for new issues that we didn't triage yet
    on Oct 27, 2026
  4. added a commit that references this issue on Oct 27, 2026
    750f849
  5. added a commit that references this issue on Oct 27, 2026
    e2edfff
  6. added
    priority:mediumBug that should be fixed before next release but would not block a release
    on Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

affected_version:3.4.0rcCan be used for bugs identified during 3.4.0 testingpriority:mediumBug that should be fixed before next release but would not block a release

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions