Repository navigation
Description
Under which category would you file this issue?
Task SDK
Apache Airflow version
main (b43705c)
What happened and how to reproduce it?
When the supervisor stops a task that a language SDK runs (TypeScript through NodeCoordinator, and the same for JavaCoordinator and ExecutableCoordinator), it signals only the runtime process. Processes the task started keep running.
The common case is clearing a running task. The heartbeat gets a 409, the supervisor sends SIGTERM, and the try ends with server_terminated, so the task state is handled correctly. But a CLI the task started (for example with child_process.spawn) is re-parented to PID 1 and keeps running alongside the next try. While that child is alive it also keeps Node's event loop open, so once the handler has returned, the runtime exits only when the SIGKILL escalation fires after [core] killed_task_cleanup_time (60 s by default).
A TypeScript handler that shows it:
import { spawn } from "node:child_process";
import { Bundle, TaskHandler, getContext } from "apache-airflow-ts-sdk";
async function runCli() {
const { signal } = getContext();
spawn("sleep", ["300"], { stdio: "ignore" }); // stands in for any CLI the task runs
await new Promise((_, reject) => signal.addEventListener("abort", () => reject(signal.reason)));
}
await new Bundle(new TaskHandler("my_dag", "run_cli", runCli)).serve();Reproduced on main with a local test. It packs this handler with airflow-ts-pack and runs it through NodeCoordinator.execute_task with the real Node runtime. The Execution API is mocked and answers the heartbeat with 409 once the handler is running, as the API server does after the task is cleared. The supervisor reported only server_terminated. But sleep 300 was still running with PPID 1, and the runtime was ended by SIGKILL (exit code -9) after the escalation delay. This was not tried in a full deployment.
Python tasks have not had this problem since #65738 (for #65505): the task process runs in its own process group, and the supervisor signals the group with os.killpg. _PopenActivitySubprocess.start overrides ActivitySubprocess.start. It calls subprocess.Popen without a process group and does not set new_process_group, so _signal_subprocess signals the runtime's pid alone:
airflow/task-sdk/src/airflow/sdk/coordinators/_subprocess.py
Lines 359 to 393 in b43705c
proc = subprocess.Popen( [ *command, "--comm={0[0]}:{0[1]}".format(comm_server.getsockname()), "--logs={0[0]}:{0[1]}".format(logs_server.getsockname()), ], stdout=stdout_w.fileno(), stderr=stderr_w.fileno(), env=_build_runtime_env(), ) tracker.track(proc) for soc in tracker.untrack(stdout_w, stderr_w): soc.close() log.info("Starting subprocess", pid=proc.pid) socks, drained = _accept_connections( {"comm": comm_server, "logs": logs_server}, {"stdout": stdout_r, "stderr": stderr_r}, proc, max_wait=startup_timeout, ) tracker.track(*socks.values()) self = cls( id=what.id, pid=proc.pid, process=PopenTracker(proc), process_log=logger or structlog.get_logger(logger_name="task").bind(), start_time=time.monotonic(), stdin=socks[comm_server], subprocess_schema_version=subprocess_schema_version, comm_server=comm_server, logs_server=logs_server, **kwargs, ) airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py
Lines 1263 to 1285 in b43705c
def _signal_subprocess(self, sig: signal.Signals) -> None: """ Deliver ``sig`` to the child process, or to its whole process group when it has its own. When ``new_process_group`` was set at ``start()`` time, the signal is sent with ``os.killpg`` so subprocesses spawned by the child (venv children, bash shells, etc.) are reached too (see issue #65505). Falls back to signalling the child PID alone when the group cannot be resolved or signalled -- and, critically, when the child still shares the supervisor's own process group (``setpgid`` failed), because ``killpg`` on our own group would signal the supervisor itself and its siblings. """ if self._new_process_group: try: pgid = os.getpgid(self._process.pid) except (ProcessLookupError, PermissionError): pgid = None if pgid is not None and pgid != os.getpgid(0): try: os.killpg(pgid, sig) return except (ProcessLookupError, PermissionError): pass self._process.send_signal(sig)
The same applies to the other ways the supervisor stops a language SDK task: execution_timeout, repeated heartbeat failures and task-success overtime.
What you think should happen instead?
Stopping a language SDK task should not leave the processes it started running. Two broad directions, each worth researching:
- In the supervisor: run the language runtime in its own process group and signal the whole group, as is done for Python tasks. Emulating this in the same test stopped the child.
- In the language runtime or the task: have the SDK, or the task author, stop the child processes when the task is aborted (for example, the TS SDK's
ctx.signalcan be passed tospawn).
Operating System
Reproduced on macOS; the launch code is the same on any POSIX worker.
Deployment
Not applicable: reproduced with a test against main.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Drafted-by: Claude Code (Opus 5.5); reviewed by @shivaam before posting
Activity
jason810496 commentedon Oct 27, 2026
Yes, it's a real issue that we should at supervisor level not in each SDK runtime.
#75387 will solve it partially.
Run a language SDK runtime in its own process group (apache#75707)
[v3-4-test] Run a language SDK runtime in its own process group (#75707…