Repository navigation
OpenLineage listener forks a ProcessPoolExecutor in the Airflow 3 API server, leaking orphaned worker processes #74544
Description
Under which category would you file this issue?
Providers
Apache Airflow version
3.3.1
What happened and how to reproduce it?
The API server leaks processes forked by the OpenLineage listener until the container is OOM-killed.
In Airflow 3, patch_dag_run_state (api_fastapi/core_api/services/public/dag_run.py) calls on_dag_run_success / on_dag_run_failed from inside the API server when a DAG run is marked success/failed through the UI or REST API. Manual task instance state changes do the same through _on_task_instance_manual_state_change. The OpenLineage listener hands these events to OpenLineageListener.executor, a ProcessPoolExecutor created without an mp_context, so on Linux with Python < 3.14 it forks the API server worker.
That causes two problems:
- Children deadlock. The uvicorn worker is multithreaded (AnyIO worker threads, DB pool, etc.), and forking a multithreaded process can leave the child holding a lock that will never be released. Some children hang in
futex_waitand never send their event, so lineage events are silently lost. - Children are never cleaned up.
before_stoppingis only called by components such as the scheduler, never by API server workers, so the pool is never shut down. Idle pool children block forever inpipe_read. They never get EOF because they inherited the write end of their own call-queue pipe. When gunicorn replaces a worker (worker_refresh_interval, timeout, crash), its pool children are reparented to PID 1 and live forever. Each one is a full copy of the worker (150-300 MiB PSS).
What we observed in one api-server pod, ~33 h after start:
master + 2 live workers ~580 MiB PSS
10 orphaned "gunicorn: worker" procs ~2,290 MiB PSS (ppid=1)
container memory.current 2,898 MiB of 3,272 MiB limit
Orphan states (/proc/<pid>/wchan): 5 × pipe_read (1 thread, idle pool worker), 5 × futex_wait_queue (2 threads, deadlocked after fork).
py-spy dump of an orphan (the stack the child inherited from its parent at fork time, followed by the pool worker loop):
Thread 184 (idle): "AnyIO worker thread"
_recv (multiprocessing/connection.py:395)
_recv_bytes (multiprocessing/connection.py:430)
recv_bytes (multiprocessing/connection.py:216)
get (multiprocessing/queues.py:103)
_process_worker (concurrent/futures/process.py:252)
run (multiprocessing/process.py:108)
_bootstrap (multiprocessing/process.py:314)
_launch (multiprocessing/popen_fork.py:71)
__init__ (multiprocessing/popen_fork.py:19)
_Popen (multiprocessing/context.py:282)
start (multiprocessing/process.py:121)
_spawn_process (concurrent/futures/process.py:799)
_launch_processes (concurrent/futures/process.py:789)
_start_executor_manager_thread (concurrent/futures/process.py:762)
submit (concurrent/futures/process.py:823)
submit_callable (openlineage/plugins/listener.py:971)
on_dag_run_success (openlineage/plugins/listener.py:893)
...pluggy...
patch_dag_run_state (airflow/api_fastapi/core_api/services/public/dag_run.py:204)
patch_dag_run (airflow/api_fastapi/core_api/routes/public/dag_run.py:232)
run (anyio/_backends/_asyncio.py:1100)
Even without worker refresh, every API server worker that has handled a manual state change keeps one extra forked process (~250 MiB) for its whole lifetime. Worker refresh, which is meant to limit memory growth, turns that into an unbounded leak.
What you think should happen instead?
The listener should not fork from the API server. The process pool makes sense in the scheduler (isolating event emission from the scheduler loop, see #39235), but in the API server the work is a few HTTP calls. Possible fixes:
- Use a
ThreadPoolExecutorwhen the listener runs outside the scheduler (e.g. when the hook is invoked from the API server). - Create the pool with a non-fork
mp_context(forkserver/spawn) and shut the pool down when an API server worker exits, so children cannot outlive their worker. This avoids the deadlocks, but each child re-imports Airflow.
Operating System
Debian (official apache/airflow:slim-3.3.1-python3.12 image), Python 3.12.13
Deployment
Other 3rd-party Helm chart
Apache Airflow Provider(s)
openlineage
Versions of Apache Airflow Providers
apache-airflow-providers-standard==1.11.1
apache-airflow-providers-cncf-kubernetes==10.17.0
apache-airflow-providers-openlineage==2.20.2
openlineage-python==1.53.0
Official Helm Chart version
Not Applicable
Kubernetes Version
1.35
Helm Chart configuration
Not relevant
Docker Image customizations
Not Applicable
Anything else?
Deployment details
airflow api-serverwith[api] server_type = gunicorn,[api] workers = 2,[api] worker_refresh_interval = 10800- gunicorn 26.2.0
- OpenLineage enabled with the
httptransport - Kubernetes, api-server container memory limit ~3.2 GiB
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
garvit-arora commentedon Oct 12, 2026
@vlieven I can work on a focused fix for this. I’ll trace where the OpenLineage listener is invoked from API-server workers and how its executor is created and shut down, then keep any change scoped to avoiding forked, orphaned workers there while preserving scheduler behavior. I did not find an existing PR for #74544. If you already have a patch underway, please let me know so we can coordinate rather than duplicate it.
vlieven commentedon Oct 12, 2026
Hi @garvit-arora, thank you! I am currently working around the issue by patching the provider to use the ThreadPoolExecutor instead of the ProcessPoolExecutor. This essentially reverts the change made in #39235, so I am not quite sure it is the correct upstream fix for the issue, and hence I did not open a PR with those changes.
garvit-arora commentedon Oct 12, 2026
Thanks, this helps clarify the tradeoff. #39235 addressed scheduler hangs with CeleryExecutor and HTTP emission, so switching the listener to threads everywhere could reintroduce that problem.
I have a local, unsubmitted patch that keeps
ProcessPoolExecutorfor the scheduler and uses a singleThreadPoolExecutoronly in API-server workers. I’m checking the API-server detection and worker-lifecycle behavior before proposing it upstream.With your thread-pool workaround, have the orphaned processes and missed OpenLineage events both stopped during API-server worker refresh? Any observations from that run would help establish the regression case for a focused PR.
Shoryamishra61 commentedon Oct 12, 2026
Thanks for catching this. Agreed that sharing the lazily initialized adapter across multiple API emission threads introduces a concurrency risk.
I'll constrain the API-server executor to a single worker while preserving the scheduler's existing ProcessPoolExecutor configuration, and add regression coverage for configurations where
dag_state_change_process_pool_size > 1, concurrent submissions, and worker shutdown.Appreciate the review -- I'd prefer to keep the fix focused on the original process-leak issue without introducing a separate adapter ownership problem.
vlieven commentedon Oct 13, 2026
@garvit-arora I don't have orphaned process anymore when swapping to a ThreadPoolExecutor. I've also not observed missed OpenLineage events, but I imagine we're not running it at a frequency that #39235 addressed.
garvit-arora commentedon Oct 14, 2026
Thanks @vlieven for checking the workaround in a running deployment. The disappearance of orphaned processes is useful operational evidence. As you noted, the observed event rate cannot establish whether thread dispatch handles the higher-frequency case that motivated #39235.
Thanks @Shoryamishra61 for taking the upstream fix in #74577. The latest PR head limits API-server dispatch to one thread, retains the scheduler's configured process pool, and adds coverage for pool sizes above one, concurrent submissions, and shutdown. That addresses the adapter-ownership concern I raised. I'll follow that PR instead of opening a competing implementation. It still needs the project's normal review and CI before merge.