Drain result_queue before dispatch to prevent runtime deadlock in LocalExecutor by SakshamKapoor2911 · Pull Request #70545 · apache/airflow

SakshamKapoor2911 · 2026-07-27T16:46:10Z

Fixes a runtime deadlock in LocalExecutor where the scheduler blocks indefinitely on activity_queue.put() during task dispatch, a gap left by #67881 which only addressed the shutdown path.

Workers fill the result_queue pipe buffer and block on put(). Blocked workers stop consuming activity_queue, so the scheduler blocks on activity_queue.put() and never reaches _read_results() to drain the pipe — a circular deadlock.

The fix: drain result_queue via _read_results() before each activity_queue.put() in _process_workloads, using the same drain-before-write pattern already applied to end().

closes: #70526

Was generative AI tooling used to co-author this PR?

Yes — OpenCode (DeepSeek v4 Pro)

Generated-by: OpenCode (DeepSeek v4 Pro) following the guidelines

SakshamKapoor2911 · 2026-07-31T11:15:03Z

@eladkal @amoghrajesh — all CI checks pass. Fixes a LocalExecutor shutdown deadlock. Ready for maintainer review.

vatsrahul1001 · 2026-08-04T08:01:19Z

LGTM! should be good to merged post code owner review

vatsrahul1001 · 2026-08-05T04:21:53Z

Moving to 3.3.2 as this is still pending code owner review and do not want to rush on merging this as it not critical

SakshamKapoor2911 · 2026-08-11T12:41:42Z

@vatsrahul1001 thanks for the approval and for moving this to the 3.3.2 milestone.

@XD-DENG @ashb @o-nikolas @pierrejeambrun @hussein-awala @dheerajturaga CI is green; PR needs a code owner sign-off to merge. It is a relatively small fix; could one of you take a look when you have a moment?

darren-cheung-sa · 2026-09-17T14:07:43Z

Moving to 3.3.2 as this is still pending code owner review and do not want to rush on merging this as it not critical

Seems it wasn't included in 3.3.2. And looks like there are no deadlines for 3.3.3 or 3.4.0

For our system (using LocalExecutor), it is a bit urgent since it means there is always a risk of a deadlock happening (stopping all our DAGs)

darren-cheung-sa · 2026-09-21T14:09:43Z

@vatsrahul1001 CC on this

SakshamKapoor2911 · 2026-10-07T05:13:08Z

Rebased onto current main (62917eab) and re-verified.

Correction: an earlier comment described this as a shutdown deadlock fix. That was inaccurate — this PR addresses the runtime / dispatch path (_process_workloads()), which is the variant reported in #70526 and which #67881 (the end() drain) does not cover. #67881 fixed the shutdown/join path; this adds a result_queue drain before each activity_queue.put() during normal dispatch.

Why: workers can block writing results into a full result_queue pipe → they stop consuming activity_queue → the scheduler blocks in activity_queue.put() and, being single-threaded at that point, never reaches _read_results() → circular wait.

Change: one line in local_executor.py — self._read_results() before self.activity_queue.put(workload) in _process_workloads. The branch had gone stale against main's workload-API refactor, so I rebased and adapted the tests to the new API (get_workload_key / executor_queues / _dispatch_counts).

Tests (run locally on this branch, fork start method, Python 3.12):

pytest airflow-core/tests/unit/executors/test_local_executor.py → 59 passed, 1 skipped.
Added test_process_workloads_drains_results_before_put (asserts drain-before-dispatch ordering).
Added a bounded real-queue regression test test_process_workloads_progresses_under_concurrent_result_refill that runs the actual _process_workloads with real multiprocessing.SimpleQueues and a worker concurrently refilling results. Under identical conditions it fails (times out) without the patch and completes with it (measured pipe capacity F_GETPIPE_SZ = 65536, kernel 6.8.0).

I have not verified a production deployment — this is a synthetic reproduction that passes with the patch. Feedback on the approach is welcome.

…e dispatch deadlock Fixes the runtime/dispatch-path deadlock reported in apache#70526: during _process_workloads the scheduler could block in activity_queue.put() while workers were blocked writing results into a full result_queue pipe, a circular wait. Draining the result queue before each put breaks the cycle. The end()/join path is already handled by apache#67881; this covers normal dispatch. Adds a call-order unit test and a bounded real-queue regression test.

github-actions · 2026-10-18T06:16:45Z

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-4-test

…event runtime dispatch deadlock (#70545) (#74949) Fixes the runtime/dispatch-path deadlock reported in #70526: during _process_workloads the scheduler could block in activity_queue.put() while workers were blocked writing results into a full result_queue pipe, a circular wait. Draining the result queue before each put breaks the cycle. The end()/join path is already handled by #67881; this covers normal dispatch. Adds a call-order unit test and a bounded real-queue regression test. (cherry picked from commit b00cfeb) Co-authored-by: Saksham Kapoor <sakshamkapoor2911@gmail.com>

…e dispatch deadlock (apache#70545) Fixes the runtime/dispatch-path deadlock reported in apache#70526: during _process_workloads the scheduler could block in activity_queue.put() while workers were blocked writing results into a full result_queue pipe, a circular wait. Draining the result queue before each put breaks the cycle. The end()/join path is already handled by apache#67881; this covers normal dispatch. Adds a call-order unit test and a bounded real-queue regression test.

Draining the result queue once before each put did not remove the deadlock reported in apache#70526. A worker can write a result between the drain and the put; if the activity pipe is full at that point the put blocks, and the worker keeps writing results until the result pipe is full too, so each side waits on the other. Linux pipes free space only a page at a time, which is why the regression test from apache#70545 hangs on loaded CI runners but not on macOS. Make the put on a short-lived helper thread and keep reading results on the main thread until it completes, so the result pipe can never fill up behind a blocked put. The workload is registered as running before it is sent, so a "running" result read while the put is in flight is not dropped. Generated-by: Claude Opus 5

* Keep draining LocalExecutor results while a workload put is blocked Draining the result queue once before each put did not remove the deadlock reported in #70526. A worker can write a result between the drain and the put; if the activity pipe is full at that point the put blocks, and the worker keeps writing results until the result pipe is full too, so each side waits on the other. Linux pipes free space only a page at a time, which is why the regression test from #70545 hangs on loaded CI runners but not on macOS. Make the put on a short-lived helper thread and keep reading results on the main thread until it completes, so the result pipe can never fill up behind a blocked put. The workload is registered as running before it is sent, so a "running" result read while the put is in flight is not dropped. Generated-by: Claude Opus 5 * Undo a failed LocalExecutor put and spawn workers while one is blocked Registering a workload before its put meant a put that raised left the key running and counted although nothing was sent; restore the queued workload and its bookkeeping instead. Counting each workload before its put keeps a worker from decrementing the outstanding count below zero when a batch is interrupted part way. A blocked put can only complete once a live worker reads the activity pipe, but workers were only spawned or replaced after the batch, so with the spawn start method (or once every worker had died) a batch larger than the pipe waited forever. Spawn them from the drain loop whenever fewer than parallelism are alive: spawn starts one per call, so a single busy worker would otherwise hold the put until its task finished. Generated-by: Claude Opus 5 * Cover undoing a failed put while an earlier dispatch of the key is running The test only reached the no-earlier-dispatch path, so the remaining > 0 branch of _undo_dispatch had no coverage. Generated-by: Claude Opus 5

…5086) (#76223) * Keep draining LocalExecutor results while a workload put is blocked Draining the result queue once before each put did not remove the deadlock reported in #70526. A worker can write a result between the drain and the put; if the activity pipe is full at that point the put blocks, and the worker keeps writing results until the result pipe is full too, so each side waits on the other. Linux pipes free space only a page at a time, which is why the regression test from #70545 hangs on loaded CI runners but not on macOS. Make the put on a short-lived helper thread and keep reading results on the main thread until it completes, so the result pipe can never fill up behind a blocked put. The workload is registered as running before it is sent, so a "running" result read while the put is in flight is not dropped. Generated-by: Claude Opus 5 * Undo a failed LocalExecutor put and spawn workers while one is blocked Registering a workload before its put meant a put that raised left the key running and counted although nothing was sent; restore the queued workload and its bookkeeping instead. Counting each workload before its put keeps a worker from decrementing the outstanding count below zero when a batch is interrupted part way. A blocked put can only complete once a live worker reads the activity pipe, but workers were only spawned or replaced after the batch, so with the spawn start method (or once every worker had died) a batch larger than the pipe waited forever. Spawn them from the drain loop whenever fewer than parallelism are alive: spawn starts one per call, so a single busy worker would otherwise hold the put until its task finished. Generated-by: Claude Opus 5 * Cover undoing a failed put while an earlier dispatch of the key is running The test only reached the no-earlier-dispatch path, so the remaining > 0 branch of _undo_dispatch had no coverage. (cherry picked from commit 1d1d17b) Generated-by: Claude Opus 5 Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

SakshamKapoor2911 requested review from XD-DENG, ashb, dheerajturaga, hussein-awala, o-nikolas and pierrejeambrun as code owners July 27, 2026 16:46

boring-cyborg Bot added the area:Executors-core LocalExecutor & SequentialExecutor label Jul 27, 2026

SakshamKapoor2911 mentioned this pull request Jul 27, 2026

Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 #70526

Closed

1 of 2 tasks

SakshamKapoor2911 force-pushed the fix-local-executor-dispatch-deadlock branch from f03a04a to df19a88 Compare July 27, 2026 17:10

potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jul 28, 2026

eladkal added this to the Airflow 3.3.1 milestone Jul 31, 2026

eladkal added type:bug-fix Changelog: Bug Fixes backport-to-v3-3-test labels Jul 31, 2026

vatsrahul1001 approved these changes Aug 4, 2026

View reviewed changes

vatsrahul1001 modified the milestones: Airflow 3.3.1, Airflow 3.3.2 Aug 5, 2026

pierrejeambrun approved these changes Aug 17, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/executors/test_local_executor.py Resolved

pierrejeambrun added backport-to-v3-3-test and removed backport-to-v3-3-test labels Aug 17, 2026

pierrejeambrun mentioned this pull request Aug 17, 2026

Drain result_queue in LocalExecutor to prevent process join deadlock #67881

Merged

1 task done

pierrejeambrun modified the milestones: Airflow 3.3.2, Airflow 3.4.0 Aug 20, 2026

pierrejeambrun removed the backport-to-v3-3-test label Aug 20, 2026

pierrejeambrun modified the milestones: Airflow 3.4.0, Airflow 3.3.2 Aug 20, 2026

pierrejeambrun added the backport-to-v3-3-test label Aug 20, 2026

pierrejeambrun modified the milestones: Airflow 3.3.2, Airflow 3.4.0 Aug 20, 2026

pierrejeambrun removed the backport-to-v3-3-test label Aug 20, 2026

SakshamKapoor2911 force-pushed the fix-local-executor-dispatch-deadlock branch from ac7fd59 to a4e3b06 Compare October 7, 2026 05:12

ashb approved these changes Oct 15, 2026

View reviewed changes

ashb force-pushed the fix-local-executor-dispatch-deadlock branch from a4e3b06 to 22c2219 Compare October 15, 2026 12:29

ashb added the backport-to-v3-4-test Backport to v3-4-test label Oct 15, 2026

vatsrahul1001 merged commit b00cfeb into apache:main Oct 18, 2026
79 checks passed

potiuk mentioned this pull request Oct 19, 2026

Fix LocalExecutor deadlock when a workload put blocks #75086

Merged

1 task done

o-nikolas mentioned this pull request Oct 23, 2026

Keep LocalExecutor running when a worker dies holding the queue read lock #74588

Open

This was referenced Oct 28, 2026

Status of testing of Apache Airflow 3.4.0rc3 #75865

Closed

[v3-4-test] Fix flaky LocalExecutor concurrent result refill test (#75030) #76117

Merged

github-actions Bot mentioned this pull request Nov 1, 2026

[v3-4-test] Fix LocalExecutor deadlock when a workload put blocks (#75086) #76223

Merged

Navigation Menu

Repository navigation

Drain result_queue before dispatch to prevent runtime deadlock in LocalExecutor - #70545

Drain result_queue before dispatch to prevent runtime deadlock in LocalExecutor#70545
vatsrahul1001 merged 1 commit into
apache:mainfrom
SakshamKapoor2911:fix-local-executor-dispatch-deadlock

SakshamKapoor2911 commented Jul 27, 2026 •

edited

Loading

Uh oh!

SakshamKapoor2911 commented Jul 31, 2026

Uh oh!

vatsrahul1001 commented Aug 4, 2026

Uh oh!

vatsrahul1001 commented Aug 5, 2026

Uh oh!

SakshamKapoor2911 commented Aug 11, 2026

Uh oh!

Uh oh!

darren-cheung-sa commented Sep 17, 2026 •

edited

Loading

Uh oh!

darren-cheung-sa commented Sep 21, 2026

Uh oh!

SakshamKapoor2911 commented Oct 7, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

SakshamKapoor2911 commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was generative AI tooling used to co-author this PR?

Uh oh!

SakshamKapoor2911 commented Jul 31, 2026

Uh oh!

vatsrahul1001 commented Aug 4, 2026

Uh oh!

vatsrahul1001 commented Aug 5, 2026

Uh oh!

SakshamKapoor2911 commented Aug 11, 2026

Uh oh!

Uh oh!

darren-cheung-sa commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

darren-cheung-sa commented Sep 21, 2026

Uh oh!

SakshamKapoor2911 commented Oct 7, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 18, 2026

Backport successfully created: v3-4-test

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

SakshamKapoor2911 commented Jul 27, 2026 •

edited

Loading

darren-cheung-sa commented Sep 17, 2026 •

edited

Loading