Skip to content

[v3-4-test] Fix LocalExecutor deadlock when a workload put blocks (#75086) - #76223

Merged
potiuk merged 1 commit into
v3-4-testfrom
backport-1d1d17b-v3-4-test
Nov 1, 2026
Merged

potiuk merged 1 commit into
v3-4-testfrom
backport-1d1d17b-v3-4-test

Conversation

@github-actions

@github-actions github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor
  • Keep draining LocalExecutor results while a workload put is blocked

Draining the result queue once before each put did not remove the
deadlock reported in #70526. A worker can write a result between the
drain and the put; if the activity pipe is full at that point the put
blocks, and the worker keeps writing results until the result pipe is
full too, so each side waits on the other. Linux pipes free space only a
page at a time, which is why the regression test from #70545 hangs on
loaded CI runners but not on macOS.

Make the put on a short-lived helper thread and keep reading results on
the main thread until it completes, so the result pipe can never fill up
behind a blocked put. The workload is registered as running before it is
sent, so a "running" result read while the put is in flight is not
dropped.

Generated-by: Claude Opus 5

  • Undo a failed LocalExecutor put and spawn workers while one is blocked

Registering a workload before its put meant a put that raised left the
key running and counted although nothing was sent; restore the queued
workload and its bookkeeping instead. Counting each workload before its
put keeps a worker from decrementing the outstanding count below zero
when a batch is interrupted part way.

A blocked put can only complete once a live worker reads the activity
pipe, but workers were only spawned or replaced after the batch, so with
the spawn start method (or once every worker had died) a batch larger
than the pipe waited forever. Spawn them from the drain loop whenever
fewer than parallelism are alive: spawn starts one per call, so a single
busy worker would otherwise hold the put until its task finished.

Generated-by: Claude Opus 5

  • Cover undoing a failed put while an earlier dispatch of the key is running

The test only reached the no-earlier-dispatch path, so the remaining > 0 branch of _undo_dispatch had no coverage.
(cherry picked from commit 1d1d17b)

Co-authored-by: Jarek Potiuk jarek@potiuk.com
Generated-by: Claude Opus 5

…5086)

* Keep draining LocalExecutor results while a workload put is blocked

Draining the result queue once before each put did not remove the
deadlock reported in #70526. A worker can write a result between the
drain and the put; if the activity pipe is full at that point the put
blocks, and the worker keeps writing results until the result pipe is
full too, so each side waits on the other. Linux pipes free space only a
page at a time, which is why the regression test from #70545 hangs on
loaded CI runners but not on macOS.

Make the put on a short-lived helper thread and keep reading results on
the main thread until it completes, so the result pipe can never fill up
behind a blocked put. The workload is registered as running before it is
sent, so a "running" result read while the put is in flight is not
dropped.

Generated-by: Claude Opus 5

* Undo a failed LocalExecutor put and spawn workers while one is blocked

Registering a workload before its put meant a put that raised left the
key running and counted although nothing was sent; restore the queued
workload and its bookkeeping instead. Counting each workload before its
put keeps a worker from decrementing the outstanding count below zero
when a batch is interrupted part way.

A blocked put can only complete once a live worker reads the activity
pipe, but workers were only spawned or replaced after the batch, so with
the spawn start method (or once every worker had died) a batch larger
than the pipe waited forever. Spawn them from the drain loop whenever
fewer than parallelism are alive: spawn starts one per call, so a single
busy worker would otherwise hold the put until its task finished.

Generated-by: Claude Opus 5

* Cover undoing a failed put while an earlier dispatch of the key is running

The test only reached the no-earlier-dispatch path, so the remaining > 0 branch of _undo_dispatch had no coverage.
(cherry picked from commit 1d1d17b)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Generated-by: Claude Opus 5
@boring-cyborg boring-cyborg Bot added the area:Executors-core LocalExecutor & SequentialExecutor label Nov 1, 2026
@potiuk
potiuk marked this pull request as ready for review November 1, 2026 12:33
@potiuk
potiuk requested review from XD-DENG and ashb as code owners November 1, 2026 12:33
@potiuk potiuk added this to the Airflow 3.4.1 milestone Nov 1, 2026
@potiuk
potiuk merged commit 0db6726 into v3-4-test Nov 1, 2026
94 of 109 checks passed
@potiuk
potiuk deleted the backport-1d1d17b-v3-4-test branch November 1, 2026 13:02
vatsrahul1001 pushed a commit that referenced this pull request Nov 4, 2026
…5086) (#76223)

* Keep draining LocalExecutor results while a workload put is blocked

Draining the result queue once before each put did not remove the
deadlock reported in #70526. A worker can write a result between the
drain and the put; if the activity pipe is full at that point the put
blocks, and the worker keeps writing results until the result pipe is
full too, so each side waits on the other. Linux pipes free space only a
page at a time, which is why the regression test from #70545 hangs on
loaded CI runners but not on macOS.

Make the put on a short-lived helper thread and keep reading results on
the main thread until it completes, so the result pipe can never fill up
behind a blocked put. The workload is registered as running before it is
sent, so a "running" result read while the put is in flight is not
dropped.

Generated-by: Claude Opus 5

* Undo a failed LocalExecutor put and spawn workers while one is blocked

Registering a workload before its put meant a put that raised left the
key running and counted although nothing was sent; restore the queued
workload and its bookkeeping instead. Counting each workload before its
put keeps a worker from decrementing the outstanding count below zero
when a batch is interrupted part way.

A blocked put can only complete once a live worker reads the activity
pipe, but workers were only spawned or replaced after the batch, so with
the spawn start method (or once every worker had died) a batch larger
than the pipe waited forever. Spawn them from the drain loop whenever
fewer than parallelism are alive: spawn starts one per call, so a single
busy worker would otherwise hold the put until its task finished.

Generated-by: Claude Opus 5

* Cover undoing a failed put while an earlier dispatch of the key is running

The test only reached the no-earlier-dispatch path, so the remaining > 0 branch of _undo_dispatch had no coverage.
(cherry picked from commit 1d1d17b)


Generated-by: Claude Opus 5

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Executors-core LocalExecutor & SequentialExecutor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants