Repository navigation
[v3-4-test] Fix flaky LocalExecutor concurrent result refill test (#75030) - #76117
Merged
Merged
Conversation
The test sent 16 KiB results for each ~0.9 KiB workload. When the dispatch loop blocked on a full activity pipe, the worker could fill the result pipe before it freed a page of the activity pipe, closing a two-pipe deadlock that drain-before-put cannot prevent at that ratio. It hangs intermittently, more often when pipes are smaller. Production results are a few hundred bytes, so the test now uses results smaller than a workload, which makes it deterministic with the fix, and dispatches enough workloads that it still deadlocks without it. (cherry picked from commit cffe6df)
boring-cyborg
Bot
added
the
area:Executors-core
vatsrahul1001
enabled auto-merge (squash)
October 31, 2026 07:10
vatsrahul1001
disabled auto-merge
October 31, 2026 07:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #75030 to
v3-4-test.Why
v3-4-testCI is red on the core DB-test jobs — and therefore the rc2 sync PR #76068 — because theOtherparallel test type deadlocks and times out the whole job (56-min total-test-timeout → exit 2).The culprit is
airflow-core/tests/unit/executors/test_local_executor.py::TestLocalExecutor::test_process_workloads_progresses_under_concurrent_result_refill, the regression test added by #70545 (backported here as #74949). That test was flaky: it emitted 16 KiB result payloads for ~0.9 KiB workloads, which can close a two-pipe deadlock that the production drain-before-put fix is not meant to prevent at that ratio. It passes most of the time (so rc1's sync PR #75755 merged green on 2026-10-27) but deadlocks reliably under heavier parallel load.#75030 fixed the test — it uses result payloads smaller than a workload (512 B) and raises the workload count so it stays deterministic with the fix. Production
local_executor.pyis unchanged and is already byte-identical tomain; this is a test-only change.#75030 merged to
mainon 2026-10-19 but was never labelledbackport-to-v3-4-test, so auto-backport never carried it over (its parent #70545 was labelled and backported, leavingv3-4-testrunning the flaky test without its fix).This unblocks the 3.4.0rc2 sync.
Verification
Reproduced the deadlock on
v3-4-testlocally (parallel core-tests, sqlite backend, all six test types) —Otherhangs on this exact test. With this cherry-pick applied the same run no longer deadlocks. The CI on this PR is the authoritative check.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8)