Repository navigation
Fix stale queued asset rows in scheduler cleanup and readiness - #76002
Merged
Merged
Conversation
uranusjr
changed the title
The cleanup after creating asset-triggered or asset-gated runs deleted consumed asset_dag_run_queue rows by a range of event IDs. A DELETE waits on rows locked by another transaction instead of skipping them, so it could deadlock with a concurrent dequeue (e.g. through the API), and the database may abort the other side instead of the scheduler. The cleanup now deletes only rows the scheduler locked, by primary key in chunks to keep the bind parameter count bounded. Their events are still matched against consumed events with a subquery. A row skipped by the lock is left for whoever holds it, even if its event is consumed; it is cleaned up in a later scheduling loop if it remains. Also correct the docstring that claimed the event ID bound excludes every event queued after the lock was taken, rename the test covering skipped rows to match the new behavior, and drop issue references from test docstrings.
A queued asset row whose event was already consumed by the Dag still counted towards its asset condition. Combined with a fresh event for another asset, a stale row could satisfy an AND condition, and the run then consumed only the fresh events. Readiness now only counts rows of unconsumed events, both when Dags are selected and when the condition is re-evaluated under row locks, which the asset-triggered path now also does. The catchup wakeup row, which deliberately kept a consumed row queued, is replaced by queuing a row for the next backlog event.
With a single row for the next backlog event, a catchup Dag with a multi-asset condition stopped being selected once that row covered only one of the assets, stalling the backlog until new events arrived. Queue the earliest unconsumed event of each scheduled asset instead, so the condition is evaluated against the backlog the same way it is against queued events without catchup.
uranusjr
force-pushed
the
fix-adrq-unbounded-in
branch
from
October 30, 2026 03:48
f8fb4d6 to
8a66a1d
Compare
vatsrahul1001
approved these changes
Oct 30, 2026
Lee-W
approved these changes
Oct 30, 2026
Lee-W
approved these changes
Oct 30, 2026
Contributor
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 9e662c2 v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
1 task done
vatsrahul1001
added a commit
that referenced
this pull request
Oct 30, 2026
vatsrahul1001
added a commit
that referenced
this pull request
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to @kaxil’s comments on #75943.
Cleanup only deletes rows the scheduler locked. It deleted consumed asset_dag_run_queue rows by event ID range, which waits on rows locked elsewhere and could deadlock with a concurrent dequeue (e.g. through the API). It now deletes locked rows by primary key in bounded chunks. A row skipped by the lock is cleaned up in a later loop.
Readiness ignores consumed rows. A stale row (its event already consumed by the Dag) still counted towards the asset condition, so it could satisfy an AND condition with a fresh event for another asset. Readiness now counts only rows of unconsumed events, both when selecting Dags and when re-evaluating under row locks. The asset-triggered path now re-evaluates under locks like the asset-gated path. This came in with #70972 and #68517 and is unreleased.
Catchup queues the next backlog event per asset. This replaces the wakeup row, which kept a consumed row queued. Queuing per asset keeps multi-asset conditions from stalling while the backlog drains. While a catchup backlog drains, this adds one query per Dag per loop that ranks the Dag’s unconsumed events by asset, similar in cost to the existing catchup event select.