Repository navigation
Allow explicit runs to drain a paused Dag - #73763
Conversation
boring-cyborg
Bot
added
area:airflow-ctl
area:API
dheerajturaga
added
the
skip newsfragment check
Let's just remove it from its own section and either put it in Advanced or have it always shown. I like the radio cards because we definitely need to tell a user how many dag runs will start running in unpaused or draining options. I would prefer the radio cards to be horizontal instead of vertical. I don't think we need a separate warning banner. It should be inline with the option so that it is very clear which action will run how many dag runs. |
@ash, I guess your concern is that it isnt fairly obvious that "Drain" will cause pre-existing in flight dag runs which were paused mid way to start executing, What do you think of the current approach with the new push where I make it obvious to the user saying that drain will cause these dag runs to start execution aswell? 
|
9f474cf to
769cd44
Compare
eladkal
left a comment
eladkal
left a comment
There was a problem hiding this comment.
LGTM
I like how it looks now!
769cd44 to
17c259a
Compare
17c259a to
ef46e73
Compare
|
Screenshot looks much better, thank you. (My concern was with the section being collapsed meaning the behaviour could be hidden) I'll give the code one last pass |
A paused Dag never starts the runs a user triggers or backfills, so the only way to run them was to unpause the Dag, which also resumes its schedule. Dags that only reprocess historical data on demand then get scheduled runs that fail or waste resources (see apache#60768). Letting runs execute on a paused Dag was rejected, because a paused Dag should never run. Draining already means "create no scheduled runs, finish the existing ones, then pause", which is the behaviour these users want. Starting the drain together with the explicit run keeps the paused contract intact and lands the Dag back on paused once its runs finish, without any new scheduling state.
Draining changes a Dag's scheduling state, which the Dag update endpoint only allows for users who may edit the Dag. The run-creating endpoints check just the permission to create runs, so without this a user who may trigger runs but not pause the Dag could still stop its schedule by asking to drain it.
The backfill drain was committed with the backfill row, and the runs were created in a later transaction. When creating them failed, the Dag stayed draining although the request failed: runs already queued on the paused Dag started, and an active Dag stopped its schedule and ended up paused. Taking the drain in the transaction that creates the runs makes it commit or roll back with them, and its row lock keeps the drain finalizer from pausing the Dag before they exist. The trigger, backfill and materialize forms chose whether to drain from Dag data cached for up to five minutes, so a Dag unpaused elsewhere in the meantime could be drained and paused again. The drain warning also counted the runs of a paused backfill, which do not start. airflowctl sends drain_dag as false, never null, so the trigger body no longer accepts null.
"Run without resuming the schedule" read like an option on the new run alone, and could be read as "no scheduled runs resume either". What it actually does is drain the whole Dag, so runs already queued while the Dag was paused start as well. Naming the option "Drain" and stating that consequence first keeps it from surprising users, and matches the short-state-plus-detail format reviewers asked for across the paused-Dag options.
Unpausing or draining a paused Dag lets every unfinished run proceed: queued runs start, and runs that were already running resume their held tasks. Describing that as "queued runs start" understated the effect, and reviewers flagged the running-run case as the main surprise. Showing the count inside the options that release those runs, with all three options always visible, puts the consequence next to the choice instead of in a separate warning behind a collapsed section. The trigger modal already fetches the Dag when it opens, so its paused state now comes only from that fetch rather than being overridden halfway through the props from a value the caller may have cached.
Older servers reject newly serialized defaults, and failed drain requests leave users unable to retry with Keep paused. Cached Dag state and unfinished-run counts can also describe the wrong action. Backfill insertion can hold the Dag row longer than necessary. Drain documentation must distinguish runs held by paused backfills and avoid promising protection from the existing finalizer race.
ef46e73 to
65d0f89
Compare
|
Merging so it won't miss the 3.4 cut time. We can iterate over fixes if needed |
Backport failed to create: airflow-ctl/v0-1-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker a44fd6d airflow-ctl/v0-1-testThis should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
A paused Dag cannot execute an explicitly triggered run without changing its scheduling state. Unpausing also resumes the schedule, which is undesirable for Dags used to reprocess historical data on demand.
This change lets manual runs, backfills, and asset materializations put the Dag into the existing
drainingstate. Unfinished runs can start or resume, the scheduler creates no scheduled or asset-triggered runs, and the Dag returns to paused when its unfinished runs finish. Runs held by a paused backfill remain held; that backfill must be resumed for its runs to finish and draining to complete.The state change affects the whole Dag. A paused Dag still starts nothing: draining clears its paused state, without adding a per-run override or changing the scheduler's paused checks.
closes: #60768
POST /dags/{dag_id}/dagRuns,POST /backfills, andPOST /assets/{asset_id}/materializeacceptdrain_dag, defaulting tofalse. Draining requires Dag edit permission in addition to the existing creation permission. On an active Dag, it stops scheduling and leaves the Dag paused after its unfinished runs finish.airflowctl dags triggerandairflowctl backfill createexpose--drain-dag. Default fields are omitted from outgoing requests for compatibility with older strict API models, while required nullablelogical_dateis preserved.The drain state change commits or rolls back with run creation. Backfill creation retains its early backfill-row commit, then creates the runs before taking the Dag drain lock, keeping that lock out of the bulk insertion work. The scheduler's initializing-backfill guard covers this interval. The pre-existing finalizer race for a Dag that is already draining remains outside this change, as described in the review discussion.
Documentation, the existing drain release note, API descriptions, and generated clients describe the paused-backfill exception.
Validation: 56 affected UI tests and 27 airflowctl operation tests passed. PostgreSQL API/model/scheduler tests and rollback/cleanup tests passed; the independent-session lock tests passed on both PostgreSQL and MySQL. Regular and manual prek checks, including the production UI build, passed.
The radio-card layout:
These existing screenshots show the layout; the updated descriptions also explain that paused backfills stay paused.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5), Codex (GPT-6) following the guidelines