Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion airflow-core/docs/core-concepts/backfill.rst
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,12 @@ For UI, follow the following steps:
- **Max active runs**: limit concurrent backfill runs for this backfill.
- **Run backwards**: execute most recent intervals first.
- **Advanced Config**: optionally provide JSON ``dag_run.conf``.
- If the Dag is paused, you can **Unpause** it in the same window.
- If the Dag is paused, choose under **Dag is paused** what happens to the Dag: **Unpause** (the
default) makes the Dag active and resumes its schedule; **Drain** lets the unfinished Dag runs,
including the backfill's, start or resume without creating scheduled ones, and pauses the Dag again once
they finish (see :ref:`concepts:dag-pausing`); **Keep paused** leaves the backfill runs queued until the
Dag is unpaused. Both **Unpause** and **Drain** show how many unfinished Dag runs will start or resume.
Paused backfills remain paused under all three choices.

.. image:: ../img/ui-light/backfill.png
:alt: Backfill pop-up window (Light Mode)
Expand Down
13 changes: 13 additions & 0 deletions airflow-core/docs/core-concepts/dags.rst
Original file line number Diff line number Diff line change
Expand Up @@ -796,6 +796,8 @@ with different data intervals. These dependencies are calculated by the schedule
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`

.. _concepts:dag-pausing:

Dag pausing, deactivation and deletion
--------------------------------------

Expand All @@ -813,6 +815,17 @@ the scheduler automatically changes the Dag to paused. A backfill started while
the drain until its Dag runs have been created and finished. While a Dag is draining, you can cancel the drain
to make the Dag active again.

A manual run, backfill or asset materialization on a paused Dag can drain the Dag instead of unpausing it.
Choose **Drain** under **Dag is paused** in the UI form, or set ``drain_dag`` to ``true`` in the REST API
request. This changes the whole Dag, not only the new run: unfinished Dag runs can start or resume, except
those held by paused backfills. Paused backfills remain paused, and their unfinished runs prevent draining
from completing until those backfills are resumed and their runs finish. The scheduler creates no scheduled
or asset-triggered Dag runs, and the Dag pauses again once all unfinished runs finish. The Dag starts draining
in the same transaction that creates the run, so if the run is rejected, the Dag stays paused. Setting ``drain_dag`` on an
active Dag drains it the same way, so the Dag also ends up paused. Because it changes the Dag's scheduling
state, ``drain_dag`` requires the same permission as pausing the Dag, in addition to the permission to create
the run.

Dags can be deactivated (do not confuse it with ``Active`` tag in the UI) by removing them from the
``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER`` and misses the Dag that it had seen
before and stored in the database it will set is as deactivated. The metadata and history of the
Expand Down
10 changes: 9 additions & 1 deletion airflow-core/newsfragments/72407.significant.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Gracefully pause Dags by draining active runs (#72407, #73225, #73055)
Gracefully pause Dags by draining active runs (#72407, #73225, #73055, #73763)

Airflow now supports a ``draining`` scheduling state for Dags. Draining stops
the scheduler from creating new scheduled or asset-triggered Dag runs while
Expand All @@ -8,6 +8,14 @@ runs remain allowed.
After outstanding Dag runs finish, the Dag automatically transitions to
``paused``. Users can cancel draining to return the Dag to ``active``.

Manual runs, backfills and asset materializations on a paused Dag can drain
the Dag instead of unpausing it: choose **Drain** in the UI, or set
``drain_dag`` in the REST API request. Draining lets unfinished runs of
the Dag start or resume, except those held by paused backfills, creates no
scheduled runs, and pauses the Dag again once all unfinished runs finish.
Paused backfills remain paused and must be resumed for their runs to finish
and draining to complete.

The UI, REST API, and ``airflowctl`` expose the ``active``, ``draining``, and
``paused`` scheduling states, and the Dags list can pause, drain, or unpause
several selected Dags at once. Existing ``is_paused`` API usage remains
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,14 @@ class BackfillPostBody(StrictBaseModel):
"then the ``[core] rerun_with_latest_version`` config option, "
"and finally ``True`` (the historical default for backfills).",
)
drain_dag: bool = Field(
default=False,
description="Drain the Dag together with the backfill. Draining changes the whole Dag: unfinished "
"runs can start or resume except those held by paused backfills. No scheduled runs are created, "
"and the Dag pauses once all unfinished runs finish. Paused backfills must be resumed for draining "
"to complete. Requires the same permission as pausing the Dag. "
"Ignored by the dry-run endpoint.",
)


class BackfillResponse(BaseModel):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,13 @@ class TriggerDAGRunPostBody(StrictBaseModel):
note: str | None = None
partition_key: str | None = None
bundle_version: str | None = None
drain_dag: bool = Field(
default=False,
description="Drain the Dag together with this run. Draining changes the whole Dag: unfinished "
"runs can start or resume except those held by paused backfills. No scheduled runs are created, "
"and the Dag pauses once all unfinished runs finish. Paused backfills must be resumed for draining "
"to complete. Requires the same permission as pausing the Dag.",
)

@model_validator(mode="after")
def check_data_intervals(self):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12008,6 +12008,16 @@ components:
run. If not specified, falls back to the DAG-level ``rerun_with_latest_version``
parameter, then the ``[core] rerun_with_latest_version`` config option,
and finally ``True`` (the historical default for backfills).
drain_dag:
type: boolean
title: Drain Dag
description: 'Drain the Dag together with the backfill. Draining changes
the whole Dag: unfinished runs can start or resume except those held by
paused backfills. No scheduled runs are created, and the Dag pauses once
all unfinished runs finish. Paused backfills must be resumed for draining
to complete. Requires the same permission as pausing the Dag. Ignored
by the dry-run endpoint.'
default: false
additionalProperties: false
type: object
required:
Expand Down Expand Up @@ -15723,6 +15733,15 @@ components:
- type: string
- type: 'null'
title: Bundle Version
drain_dag:
type: boolean
title: Drain Dag
description: 'Drain the Dag together with this run. Draining changes the
whole Dag: unfinished runs can start or resume except those held by paused
backfills. No scheduled runs are created, and the Dag pauses once all
unfinished runs finish. Paused backfills must be resumed for draining
to complete. Requires the same permission as pausing the Dag.'
default: false
additionalProperties: false
type: object
title: MaterializeAssetBody
Expand Down Expand Up @@ -17379,6 +17398,15 @@ components:
- type: string
- type: 'null'
title: Bundle Version
drain_dag:
type: boolean
title: Drain Dag
description: 'Drain the Dag together with this run. Draining changes the
whole Dag: unfinished runs can start or resume except those held by paused
backfills. No scheduled runs are created, and the Dag pauses once all
unfinished runs finish. Paused backfills must be resumed for draining
to complete. Requires the same permission as pausing the Dag.'
default: false
additionalProperties: false
type: object
required:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,16 @@ def materialize_asset(
)

params = body.validate_context(context_dag)
if body.drain_dag:
if not get_auth_manager().is_authorized_dag(
method="PUT",
details=DagDetails(id=dag_id, team_name=DagModel.get_team_name(dag_id, session=session)),
user=user,
):
raise HTTPException(
status.HTTP_403_FORBIDDEN, f"Draining requires permission to edit Dag: {dag_id}"
)
DagModel.start_drain(dag_id, session=session)
return dag.create_dagrun(
run_id=params["run_id"],
logical_date=params["logical_date"],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@
from sqlalchemy.orm import joinedload

from airflow._shared.timezones import timezone
from airflow.api_fastapi.app import get_auth_manager
from airflow.api_fastapi.auth.managers.models.resource_details import DagDetails
from airflow.api_fastapi.common.dagbag import resolve_run_on_latest_version
from airflow.api_fastapi.common.db.common import (
SessionDep,
Expand All @@ -51,7 +53,7 @@
)
from airflow.api_fastapi.logging.decorators import action_logging
from airflow.exceptions import DagNotFound, DagRunTypeNotAllowed
from airflow.models import DagRun
from airflow.models import DagModel, DagRun
from airflow.models.backfill import (
AlreadyRunningBackfill,
Backfill,
Expand Down Expand Up @@ -302,6 +304,18 @@ def create_backfill(
session,
fallback=True,
)
if backfill_request.drain_dag and not get_auth_manager().is_authorized_dag(
method="PUT",
details=DagDetails(
id=backfill_request.dag_id,
team_name=DagModel.get_team_name(backfill_request.dag_id, session=session),
),
user=user,
):
raise HTTPException(
status.HTTP_403_FORBIDDEN,
f"Draining requires permission to edit Dag: {backfill_request.dag_id}",
)
try:
backfill_obj = _create_backfill(
dag_id=backfill_request.dag_id,
Expand All @@ -313,6 +327,7 @@ def create_backfill(
triggering_user_name=user.get_display_name(),
reprocess_behavior=backfill_request.reprocess_behavior,
run_on_latest_version=resolved_run_on_latest,
drain_dag=backfill_request.drain_dag,
)
return BackfillResponse.model_validate(backfill_obj)
except OperationalError as e:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -817,6 +817,17 @@ def trigger_dag_run(
f"DAG with dag_id: '{dag_id}' does not support bundle versioning",
)

if body.drain_dag:
if not get_auth_manager().is_authorized_dag(
method="PUT",
details=DagDetails(id=dag_id, team_name=DagModel.get_team_name(dag_id, session=session)),
user=user,
):
raise HTTPException(
status.HTTP_403_FORBIDDEN, f"Draining requires permission to edit Dag: {dag_id}"
)
DagModel.start_drain(dag_id, session=session)

dag_run = dag.create_dagrun(
run_id=params["run_id"],
logical_date=params["logical_date"],
Expand Down
3 changes: 3 additions & 0 deletions airflow-core/src/airflow/models/backfill.py
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,7 @@ def _create_backfill(
triggering_user_name: str | None,
reprocess_behavior: ReprocessBehavior | None = None,
run_on_latest_version: bool = False,
drain_dag: bool = False,
) -> Backfill:
from airflow.models import DagModel
from airflow.models.serialized_dag import SerializedDagModel
Expand Down Expand Up @@ -726,6 +727,8 @@ def _create_backfill(
run_on_latest_version=run_on_latest_version,
session=session,
)
if drain_dag:
DagModel.start_drain(dag_id, session=session)
except OperationalError as e:
if is_lock_not_available_error(e):
# Lock error: clean up the orphan so the user can retry. The
Expand Down
16 changes: 16 additions & 0 deletions airflow-core/src/airflow/models/dag.py
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,22 @@ def set_scheduling_state(self, state: DagSchedulingState) -> None:
self.is_paused = state == DagSchedulingState.PAUSED
self.is_draining = state == DagSchedulingState.DRAINING

@classmethod
def start_drain(cls, dag_id: str, *, session: Session) -> None:
"""
Put the Dag into the draining state.

Lock the Dag row and change its scheduling state in the caller's transaction.
Call this in the transaction that creates the explicit runs so the state change
commits or rolls back with them.
"""
dag_model = session.scalars(
with_row_locks(
select(cls).where(cls.dag_id == dag_id), of=cls, session=session
).execution_options(populate_existing=True)
).one()
dag_model.set_scheduling_state(DagSchedulingState.DRAINING)

def is_rollup_asset(self, *, name: str, uri: str) -> bool:
"""
Return whether the asset identified by *name*/*uri* uses a rollup mapper.
Expand Down
18 changes: 18 additions & 0 deletions airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -955,6 +955,12 @@ export const $BackfillPostBody = {
],
title: 'Run On Latest Version',
description: 'Run on the latest bundle version of the Dag for each backfilled run. If not specified, falls back to the DAG-level ``rerun_with_latest_version`` parameter, then the ``[core] rerun_with_latest_version`` config option, and finally ``True`` (the historical default for backfills).'
},
drain_dag: {
type: 'boolean',
title: 'Drain Dag',
description: 'Drain the Dag together with the backfill. Draining changes the whole Dag: unfinished runs can start or resume except those held by paused backfills. No scheduled runs are created, and the Dag pauses once all unfinished runs finish. Paused backfills must be resumed for draining to complete. Requires the same permission as pausing the Dag. Ignored by the dry-run endpoint.',
default: false
}
},
additionalProperties: false,
Expand Down Expand Up @@ -6502,6 +6508,12 @@ export const $MaterializeAssetBody = {
}
],
title: 'Bundle Version'
},
drain_dag: {
type: 'boolean',
title: 'Drain Dag',
description: 'Drain the Dag together with this run. Draining changes the whole Dag: unfinished runs can start or resume except those held by paused backfills. No scheduled runs are created, and the Dag pauses once all unfinished runs finish. Paused backfills must be resumed for draining to complete. Requires the same permission as pausing the Dag.',
default: false
}
},
additionalProperties: false,
Expand Down Expand Up @@ -9032,6 +9044,12 @@ export const $TriggerDAGRunPostBody = {
}
],
title: 'Bundle Version'
},
drain_dag: {
type: 'boolean',
title: 'Drain Dag',
description: 'Drain the Dag together with this run. Draining changes the whole Dag: unfinished runs can start or resume except those held by paused backfills. No scheduled runs are created, and the Dag pauses once all unfinished runs finish. Paused backfills must be resumed for draining to complete. Requires the same permission as pausing the Dag.',
default: false
}
},
additionalProperties: false,
Expand Down
12 changes: 12 additions & 0 deletions airflow-core/src/airflow/ui/openapi-gen/requests/types.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,10 @@ export type BackfillPostBody = {
* Run on the latest bundle version of the Dag for each backfilled run. If not specified, falls back to the DAG-level ``rerun_with_latest_version`` parameter, then the ``[core] rerun_with_latest_version`` config option, and finally ``True`` (the historical default for backfills).
*/
run_on_latest_version?: boolean | null;
/**
* Drain the Dag together with the backfill. Draining changes the whole Dag: unfinished runs can start or resume except those held by paused backfills. No scheduled runs are created, and the Dag pauses once all unfinished runs finish. Paused backfills must be resumed for draining to complete. Requires the same permission as pausing the Dag. Ignored by the dry-run endpoint.
*/
drain_dag?: boolean;
};

/**
Expand Down Expand Up @@ -1728,6 +1732,10 @@ export type MaterializeAssetBody = {
note?: string | null;
partition_key?: string | null;
bundle_version?: string | null;
/**
* Drain the Dag together with this run. Draining changes the whole Dag: unfinished runs can start or resume except those held by paused backfills. No scheduled runs are created, and the Dag pauses once all unfinished runs finish. Paused backfills must be resumed for draining to complete. Requires the same permission as pausing the Dag.
*/
drain_dag?: boolean;
};

/**
Expand Down Expand Up @@ -2273,6 +2281,10 @@ export type TriggerDAGRunPostBody = {
note?: string | null;
partition_key?: string | null;
bundle_version?: string | null;
/**
* Drain the Dag together with this run. Draining changes the whole Dag: unfinished runs can start or resume except those held by paused backfills. No scheduled runs are created, and the Dag pauses once all unfinished runs finish. Paused backfills must be resumed for draining to complete. Requires the same permission as pausing the Dag.
*/
drain_dag?: boolean;
};

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
"title": "تم إنشاء التعبئة الرجعية"
}
},
"unpause": "إلغاء إيقاف {{dag_display_name}} عند التشغيل",
"validation": {
"datesRequired": "يجب توفير بيانات كل من تاريخ بدء فترة وتاريخ الانتهاء.",
"startBeforeEnd": "بيانات فترة تاريخ البدء يجب ان تكون من او قبل تاريخ الانتهاء. "
Expand Down Expand Up @@ -195,8 +194,7 @@
"title": "تم تشغيل Dag"
}
},
"triggerAgainWithConfig": "تشغيل مرة أخرى باستخدام هذا التكوين",
"unpause": "إلغاء إيقاف {{dagDisplayName}} عند التشغيل"
"triggerAgainWithConfig": "تشغيل مرة أخرى باستخدام هذا التكوين"
},
"trimText": {
"details": "التفاصيل",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
"title": "Reompliment generat"
}
},
"unpause": "Reactivar {{dag_display_name}} a l'executar",
"validation": {
"datesRequired": "S'ha de proporcionar tant la data d'inici com la data de finalització.",
"startBeforeEnd": "La data d'inici de l'interval de dates ha de ser anterior o igual a la data de finalització de l'interval de dates."
Expand Down Expand Up @@ -175,8 +174,7 @@
"title": "Execució del Dag correcta"
}
},
"triggerAgainWithConfig": "Tornar a executar amb aquesta configuració",
"unpause": "Reactivar {{dagDisplayName}} a l'executar"
"triggerAgainWithConfig": "Tornar a executar amb aquesta configuració"
},
"trimText": {
"details": "Detalls",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
"title": "Auffüllung gestartet"
}
},
"unpause": "Dag {{dag_display_name}} beim Start der Auffüllung aktiv schalten",
"validation": {
"datesRequired": "Sowohl Start- als auch Enddatum müssen angegeben werden.",
"startBeforeEnd": "Das Startdatum muss vor oder auf dem Enddatum liegen."
Expand Down Expand Up @@ -176,8 +175,7 @@
"title": "Dag-Lauf ausgelöst"
}
},
"triggerAgainWithConfig": "Erneut mit dieser Konfiguration auslösen",
"unpause": "Dag {{dagDisplayName}} beim Auslösen des Laufes aktiv schalten"
"triggerAgainWithConfig": "Erneut mit dieser Konfiguration auslösen"
},
"trimText": {
"details": "Details",
Expand Down
Loading
Loading