Repository navigation
DagRun.created_dag_version_id no longer matches its documented contract #71453
Description
Background
created_dag_version_id (airflow-core/src/airflow/models/dagrun.py:301-304) has existed as a column since AIP-65 DAG versioning (#42913/#43735). #49097 added the ORM relationship (created_dag_version) on top of it, with a docstring restating what the name already communicated:
"The id of the dag version column that was in effect at dag run creation time."
and the paired relationship docstring:
"The dag version that was active when the dag run was created, if available."
Documented or not, the name itself was always the contract — a write-once historical fact, not a live pointer. For a bundle-versioned (pinned) run, the version recorded at creation was — by design — the version that run would run forever. Per a conversation with the original author of AIP-65 DAG versioning: bundle-versioned runs were always meant to execute the version pinned at creation; the way to get "run with the latest code" behavior was to not use a versioned bundle at all (unpinned/local bundles always resolve to the latest DagVersion — see DBDagBag._version_from_dag_run, airflow-core/src/airflow/models/dagbag.py:210-216). There was never supposed to be a way to move a pinned run off the version it was created with.
What changed
Starting with #52177 (introduces the run_on_latest_version flag) and #54984 ("Run verify_integrity when cleared to run on latest"), and extended by #59764 and #65835/#66901, created_dag_version_id began being mutated after creation:
_update_dagrun_to_latest_version—airflow-core/src/airflow/models/taskinstance.py:344clear_task_instances(..., run_on_latest_version=True)—airflow-core/src/airflow/models/taskinstance.py:474and:497
An open PR, #71425, is currently extending this same mutation to running/queued DagRuns.
None of these renamed or re-documented the field. It still says "recorded at creation" while several code paths now treat it as "the version this run should currently be considered pinned to" — two different, and now conflated, meanings living in one column.
Why this matters
The mismatch between the documented contract and actual behavior has already produced concrete defects downstream, because other code correctly relied on the original contract and nothing flagged that the contract had changed underneath it:
- DagRun.dag_versions silently drops versions after a partial "run on latest version" clear #71454 —
DagRun.dag_versionssilently drops versions still in use by uncleared task instances after a partial "run on latest version" clear. - only_new clear can silently miss real new tasks after a prior "run on latest version" clear #71455 —
only_newclear (_get_new_task_ids) can report zero new tasks when there genuinely are some, for the same reason.
Both are symptoms, not the root cause. The root cause is that one field is now asked to represent two different things — an immutable historical fact, and a mutable "current pinned version" pointer — with no way for a reader to tell which one they're getting, or whether the run has been through a partial version bump that leaves it internally inconsistent.
What needs deciding
This is a design question as much as a naming one:
- Does "run on latest version" belong on bundle-versioned/pinned runs at all? The original design intent was that pinning is absolute, and "latest" is achieved by not pinning. If the answer is no, the mutation added since Run verify_integrity when cleared to run on latest #54984 should be reconsidered/reverted for the pinned case.
- If the capability is intentionally kept,
created_dag_version_idshould stop being overloaded: introduce a genuinely separate, clearly-named mutable field for "the version this pinned run is currently expected to run at" (e.g.pinned_dag_version_idorcurrent_dag_version_id), keepcreated_dag_version_idstrictly immutable as documented, and audit every consumer (_version_from_dag_run,get_dag_for_run_or_latest_version,DagRun.dag_versions,_get_new_task_ids, and anything else reading this field) to use the correct one.
Either way, the current state — one field, two incompatible meanings, no rename, no updated docs — should be resolved before more functionality (e.g. #71425) is built on top of it.
Drafted-by: Claude Code (Sonnet 5) (no human review before posting)
Activity
ephraimbuddy commentedon Aug 12, 2026
I agree with the core problem:
created_dag_version_idis documented as the version in effect when the Dag run was created, but the field is now used as the run's current version pointer.DagRun.bundle_versionmoves with it, so the two fields should be treated as one resolution mechanism.There are a few corrections that affect the proposed fix, though.
First, #71425 does not introduce this mutation for queued or running Dag runs.
mainalready updatescreated_dag_version_idandbundle_versionin that path; #65835 introduced it onmain, and #66901 backported it to 3.2.2. #71425 makes the existing behavior consistent: it moves a running task that becomesRESTARTING, runsverify_integrity, and applies the version update even whendag_run_state=False.Second, the #71455 repro as written does not reproduce on the finished-run path. That path already calls
verify_integrity, which creates the TI for newly added task C during the first clear. The subsequentonly_newresult is therefore correctly empty. There is a reachable variant on currentmain: the queued/running branch moves the run's pointer without callingverify_integrity, so the pointer can say latest while the corresponding TI does not exist. #71425 fixes that variant.Third, #71454 is valid and is not limited to the Grid chips.
DagRun.dag_versionsalso feeds the publicDAGRunResponseandDagRun.version_number. That should be fixed independently by reporting the versions represented by the run's TI/TIH rows instead of assuming a bundle-versioned run can only contain the run's current pointer.On the two design questions:
Keep run-on-latest for versioned bundles. It has been a released user-facing capability since 3.1.0 and now exists across the clear UI/API, backfills,
[core] rerun_with_latest_version, and the Dag-level parameter. The API still marks it experimental, but removing the capability would nevertheless be a separate user-visible design and compatibility decision; it should not be folded into Respect run on latest version when clearing a running Dag run #71425. Disabling bundle versioning is not equivalent: users can reasonably want runs pinned by default while deliberately rerunning a failed task against fixed code.Do not add a second column without a concrete provenance consumer. The execution-path consumers I found use this value as the version the run currently resolves to. Reporting consumers such as
dag_versions, and task-existence logic such as_get_new_task_ids, should derive their answers from TI/TIH rows instead.An additional immutable column would add a migration, dual-write rules, and a new choice at every reader. It would also not restore the documented contract for existing rows reliably:
_update_dagrun_to_latest_versioncan rewrite all current TI version ids without creating TI history, so the original creation version may already be unrecoverable. If immutable creation-time provenance has a real consumer, we should define its retention and upgrade semantics first. I could not find such a consumer in the current code.My suggested split is:
created_dag_version_idas the Dag version the run currently resolves to, initially set at creation and updated when the run is explicitly moved to latest. Discuss a physical rename separately for 3.4._get_new_task_idsto compare the latest Dag's task ids with the run's existing TI rows. That directly answers which tasks are missing and removes its dependency on the version-pointer invariant.I am open to a separate immutable field if there is a concrete consumer for creation-time provenance, but I do not think the current issue establishes one.
I agree with the core problem:
created_dag_version_idis documented as the version in effect when the Dag run was created, but the field is now used as the run's current version pointer.DagRun.bundle_versionmoves with it, so the two fields should be treated as one resolution mechanism.There are a few corrections that affect the proposed fix, though.
First, #71425 does not introduce this mutation for queued or running Dag runs.
mainalready updatescreated_dag_version_idandbundle_versionin that path; #65835 introduced it onmain, and #66901 backported it to 3.2.2. #71425 makes the existing behavior consistent: it moves a running task that becomesRESTARTING, runsverify_integrity, and applies the version update even whendag_run_state=False.Second, the #71455 repro as written does not reproduce on the finished-run path. That path already calls
verify_integrity, which creates the TI for newly added task C during the first clear. The subsequentonly_newresult is therefore correctly empty. There is a reachable variant on currentmain: the queued/running branch moves the run's pointer without callingverify_integrity, so the pointer can say latest while the corresponding TI does not exist. #71425 fixes that variant.Third, #71454 is valid and is not limited to the Grid chips.
DagRun.dag_versionsalso feeds the publicDAGRunResponseandDagRun.version_number. That should be fixed independently by reporting the versions represented by the run's TI/TIH rows instead of assuming a bundle-versioned run can only contain the run's current pointer.On the two design questions:
Keep run-on-latest for versioned bundles. It has been a released user-facing capability since 3.1.0 and now exists across the clear UI/API, backfills,
[core] rerun_with_latest_version, and the Dag-level parameter. The API still marks it experimental, but removing the capability would nevertheless be a separate user-visible design and compatibility decision; it should not be folded into Respect run on latest version when clearing a running Dag run #71425. Disabling bundle versioning is not equivalent: users can reasonably want runs pinned by default while deliberately rerunning a failed task against fixed code.Do not add a second column without a concrete provenance consumer. The execution-path consumers I found use this value as the version the run currently resolves to. Reporting consumers such as
dag_versions, and task-existence logic such as_get_new_task_ids, should derive their answers from TI/TIH rows instead.An additional immutable column would add a migration, dual-write rules, and a new choice at every reader. It would also not restore the documented contract for existing rows reliably:
_update_dagrun_to_latest_versioncan rewrite all current TI version ids without creating TI history, so the original creation version may already be unrecoverable. If immutable creation-time provenance has a real consumer, we should define its retention and upgrade semantics first. I could not find such a consumer in the current code.My suggested split is:
created_dag_version_idas the Dag version the run currently resolves to, initially set at creation and updated when the run is explicitly moved to latest. Discuss a physical rename separately for 3.4._get_new_task_idsto compare the latest Dag's task ids with the run's existing TI rows. That directly answers which tasks are missing and removes its dependency on the version-pointer invariant.I am open to a separate immutable field if there is a concrete consumer for creation-time provenance, but I do not think the current issue establishes one.
Drafted-by: Codex (5.6 Sol); reviewed by @ephraimbuddy before posting