Repository navigation
Prevent duplicate dag_id from overwriting serialized Dags - #68678
takayoshi-makabe wants to merge 5 commits into
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
4d52dec to
50806b9
Compare
|
How would this work if you move a dag to a different file? To make this situation not cause an import error, the dag processor must remove the previous entry, but how can the processor know the previous dag is gone before parsing the file? |
| return {dm.dag_id: dm for dm in session.scalars(stmt).unique()} | ||
|
|
||
| def find_dag_id_collisions(self, *, session: Session) -> dict[str, AirflowDagDuplicatedIdException]: | ||
| """Find Dag IDs already used by a different active DAG file.""" |
There was a problem hiding this comment.
| """Find Dag IDs already used by a different active DAG file.""" | |
| """Find Dag IDs already used by a different active Dag file.""" |
nit: "DAG file" -> "Dag file" to match the title-case convention used elsewhere
|
Thanks for the question. The concrete scenario I had in mind is: before: file_a.py → dag_id="hoge" after: file_a.py → dag_id="fuga" (hoge removed, fuga added) If file_b.py is parsed before file_a.py, this PR raises a transient import error on file_b.py since hoge → file_a.py is still active in the DB. However, this self-heals automatically: once file_a.py is re-parsed, hoge's last_parsed_time stops being updated, and DagProcessorManager._scan_stale_dags marks it stale after stale_dag_threshold seconds — so the next parse of file_b.py succeeds without collision. So the current implementation is already eventually consistent in periodic-parsing environments. What I'd rather avoid is silently letting the new file overwrite the existing entry — that's what current Airflow does today, and it's exactly how our team ended up with a broken DAG that nobody noticed. Making the collision visible as an import error, even transiently, is the whole point of this PR. If the transient error window is a concern, one option would be to proactively mark disappeared DAGs as stale inside update_dag_parsing_results_in_db at parse time, which would reduce the window to just the next parse of the original file. |
2d2f248 to
0e26ee8
Compare
This prevents a Dag parsed from one file from overwriting an existing active Dag with the same
dag_idfrom a different file.When the Dag processor parses files one at a time, duplicate
dag_idchecks inside a singleDagBagcan miss duplicates that live in separate files. Before this change, syncing the later parsed file to the metadata database could update the existingDagModeland serialized Dag row, effectively making the last parsed file win.This change adds a database-level collision check during Dag parsing result sync. If an active Dag with the same
dag_idalready exists for a different file, the incoming Dag is skipped and recorded as an import error using the existingAirflowDagDuplicatedIdExceptionbehavior.related: #29321
Tests
Added a regression test covering this flow:
original.pydag_idfromduplicate.pyDagModeland serialized Dag remain unchangedValidation run locally:
uv run --project airflow-core ruff format airflow-core/src/airflow/dag_processing/collection.py airflow-core/tests/unit/dag_processing/test_collection.pyuv run --project airflow-core ruff check --fix airflow-core/src/airflow/dag_processing/collection.py airflow-core/tests/unit/dag_processing/test_collection.pybreeze run pytest airflow-core/tests/unit/dag_processing/test_collection.py::TestUpdateDagParsingResults::test_duplicate_dag_id_from_different_file_is_import_error -xvsWas generative AI tooling used to co-author this PR?