Skip to content

AIP-65: Add DAG versioning support#42913

Merged
ephraimbuddy merged 49 commits intoapache:mainfrom
astronomer:versioned-dag2
Nov 5, 2024
Merged

AIP-65: Add DAG versioning support#42913
ephraimbuddy merged 49 commits intoapache:mainfrom
astronomer:versioned-dag2

Conversation

@ephraimbuddy
Copy link
Copy Markdown
Contributor

@ephraimbuddy ephraimbuddy commented Oct 10, 2024 •
edited
Loading

This commit introduces versioning for DAGs.

Changes:

  • Introduced DagVersion model to handle versioning of DAGs.
  • Added version_name field to DAG for use in tracking the dagversion by users
  • Modified DAG execution logic to reference dag_version_id instead of the dag_hash to ensure DAG runs are linked to specific versions.

The table relations:
Screenshot 2024-10-25 at 09 21 19

The versioning is based on the serialized dict changing. If a dag's serialized dict changes, a new serialized dag will be registered based on the hash diference, and consequently, a new dag version and dag code. The link from dag_version to TI is because of TaskInstance clearing. It helps us retain the previous dag version the task ran with.

Closes: #42333, #42334, #42336

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:db-migrations PRs with DB migration area:dev-tools area:Scheduler including HA (high availability) scheduler area:serialization area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues kind:documentation labels Oct 10, 2024
jedcunningham
jedcunningham reviewed Oct 14, 2024
View reviewed changes
This was referenced Oct 15, 2024
Closed
Closed
@ephraimbuddy ephraimbuddy force-pushed the versioned-dag2 branch 5 times, most recently from ad4f57c to dd272b0 Compare October 16, 2024 12:27
@ephraimbuddy ephraimbuddy added the legacy api Whether legacy API changes should be allowed in PR label Oct 16, 2024
@ephraimbuddy ephraimbuddy force-pushed the versioned-dag2 branch 3 times, most recently from eb13cdd to 1b81f2f Compare October 16, 2024 13:27
@ephraimbuddy ephraimbuddy marked this pull request as ready for review October 16, 2024 13:27
ephraimbuddy and others added 23 commits November 5, 2024 11:26
@ephraimbuddy
fix test
72f2cc0
@ephraimbuddy
fixup! fix test
903ed3c
@ephraimbuddy
use binary=False for uuid field to fix sqlite issue
db44170
@ephraimbuddy
apply suggestions from code review
c4ed8af
@ephraimbuddy
Remove unnecessary version_name on dagmodel
4f011d2
@ephraimbuddy
Fix sqlalchemy 2 warning
4bf8b1f
@ephraimbuddy
Fix conflicts
0e0c127
@ephraimbuddy @jedcunningham
Apply suggestions from code review
e410f2c 
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Apply suggestions from code review
76982f3
@ephraimbuddy
fixup! fixup! Apply suggestions from code review
c4171cb
@ephraimbuddy
add test for dagversion model and make version_name, number and dag_i…
dad0f36 
…d unique
@ephraimbuddy
Remove commented test as serdag can no longer disappear
13777e7
@ephraimbuddy
Add SQLAlchemy-utils to requirements
9f96cb0
@ephraimbuddy
mark test_dag_version.py as db_test
b2c4844
@ephraimbuddy
make version_name nullable
611a5cb
@ephraimbuddy
Apply suggestions from code review
cc41fb6
@ephraimbuddy
fixup! Apply suggestions from code review
621a407
@ephraimbuddy
remove file_updater
108965b
@ephraimbuddy
Use dag_version for creating dagruns instead of dag_version_id
a51585e
@ephraimbuddy
fix conflicts
01f3c58
@ephraimbuddy
use if TYPE_CHECKING
11de1e9
@ephraimbuddy
Add docstrings to methods
569eb53
@ephraimbuddy
Move getting latest serdags to SerializedDagModel
a7bba02
@ephraimbuddy ephraimbuddy requested a review from uranusjr November 5, 2024 10:58
uranusjr
uranusjr approved these changes Nov 5, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good enough

@ephraimbuddy ephraimbuddy merged commit 1116f28 into apache:main Nov 5, 2024
@ephraimbuddy ephraimbuddy deleted the versioned-dag2 branch November 5, 2024 14:19
potiuk added a commit that referenced this pull request Nov 6, 2024
@potiuk
Revert "AIP-65: Add DAG versioning support (#42913)"
37c22b4 
This reverts commit 1116f28.
@potiuk potiuk mentioned this pull request Nov 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashb ashb ashb left review comments

@kaxil kaxil kaxil left review comments

@dstandish dstandish dstandish left review comments

@jscheffl jscheffl jscheffl left review comments

@uranusjr uranusjr uranusjr approved these changes

@jedcunningham jedcunningham jedcunningham approved these changes

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton

@bbovenzi bbovenzi Awaiting requested review from bbovenzi

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

Assignees

No one assigned

Labels

AIP-65: DAG history in UI area:API Airflow's REST/HTTP API area:CLI area:db-migrations PRs with DB migration area:dev-tools area:Scheduler including HA (high availability) scheduler area:serialization area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues kind:documentation legacy api Whether legacy API changes should be allowed in PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Calculate and track DAG version

7 participants

@ephraimbuddy @kaxil @dstandish @ashb @uranusjr @jedcunningham @jscheffl