Skip to content

Include task_reschedule in task_instance dependent tables for db clean - #71990

Merged
ferruzzi merged 4 commits into
apache:mainfrom
Alwaysgaurav1:fix-db-clean-airflow3-tables
Sep 22, 2026
Merged

ferruzzi merged 4 commits into
apache:mainfrom
Alwaysgaurav1:fix-db-clean-airflow3-tables

Conversation

@Alwaysgaurav1

@Alwaysgaurav1 Alwaysgaurav1 commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

In Airflow 3, task_reschedule rows reference task_instance via the ti_id foreign key with CASCADE delete. When running airflow db clean on task_instance, task_reschedule rows must be archived and cleaned prior to task instance removal.

This PR:

  • Adds "task_reschedule" to task_instance.dependent_tables in airflow.utils.db_cleanup so that dependent reschedules are cleaned and archived before parent task instances.
  • Adds test coverage in TestTaskRescheduleCleanup (test_db_cleanup.py) verifying both direct task_reschedule cleanup and cascading cleanup/archival when task_instance is cleaned.
  • Adds newsfragment 71990.bugfix.rst.

Testing

  • Ran unit tests:
    uv run --project airflow-core pytest airflow-core/tests/unit/utils/test_db_cleanup.py -k TestTaskRescheduleCleanup

@ferruzzi ferruzzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a good bit of overlap with #70923, which registers the same table with a guard and repairs the same test_no_models_missing bug. If #70923 lands first, then this PR can be reduced to the task_reschedule and asset_event fixes (plus their tests).

Committer: Please make sure that other PR is merged first and this gets rebased on top

@Alwaysgaurav1
Alwaysgaurav1 force-pushed the fix-db-clean-airflow3-tables branch from 8003fa9 to 907b127 Compare September 10, 2026 08:28
@Alwaysgaurav1

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @ferruzzi!

Now that #70923 has landed on main, I have rebased this branch on top of latest main.

I've removed the duplicate callback table config and model discovery adjustments from this PR so that it is now cleanly focused on:

  1. task_reschedule: Removing obsolete dag_id_column_name="dag_id" and adding "task_reschedule" to task_instance.dependent_tables (aligning with migration 0063's transition to ti_id).
  2. asset_event: Updating dag_id_column_name to "source_dag_id".
  3. Unit tests covering both task_reschedule and asset_event cleanups.

Ready for another look!

@Alwaysgaurav1
Alwaysgaurav1 force-pushed the fix-db-clean-airflow3-tables branch from 46db6fa to 692dde7 Compare September 10, 2026 19:23
@ferruzzi

ferruzzi commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

@Alwaysgaurav1 - Needs another rebase, sorry. There are several PRs all poking in the same area with overlapping fixes and I'm trying to merge these in a way that minimizes the rebases, but some can't be helped. #73173 and #70923 stole a bit of your thunder by already merging part of what you did, but there is some work here that nobody else has caught though, and it's worth saving if you are willing to trim this down again.

You added task_reschedule to the dependent_tables for task_instance and that's still valuable.

In Airflow 3, task_reschedule rows reference task_instance via ti_id foreign key with CASCADE delete. When cleaning task_instance rows via airflow db clean, task_reschedule rows must be cleaned and archived before task_instance rows are removed.

- Add task_reschedule to task_instance.dependent_tables in airflow.utils.db_cleanup.
- Add test coverage verifying task_reschedule cleanup and archiving when task_instance is cleaned.
@Alwaysgaurav1
Alwaysgaurav1 force-pushed the fix-db-clean-airflow3-tables branch from 692dde7 to f54930e Compare September 18, 2026 06:21
@Alwaysgaurav1

Copy link
Copy Markdown
Contributor Author

Thanks for the guidance, @ferruzzi!

I have rebased onto latest main and trimmed the PR down to focus specifically on what was missing:

  • Added task_reschedule to task_instance.dependent_tables in airflow.utils.db_cleanup so that reschedules are properly cleaned and archived prior to task instance removal.
  • Added unit tests in TestTaskRescheduleCleanup to verify direct cleanup as well as dependent cleanup and archive table creation when cleaning task_instance.
  • Updated the newsfragment and commit message accordingly.

Ready for another look!

@eladkal eladkal added this to the Airflow 3.4.0 milestone Sep 18, 2026
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Sep 18, 2026

@ferruzzi ferruzzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @Alwaysgaurav1 . I appreciate your patience while we sorted that out.

Approved pending CI success.

@ferruzzi

Copy link
Copy Markdown
Contributor

Actually, one small last request: can you trim down the PR description to reflect what's left in the PR?

@ferruzzi ferruzzi changed the title Fix database cleanup table configurations for Airflow 3 Include task_reschedule in task_instance dependent tables for db clean Sep 19, 2026
@Alwaysgaurav1

Copy link
Copy Markdown
Contributor Author

Done! I've trimmed down the description to focus specifically on the task_reschedule dependent table addition and its tests. Thanks for catching that, @ferruzzi!

@ferruzzi
ferruzzi merged commit 6f7e635 into apache:main Sep 22, 2026
155 checks passed
@boring-cyborg

boring-cyborg Bot commented Sep 22, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants