Repository navigation
Fix airflow db clean never purging the callback table (#70923) - #72870
Closed
vatsrahul1001 wants to merge 1 commit into
Closed
vatsrahul1001 wants to merge 1 commit into
vatsrahul1001 wants to merge 1 commit into
Conversation
vincbeck
approved these changes
Sep 10, 2026
* Fix airflow db clean never purging the callback table The table was renamed from callback_request in Airflow 3.2.0 and the cleanup configuration was not updated with it. A configured table that does not exist is skipped with only a warning, so these rows were never deleted and the table grew without bound, with no way for an operator to purge it. Only callbacks that can no longer run are removed. A callback still awaiting execution owns its deadline row through an ON DELETE CASCADE foreign key, so deleting one would silently drop a deadline that has not fired yet. The sla_miss entry is dropped as well, since that table no longer exists in Airflow 3. * Purge orphaned partitioned_asset_key_log rows and restore the db clean coverage check The check that is meant to catch a metadata table being left out of airflow db clean looked for the models package at a path relative to the test file. That path stopped resolving when the sources moved under airflow-core, so the check examined no models at all and its assertions held trivially for an empty set. It has not been able to report a missing table since, which is how the callback table went unpurged for several releases. Walking the package's own search path instead keeps the check working wherever the sources live, and asserting that models were found stops it from passing on an empty set again. With the check restored, partitioned_asset_key_log is the one table it reports that genuinely has no way to be purged: it carries no foreign key, so its rows are left behind when the partition Dag run they describe is cascade-deleted with its dag_run. Only those orphans are deleted -- rows whose partition Dag run still exists are the evidence the scheduler evaluates to decide when that pending run fires, so they are kept regardless of age. The other uncovered tables are recorded in the exclusion list with the reason each is safe. * Purge orphaned scheduled callbacks while keeping unfired deadlines in db clean Every callback starts in SCHEDULED, a state outside both the active and the terminal sets, and an unfired deadline's callback stays there until the deadline is missed. Deny-listing active states would therefore purge those callbacks and, through the ON DELETE CASCADE foreign key, silently drop deadlines that have not fired. The finished states stay allow-listed, so an unknown state keeps its rows. A SCHEDULED callback is purged only once no deadline references it: deleting a Dag run cascades away its deadline at the database level, which leaves the callback behind forever. (cherry picked from commit 478745a)
vatsrahul1001
force-pushed
the
backport-478745af84-v3-3-test
branch
from
September 10, 2026 15:41
f6018bb to
8858278
Compare
Contributor
Author
|
Closing in favor of #72899 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #70923 to
v3-3-testfor the 3.3.2 release.Fixes
airflow db cleannever purging thecallbacktable (it was registered under the oldcallback_requestname, so the config entry named a non-existent table and rows were silently skipped). Thedb_cleanup.pyfix + newsfragment applied clean; the test's import conflicts resolved by taking the incoming sqlalchemy/mock supersets (ruff then droppedPath/call/IntegrityErroras unused on v3-3-test). All 14 callback-cleanup tests pass.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines