[kaxil]

Follow-up to #70923, which restored the coverage check for db clean. That check compares table names against the config, not the columns the config names, so it cannot catch this.

airflow db clean --dag-ids or --exclude-dag-ids silently purges nothing from three tables. Each names a dag_id column it does not have: task_reschedule.dag_id was dropped in 3.0.0, deadline.dag_id in 3.1.0, and asset_event has only ever had source_dag_id. The query fails on a missing column, run_cleanup suppresses a failing table into a warning, and the command exits 0. An operator who always scopes by Dag has been growing those three tables forever while cleanup reported success.

deadline and task_reschedule now reach their Dag the way the schema does, through dag_run and task_instance.

Why scope them rather than just drop the filter, which would be the smaller diff: deadline is cleaned as a dependent of dag_run precisely so its rows are archived before the ON DELETE CASCADE removes them. An unscoped pass would purge deadlines belonging to the very Dags whose runs --exclude-dag-ids is preserving, turning a silent no-op into silent data loss.

--exclude-dag-ids now also treats a NULL dag id as eligible. A row attributed to no Dag is not one of the excluded Dags' rows, but NOT IN alone yields NULL for it and retains it forever. That is every job row, since core never sets Job.dag_id, and it becomes reachable for asset_event and log too. This changes behaviour for those tables, in the direction the flag already documents.

The guard test is the part that stops this recurring. It only ever called run_cleanup without Dag filters, and the Dag filter is the only thing that dereferences these column names, so all three configs drifted unnoticed across two releases. It is now parametrized over the filter combinations and fails on each of them.

[ferruzzi]

Two PRs were already working on this, both are tagged "closes: #69030", and I've been working with those authors. #66350 has been open since 4 May and has been through several rounds of review. #71732 came in August and was cherry-picked into #66350 to avoid the two conflicting. Same three tables, same cause, same fix.

Your --exclude-dag-ids handling is a real catch, and broader than anything either PR had. But that's multiple months of working with a new contributor sidestepped when a search for those table names would have turned up both of those PRs.

Please don't close #66350. It also re-applies skip_if_referenced on the DELETE which fixes a race condition where a task_instance which points at a dag_version might appear between the archive INSERT and the DELETE, and it has a guard asserting every dag_id_column_name in config_dict is a real column of its table, to catch this class of config drift directly rather than through the filter path. Neither of those is in #73173, so I've asked @jakubmatyszewski to rescope his PR down to those two.

[kaxil]

Two PRs were already working on this, both are tagged "closes: #69030", and I've been working with those authors. #66350 has been open since 4 May and has been through several rounds of review. #71732 came in August and was cherry-picked into #66350 to avoid the two conflicting. Same three tables, same cause, same fix.

Your --exclude-dag-ids handling is a real catch, and broader than anything either PR had. But that's multiple months of working with a new contributor sidestepped when a search for those table names would have turned up both of those PRs.

Please don't close #66350. It also re-applies skip_if_referenced on the DELETE which fixes a race condition where a task_instance which points at a dag_version might appear between the archive INSERT and the DELETE, and it has a guard asserting every dag_id_column_name in config_dict is a real column of its table, to catch this class of config drift directly rather than through the filter path. Neither of those is in #73173, so I've asked @jakubmatyszewski to rescope his PR down to those two.

Commented on that PR too, thanks Dennis