Skip to content

fix(migrations): move UPDATEs inside disable_sqlite_fkeys in migration 0097#64876

Open
Lee-W wants to merge 1 commit intoapache:mainfrom
astronomer:fix-migration
Open

fix(migrations): move UPDATEs inside disable_sqlite_fkeys in migration 0097#64876
Lee-W wants to merge 1 commit intoapache:mainfrom
astronomer:fix-migration

Conversation

@Lee-W
Copy link
Copy Markdown
Member

@Lee-W Lee-W commented Apr 8, 2026

Why

Migration 0097 failed on SQLite with FOREIGN KEY constraint failed when running DROP TABLE dag. PRAGMA foreign_keys=off (issued inside disable_sqlite_fkeys) is silently ignored when a transaction is already active — SQLite limitation. The two UPDATE statements before disable_sqlite_fkeys triggered SQLAlchemy's autobegin, starting a transaction before the PRAGMA ran.

How to reproduce

  1. Bring up a 3.1.8 instance
breeze start-airflow  --backend sqlite --load-example-dags --use-airflow-version 3.1.8
  1. Run a few Dags randomly. We need rows in the Dag run and task instance (or whatever rows that references Dag using FK)

  2. Go to the Shell tab in the breeze terminal

  3. Run

uv pip install apache-airflow==3.2.0
uv run airflow db migrate

What

Move the UPDATE statements inside disable_sqlite_fkeys so PRAGMA foreign_keys=off executes before autobegin opens a transaction.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W Lee-W requested a review from ephraimbuddy as a code owner April 8, 2026 02:40
@boring-cyborg boring-cyborg bot added the area:db-migrations PRs with DB migration label Apr 8, 2026
@Lee-W Lee-W requested a review from vatsrahul1001 April 8, 2026 03:28
@Lee-W
Copy link
Copy Markdown
Member Author

Lee-W commented Apr 8, 2026

@vatsrahul1001 It looks like we used the wrong approach for testing, and I applied the wrong constraint during RC1 testing. I didn't have time to test it on RC2 until it was released yesterday 🤦‍♂️

After running the following command we discussed, I found out there's no dag_run in the DB.

breeze start-airflow  --backend sqlite --load-example-dags --use-airflow-version 3.1.8
breeze start-airflow  --backend sqlite --load-example-dags

So here's what I do for verifying this PR is correct (how to reproduce was detailed in the description)

Step 1

breeze start-airflow  --backend sqlite --load-example-dags --use-airflow-version 3.1.8 --terminal-multiplexer tmux

Step 2

Run a few Dags randomly. We need rows in the Dag run and task instance (or whatever rows that reference Dag using FK)

Step 3

Run Ctrl-C/Cmd-C on all airflow components in breeze but don't run stop_airflow and don't exit tmux

Step 4

git chechout to v3.2.0

Step 5

Go into the breeze terminal again

airflow db migrate

Step 6

Restart all the components in breeze (use the previous command with ↑)

@Lee-W Lee-W added the backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch label Apr 8, 2026
Copy link
Copy Markdown
Collaborator

@Dev-iL Dev-iL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we modify the migration test (or add a new regression test) to capture this sort of issues?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrations PRs with DB migration backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants