Re-introducing --use-migration-files and fix inconsistences between ORM/migration files#62234
Merged
ephraimbuddy merged 11 commits intoapache:mainfrom Mar 17, 2026
Merged
Conversation
ephraimbuddy
requested review from
amoghrajesh,
ashb,
bugraoz93,
dheerajturaga,
gopidesupavan,
jason810496,
jscheffl,
kaxil and
potiuk
as code owners
boring-cyborg
bot
added
area:CLI
area:db-migrations
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
from
86ed1f2 to
20d3a5f
Compare
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
2 times, most recently
from
91cf520 to
bedbf53
Compare
ephraimbuddy
requested review from
bbovenzi,
choo121600,
guan404ming,
pierrejeambrun,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
ephraimbuddy
commented
Feb 25, 2026
...low/providers/edge3/migrations/versions/0001_3_0_2_fix_migration_file_orm_inconsistencies.py
Outdated
Show resolved
Hide resolved
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
2 times, most recently
from
1f0e30a to
4637686
Compare
kaxil
approved these changes
Mar 16, 2026
Member
kaxil
left a comment
kaxil
left a comment
There was a problem hiding this comment.
LGTM — previous comments addressed. Two minor items on the new migration code.
kaxil
reviewed
Mar 16, 2026
kaxil
reviewed
Mar 16, 2026
...s/fab/src/airflow/providers/fab/migrations/versions/0001_3_5_0_fix_fab_db_inconsistencies.py
Show resolved
Hide resolved
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
3 times, most recently
from
4f5ccb1 to
e5b0512
Compare
… ORM/migration files The `--use-migration-files` flag was removed in apache#41120 when we pruned the migration history for Airflow 3. At the time, we couldn't create a database from scratch using only migration files because the migration chain no longer started from an empty database. The oldest migration assumed an existing 2.6.2 schema. Now that we have a squashed migration (0000_2_6_2) with `down_revision = None` that creates the full baseline schema from scratch, we can re-introduce this flag. This enables a critical development workflow, which is: creating the database purely from migration files, then using `alembic revision --autogenerate` to detect schema drift between the ORM models and the migration-produced schema. Without this flag, `autogenerate` compares against an ORM-created database (via `create_all`), which masks any drift since both sides come from the same ORM definitions.
…between ORM/migration files
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
from
e5b0512 to
49f15fd
Compare
Backport failed to create: v3-1-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 03c6388 v3-1-testThis should apply the commit to the v3-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
ephraimbuddy
added a commit
to astronomer/airflow
that referenced
this pull request
Mar 18, 2026
migration 0108 in PR apache#62234 to backfill existing NULLs. After revisiting the change, we found the migration was correcting nullability drift introduced by that PR itself, and that the PR has not been released yet. Instead of shipping a large corrective migration, remove 0108 and restore the affected ORM fields to nullable so the models stay aligned with the actual schema and previous releases behavior. Also optimize migration 0094 by keeping the callback data migration correct while reducing repeated scans during batching.
ephraimbuddy
added a commit
to astronomer/airflow
that referenced
this pull request
Mar 18, 2026
PR apache#56330 mistakenly changed several ORM-backed columns from nullable to non-nullable. I later added a corrective migration in PR apache#62234 to backfill existing NULLs, but after revisiting the issue we found the migration was only compensating for the nullability mistake introduced by PR apache#56330 itself, which has not been released. Instead of shipping a large corrective migration for an unreleased regression, delete the migration file and restore the affected ORM fields to nullable so the models stay aligned with the actual schema and intended pre-release behavior.
ephraimbuddy
added a commit
to astronomer/airflow
that referenced
this pull request
Mar 18, 2026
PR apache#56330 mistakenly changed several ORM-backed columns from nullable to non-nullable. I later added a corrective migration in PR apache#62234 to backfill existing NULLs, but after revisiting the issue we found the migration was only compensating for the nullability mistake introduced by PR apache#56330 itself, which has not been released. Instead of shipping a large corrective migration for an unreleased regression, delete the migration file and restore the affected ORM fields to nullable so the models stay aligned with the actual schema and intended pre-release behavior.
ephraimbuddy
added a commit
to astronomer/airflow
that referenced
this pull request
Mar 19, 2026
PR apache#56330 mistakenly changed several ORM-backed columns from nullable to non-nullable. I later added a corrective migration in PR apache#62234 to backfill existing NULLs, but after revisiting the issue we found the migration was only compensating for the nullability mistake introduced by PR apache#56330 itself, which has not been released. Instead of shipping a large corrective migration for an unreleased regression, delete the migration file and restore the affected ORM fields to nullable so the models stay aligned with the actual schema and intended pre-release behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
--use-migration-filesflag was removed in #41120 when we pruned the migration history for Airflow 3. At the time, we couldn't create a database from scratch using only migration files because the migration chain no longer started from an empty database. The oldest migration assumed an existing 2.6.2 schema.Now that we have a squashed migration (0000_2_6_2) with
down_revision = Nonethat creates the full baseline schema from scratch, we can re-introduce this flag. This enables a critical development workflow, which is: creating the database purely from migration files, then usingalembic revision --autogenerateto detect schema drift between the ORM models and the migration-produced schema. Without this flag,autogeneratecompares against an ORM-created database (viacreate_all), which masks any drift since both sides come from the same ORM definitions.