Skip to content

Pydantic 2.0.0 support for Airflow #32311

New issue
New issue
Closed
#32366
Closed
Pydantic 2.0.0 support for Airflow#32311
#32366
Assignees
potiukuranusjr
Labels
kind:metaHigh-level information important to the community

Description

@potiuk
potiuk
opened on Jul 2, 2023

Body

Currently Pydantic 2.0.0 released on 30th of June 2023 breaks Airflow CI - building providers and running Kubernetes tests.

Therefore we limit Pydantic to < 2.0.0 in setup.cfg for now.

This should be fixed, especially that 2.0.0 brings a number of speed improvements.

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Activity

  1. potiuk
    added
    kind:metaHigh-level information important to the community
    on Jul 2, 2023
  2. potiuk
    self-assigned this
    on Jul 2, 2023
  3. potiuk
    added 3 commits that reference this issue on Jul 2, 2023

    Limit Pydantic to < 2.0.0 until we solve 2.0.0 incompatibilities

    b4ba2fc

    Limit Pydantic to < 2.0.0 until we solve 2.0.0 incompatibilities (#32312

    2811ba7

    Limit Pydantic to < 2.0.0 until we solve 2.0.0 incompatibilities (#32312

    1bc6c99

  4. uranusjr commented on Jul 3, 2023

    @uranusjr
    uranusjr
    on Jul 3, 2023
    Member

    Do we have a way to identify incompatibilities so they can be fixed? Maybe a CI job for example?

  5. potiuk commented on Jul 3, 2023

    @potiuk
    potiuk
    on Jul 3, 2023
    MemberAuthor

    It's enough to remove the limit and set full-tests-needed for the PR and fix the problems. No need for special job.

  6. potiuk commented on Jul 3, 2023

    @potiuk
    potiuk
    on Jul 3, 2023
    MemberAuthor

    Feel free to pick it up if you want :)

  7. uranusjr commented on Jul 3, 2023

    @uranusjr
    uranusjr
    on Jul 3, 2023
    Member

    Ah right, will do that

  8. potiuk
    assigned
    uranusjr
    on Jul 3, 2023

  9. uranusjr commented on Jul 5, 2023

    @uranusjr
    uranusjr
    on Jul 5, 2023
    Member

    It seems that some of our dependencies have added the <2 pin as well and removing it from our direct dependencies is no longer sufficient to install Pydantic 2 in CI. Adding >=2 results in a resolution conflict. Will need to find another way.

  10. potiuk commented on Jul 5, 2023

    @potiuk
    potiuk
    on Jul 5, 2023
    MemberAuthor

    It seems that some of our dependencies have added the <2 pin as well and removing it from our direct dependencies is no longer sufficient to install Pydantic 2 in CI. Adding >=2 results in a resolution conflict. Will need to find another way.

    Likely waiting for them to support Pydantic 2 :)

  11. potiuk commented on Jul 5, 2023

    @potiuk
    potiuk
    on Jul 5, 2023
    MemberAuthor

    Also it is likely some other incompatibilities (aka "teething problems" with Pydantic 2) might simply get fixed - Pydantic 2.0.1 has been released yesterday with ~40 bugs fixed. so waiting is probably a very good strategy in general.

    Andthat's a good news actually that other deps limited it for now, because it seems that likely even no-constraints installation will be fine for 2.6.0 - 2.6.2.

  12. tatiana
    added a commit that references this issue on Jul 5, 2023

    Limit Pydantic < 2.0.0 until Airflow resolves incompatibilities

    52247cb

  13. uranusjr commented on Jul 5, 2023

    @uranusjr
    uranusjr
    on Jul 5, 2023
    Member

    Hey the CI is actually passing with Pydantic 2.0.1. Maybe this is it and there’s nothing to fix at our part.

  14. potiuk commented on Jul 5, 2023

    @potiuk
    potiuk
    on Jul 5, 2023
    MemberAuthor

    Nope. Not passing. There are K8S and Docker compose failing (and errors in it suggest Pydantic being the reason).

  15. potiuk commented on Jul 5, 2023

    @potiuk
    potiuk
    on Jul 5, 2023
    MemberAuthor

    I might take a look at that later - but I think there are a few more changes to be added.

  16. potiuk commented on Jul 5, 2023

    @potiuk
    potiuk
    on Jul 5, 2023
    MemberAuthor

    Example error from migration Job in K8S tests (when you look at summary and download the failed log artifacts you can brows them and in the migration job you will find this):

    2023-07-05T09:31:04.936389433Z stderr F /home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_config.py:257 UserWarning: Valid config keys have changed in V2:
2023-07-05T09:31:04.936397782Z stderr F * 'orm_mode' has been renamed to 'from_attributes'
2023-07-05T09:31:05.062051199Z stdout F [2023-07-05T09:31:05.061+0000] {cli_action_loggers.py:83} DEBUG - Calling callbacks: []
2023-07-05T09:31:05.062066769Z stderr F Traceback (most recent call last):
2023-07-05T09:31:05.062072845Z stderr F   File "/home/airflow/.local/bin/airflow", line 8, in <module>
2023-07-05T09:31:05.062162218Z stderr F     sys.exit(main())
2023-07-05T09:31:05.062165605Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 48, in main
2023-07-05T09:31:05.062213762Z stderr F     args.func(args)
2023-07-05T09:31:05.062216746Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_config.py", line 52, in command
2023-07-05T09:31:05.062285355Z stderr F     return func(*args, **kwargs)
2023-07-05T09:31:05.062289578Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 112, in wrapper
2023-07-05T09:31:05.062340064Z stderr F     return f(*args, **kwargs)
2023-07-05T09:31:05.062346687Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 88, in upgradedb
2023-07-05T09:31:05.062417707Z stderr F     db.upgradedb(
2023-07-05T09:31:05.062421133Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 76, in wrapper
2023-07-05T09:31:05.062488343Z stderr F     return func(*args, session=session, **kwargs)
2023-07-05T09:31:05.062495223Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 1569, in upgradedb
2023-07-05T09:31:05.062880304Z stderr F     import_all_models()
2023-07-05T09:31:05.062884917Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/__init__.py", line 60, in import_all_models
2023-07-05T09:31:05.062926532Z stderr F     __getattr__(name)
2023-07-05T09:31:05.062929423Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/__init__.py", line 78, in __getattr__
2023-07-05T09:31:05.062997617Z stderr F     val = import_string(f"{path}.{name}")
2023-07-05T09:31:05.063001825Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 36, in import_string
2023-07-05T09:31:05.063055324Z stderr F     module = import_module(module_path)
2023-07-05T09:31:05.063059364Z stderr F   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
2023-07-05T09:31:05.063134198Z stderr F     return _bootstrap._gcd_import(name[level:], package, level)
2023-07-05T09:31:05.063142828Z stderr F   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2023-07-05T09:31:05.06323632Z stderr F   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2023-07-05T09:31:05.06328635Z stderr F   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
2023-07-05T09:31:05.063339005Z stderr F   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
2023-07-05T09:31:05.063399209Z stderr F   File "<frozen importlib._bootstrap_external>", line 843, in exec_module
2023-07-05T09:31:05.063487119Z stderr F   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2023-07-05T09:31:05.06353937Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/skipmixin.py", line 25, in <module>
2023-07-05T09:31:05.063609088Z stderr F     from airflow.serialization.pydantic.dag_run import DagRunPydantic
2023-07-05T09:31:05.063612022Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/serialization/pydantic/dag_run.py", line 27, in <module>
2023-07-05T09:31:05.063656021Z stderr F     class DagRunPydantic(BaseModelPydantic):
2023-07-05T09:31:05.063657927Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_model_construction.py", line 172, in __new__
2023-07-05T09:31:05.06374619Z stderr F     complete_model_class(
2023-07-05T09:31:05.063749628Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_model_construction.py", line 420, in complete_model_class
2023-07-05T09:31:05.063884186Z stderr F     schema = cls.__get_pydantic_core_schema__(cls, handler)
2023-07-05T09:31:05.063887206Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/main.py", line 533, in __get_pydantic_core_schema__
2023-07-05T09:31:05.064028145Z stderr F     return __handler(__source)
2023-07-05T09:31:05.064040108Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
2023-07-05T09:31:05.06409642Z stderr F     schema = self._handler(__source_type)
2023-07-05T09:31:05.064099722Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 264, in generate_schema
2023-07-05T09:31:05.064204129Z stderr F     return self._generate_schema_for_type(
2023-07-05T09:31:05.064206503Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 285, in _generate_schema_for_type
2023-07-05T09:31:05.064311296Z stderr F     schema = self._generate_schema(obj)
2023-07-05T09:31:05.064316069Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 475, in _generate_schema
2023-07-05T09:31:05.064453975Z stderr F     return self._model_schema(obj)
2023-07-05T09:31:05.064457652Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 351, in _model_schema
2023-07-05T09:31:05.064563542Z stderr F     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
2023-07-05T09:31:05.064566708Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 351, in <dictcomp>
2023-07-05T09:31:05.064668327Z stderr F     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
2023-07-05T09:31:05.06467106Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 616, in _generate_md_field_schema
2023-07-05T09:31:05.064819744Z stderr F     common_field = self._common_field_schema(name, field_info, decorators)
2023-07-05T09:31:05.064822125Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 658, in _common_field_schema
2023-07-05T09:31:05.064989175Z stderr F     schema = self._apply_annotations(
2023-07-05T09:31:05.064997161Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 1316, in _apply_annotations
2023-07-05T09:31:05.065285199Z stderr F     schema = get_inner_schema(source_type)
2023-07-05T09:31:05.065288123Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
2023-07-05T09:31:05.065336079Z stderr F     schema = self._handler(__source_type)
2023-07-05T09:31:05.065338463Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 1277, in inner_handler
2023-07-05T09:31:05.065650573Z stderr F     schema = self._generate_schema(obj)
2023-07-05T09:31:05.065655344Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 537, in _generate_schema
2023-07-05T09:31:05.065809391Z stderr F     return self._arbitrary_type_schema(obj, obj)
2023-07-05T09:31:05.065859693Z stderr F   File "/home/airflow/.local/lib/python3.8/site-packages/pydantic/_internal/_generate_schema.py", line 580, in _arbitrary_type_schema
2023-07-05T09:31:05.066024014Z stderr F     raise PydanticSchemaGenerationError(
2023-07-05T09:31:05.066029887Z stderr F pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'pendulum.datetime.DateTime'>. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.
2023-07-05T09:31:05.066033448Z stderr F

  17. potiuk commented on Jul 5, 2023

    @potiuk
    potiuk
    on Jul 5, 2023
    MemberAuthor

    This is what I saw before as well

  18. potiuk commented on Jul 5, 2023

    @potiuk
    potiuk
    on Jul 5, 2023
    MemberAuthor

    BTW. This is one of the great examples where having few "integration" tests where we run end-to-end tests in our CI actually makes sense - because no unit tests were able to catch this one.

  19. tatiana
    added 3 commits that reference this issue on Jul 5, 2023

    Limit Pydantic < 2.0.0 until Airflow resolves incompatibilities

    581e5d1

    Limit Pydantic < 2.0.0 until Airflow resolves incompatibilities

    eb22989

    Limit Pydantic < 2.0.0 until Airflow resolves incompatibilities (#52)

    63d34c1

  20. uranusjr commented on Jul 6, 2023

    @uranusjr
    uranusjr
    on Jul 6, 2023
    Member

    Hey I think I got things working

  21. tatiana
    added a commit that references this issue on Jul 7, 2023

    Limit Pydantic < 2.0.0 until Airflow resolves incompatibilities (#1981)

    fd75e41
  22. potiuk
    closed this as completedin #32366on Jul 7, 2023
  23. utkarsharma2
    added a commit that references this issue on Aug 8, 2023

    Limit Pydantic < 2.0.0 until Airflow resolves incompatibilities (#1981)

    44cbc86

  24. ldacey commented on Sep 6, 2023

    @ldacey
    ldacey
    on Sep 6, 2023
    Contributor

    Would upgrading to pydantic 2 break core Airflow functionality? I have sometimes updated other packages (pandas, pyarrow) beyond the constraints Airflow sets. The latest release of polars depends on pydantic 2.0 but I can't break Airflow over my eagerness to update.

  25. uranusjr commented on Sep 6, 2023

    @uranusjr
    uranusjr
    on Sep 6, 2023
    Member

    I think 2.7 still has some incompatibilities in serialisation, but it should work adequately.

  26. potiuk commented on Sep 6, 2023

    @potiuk
    potiuk
    on Sep 6, 2023
    MemberAuthor

    Would upgrading to pydantic 2 break core Airflow functionality? I have sometimes updated other packages (pandas, pyarrow) beyond the constraints Airflow sets. The latest release of polars depends on pydantic 2.0 but I can't break Airflow over my eagerness to update.

    As @uranusjr wrote - no, it should not break anything once you use 2.7.1. In 2.7.0 you will have warnings about some deprecated Pydantic features, but it should be harmless otherwise. In 2.7.1 you should get the warnings suppressed. 2.8.0 will likely require you to use Pydantic 2. We are not yet "actively" using Pydantic, it's mostly in-development support. It also allows you to serialize Pydantic models (if for example you pass them as XComs) but I think upcoming 2.7.1 should generally make them work for both Pydantic 1 and 2.

    If you are not passing Pydantic models this way - you should be generally unaffected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind:metaHigh-level information important to the community

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions