Conversation
potiuk
requested review from
ashb,
jedcunningham,
kaxil and
pierrejeambrun
as code owners
|
cc: @dirrao @Taragolis -> seems like apache-beam having numpy as dependency is the next problem to solve after pendulum is solved |
|
Looks like we need NumPy 1.26+ - from that long discussion here: numpy/numpy#23808 and So likely the next best thing to do is to exclude apache-beam provider for python 3.12 This is kinda expected, Beam is always dragging us behind |
|
Pushed a fixup marking it for exclusion - let's see. |
|
Seems like 1.26.0 is the first release for numpy which officially support 3.12: |
|
Oh... I've post my comment without refresh page, and you've already found the same things |
|
Seems like Google provider also not compatible with 3.12 yet 46.9 ERROR: Could not find a version that satisfies the requirement google-ads>=22.1.0; extra == "google" (from apache-airflow[google]) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.3.0, 1.3.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.4.1, 3.0.0, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.3.0, 4.0.0, 4.1.0, 4.1.1, 5.0.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.1.0, 6.0.0, 7.0.0, 8.0.0, 8.1.0, 8.2.0, 9.0.0, 10.0.0, 11.0.0, 11.0.1, 11.0.2, 12.0.0, 13.0.0, 14.0.0, 14.0.1, 14.1.0, 15.0.0, 15.1.0, 15.1.1, 16.0.0, 17.0.0, 18.0.0, 18.1.0, 18.2.0, 19.0.0, 20.0.0, 21.0.0, 21.1.0, 21.2.0, 21.3.0, 22.0.0)
246.9 ERROR: No matching distribution found for google-ads>=22.1.0; extra == "google"Latest Issue for add support of Python 3.12 already exists googleads/google-ads-python#813 |
|
All right ... let me exclude google provider too then. At this stage I have a feeling that excluding few - even huge and important - providers and having an open -issue to bring the 3.12 support in would be a good thing. And I know for a fact that google team wants to split the google provider and splitting of ads was the first thing to try anyway, so that might accelerate things a bit. |
|
Pushed. |
|
BTW. I really like how nicely and transpartently the new |
|
I think one day we need to finally resolve Consider splitting Google Provider because google provider is really huge providers (28k+ lines which tracked by our test) and contains quite a few different components:
So if we found the way how it could be done it might prevent the situation that one of this component become a showstopper for others
|
This is precisely the plan I am discussing with Google team :) |
|
So we have |
|
Seems ❯ pip install duckdb==0.9.3.dev2258
Collecting duckdb==0.9.3.dev2258
Obtaining dependency information for duckdb==0.9.3.dev2258 from https://files.pythonhosted.org/packages/3c/43/094637a1939e8ba6ae53a788bd46adfd0b71fe0a6e182c8e6179b2966e09/duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl.metadata
Downloading duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl.metadata (768 bytes)
Downloading duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.7/13.7 MB 3.6 MB/s eta 0:00:00
Installing collected packages: duckdb
Successfully installed duckdb-0.9.3.dev2258 |
|
Pushed a change for it :) |
|
One more step and a new error, this time it is related to the LevelDB, which is also part of google provider, but I guess it has separate extra I think this is the same issue: wbolster/plyvel#158 |
|
Running |
|
Build Prod Images still picking the google ads provider. |
|
Yes. Because google provider is installed and epxected to be installed when PROD image is built. So if we do not build it locally for Pythin 3.12, it will install the one from In the meatime - we could take a close look at the failing tests for Python 3.13 https://github.com/apache/airflow/actions/runs/7509965786/job/20448000561?pr=36755) I think they mostly fail because |
potiuk
force-pushed
the
python3.12
branch
2 times, most recently
from
42c1339 to
2dee02f
Compare
|
cc: @dirrao @Taragolis -> Ads released with 3.12 support https://pypi.org/project/google-ads/ - removed the limit from Google provider, let's see. |
|
Duckdb removed devel version we had pinned for 3.12 -> replaced it with >= for the new devel |
|
🤞 🤞 🤞 🤞 🤞 🤞 |
@fruch . Just double checked it and for Python 3.12, whells for linux do not seem to be compiled with libev support. That basically makes python 3.12 cassandra-driver whl useless as it falls back to asyncore that is missing in Python 3.12 When I install it from PyPI with wheels, our test fail: I tested that when I build cassandra driver from sdist, the test start to work (so libev is used) Since it takes quite some time to build cassandra driver and it would complicate our CI builds (I'd have to specifically uninstall/install the driver from sdist - and our users will have to go through the same hoops, I will disable cassandra for Python 3.12 and open an issue in cassandra for the whl not supporting libev. I will re-enable it when the whl is compied with libev support. |
Can you try out the |
Yes. it seems to work out-of-the box with libev, Is it 100% compatible with cassandra-driver? And what are chances cassandra-driver will get it? We'de rather prefer to have dependency on the 'cassandra` one, unless there are good reasons it's not a good idea? |
I would go and open an issue for them, they can bundle it better. they might take a while to address it. We do grantee compatibility, and I think we can do a quicker turn around on issues, if you or your users gonna run into any. It's your call, but we are here if you need help with it. |
I did: https://datastax-oss.atlassian.net/jira/software/c/projects/PYTHON/issues/PYTHON-1378
Out of curiosity - also from the ASF point of view - who are WE vs. THEM? Seems like both cassandra and scylladn-client have Datastax Copyright (?) see below (and it's a fork of the datastax python-driver) - so I understand they are not part of the ASF PMC work? But Scylla is a separate compamy - somewhat competing with cassandra (?), so I am abit cautious here to use client fom scylladb. 
Any comments here for the govenance? |
Yes it's a fork, hence you still see the original licensing, along side with newer parts which are scylla specific Yes, scylla drivers are not part of the ASF. Yes, scylladb and Datastax are competitors. We still collaborate on the drivers, and try to help each other when possible. In this specific case of supporting python 3.12, we several communication with the datastax people to see we are mostly align on how to handle asyncore deprecation. As for the wheels, we are building them with cibuildwheel (relatively new project), while datastax have a private repo for those, so we can't actively help with that. At the end I'm trying to help, sorry if it comes out like I'm bashing other projects, cassandra community and datastax are doing great work with those drivers. I totally understand if it won't fit your project or your users, and the preference to stick to code which is part of ASF. (since this ASF project) |
Absolutely not. I just wanted to understand the situation :). Actually that explanation is a good one for me as I can make a comment on the issue, that we are considering swtiching to scylladb-driver for that.
Actually - I have no preference here to be honest. Technically speaking cassandra-driver is a Datastax project, not an ASF one - which is on it's own pretty strange, that cassandra python client is not part / owned by ASF and I am going to ask on the devlist so that they consider changing it - I think the name there is pretty problematic, because it suggests it is an ASF project. Both projects have good licences and technically there is no problem with including either. I will see what the devlist discussion brings. |
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in #36755 (cherry picked from commit 1301274)
Some of the providers need to be currently excluded from Python 3.12 because they have conflicting dependencies. While we are working on Python 3.12 support in #36755, in order to install airflow (for caching purposes) from GitHub URL, we need to separately merge the exclusions to main - this will help to build Python 3.12 CI image with all necessary dependencies cached. (cherry picked from commit b53fe08)
|
OK. this one is likely going to be green. Waiting for a round of reviews! |
hussein-awala
left a comment
hussein-awala
left a comment
There was a problem hiding this comment.
It looks good, I just added some nits, otherwise LGTM
|
Updated and addressed all NITS. Thanks @hussein-awala ! I also saw that we likely run out of disk space when preparing ARM CI images - so I adapted it a bit to run sequentially rather than in parallel, as it seems to build a bit more stable and it's not on critical path of the CI build. |
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
Some small finding, one comment. Open for full tests but I assume it will not relveal more than the CI anyway is covering today.
scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
Outdated
Show resolved
Hide resolved
|
don't know if it is/was a local issue but when starting on the PR the first time (after update of breeze of course) the command |
That's expected. First the image will be built with constraints - but since the constraints are not updated for 3.12, it will fail because of conflicts and there is automated attempt in this case to automatically upgrade to newer dependencies (which will succeed this time). All as expected. This should recover itself after teh PR gets merged and constraints will get updated |
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
After my short tests I can say... works.
There might be glitches when upgrading but complexity and bugs will come... anyway.
Finally after a number of dependency upgrades we seem to be able to upgrade to Python 3.12 (pending universal_pathlib 0.2.0 conversion) Several providers are excluded from being installed and wait for Python 3.12, but it should not block Airlfow's general 3.12 support.
8 participants
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.