Skip to content

Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyUPath#60519

Merged
potiuk merged 9 commits intoapache:mainfrom
ap--:update-universal-pathlib
Jan 29, 2026
Merged

Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyUPath#60519
potiuk merged 9 commits intoapache:mainfrom
ap--:update-universal-pathlib

Conversation

@ap--
Copy link
Copy Markdown
Contributor

@ap-- ap-- commented Jan 14, 2026 •
edited
Loading

Closes #56369

This PR updates airflow's ObjectStoragePath class to be compatible with universal-pathlib >=0.3.8. It now uses upath.extensions.ProxyUPath as a base class, to avoid using the now deprecated _protocol_dispatch = False mechanism for class API extensions.

Also pathlib.Path added .copy, .copy_into, .move, and .move_into methods in 3.14, which means the ObjectStoragePath behavior for these is now different from standard pathlib, which defaults to recursive copies for directories and returns the target path.

Cheers,
Andreas ☺️

kaxil
kaxil reviewed Jan 14, 2026
View reviewed changes
@kaxil kaxil requested a review from bolkedebruin January 15, 2026 16:33
@kaxil
Copy link
Copy Markdown
Member

kaxil commented Jan 15, 2026

cc @bolkedebruin

@potiuk potiuk added all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs full tests needed We need to run full set of tests for this PR to merge labels Jan 29, 2026
@potiuk potiuk force-pushed the update-universal-pathlib branch from 779bf73 to 10dbeb3 Compare January 29, 2026 13:40
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Jan 29, 2026

LGTM. Rebased and run it for all versions/full tests to see if all tests still pass.

@potiuk potiuk merged commit e166926 into apache:main Jan 29, 2026
228 checks passed
@ap--
Copy link
Copy Markdown
Contributor Author

ap-- commented Jan 29, 2026

Thanks @potiuk! Feel free to tag me in case errors regarding ObjectStoragePath get reported.
I'd be happy to be added to the code owners file if that would enable automated tagging in issues as well.

And I'm still interested in finding a way of running the airflow test suite with a new universal-pathlib version before I make a new release.
Let me know if you have any suggestions.

Cheers,
Andreas

sanchalitorpe-source pushed a commit to sanchalitorpe-source/airflow that referenced this pull request Jan 30, 2026
@ap-- @sanchalitorpe-source
Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyU…
168fcf2 
…Path (apache#60519)

* pyproject.toml: update universal-pathlib version boundaries

* task_sdk.io.path: fix test regarding relative_to

* tests: adjust the lazy_load test to reflect caching is done via STORE_CACHE

* tests: update tests to register fake remote filesystem in fsspec

* airflow.sdk.io.path: implement ObjectStoragePath via ProxyUPath

* airflow.sdk.io.path: provide a basic implementation for copy_into and move_into

* airflow.sdk.io.path: fix __str__ method

* airflow.sdk.io.path: docstring fixes

* update spelling_wordlist.txt
morelgeorge pushed a commit to morelgeorge/airflow that referenced this pull request Feb 1, 2026
@ap-- @morelgeorge
Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyU…
1282438 
…Path (apache#60519)

* pyproject.toml: update universal-pathlib version boundaries

* task_sdk.io.path: fix test regarding relative_to

* tests: adjust the lazy_load test to reflect caching is done via STORE_CACHE

* tests: update tests to register fake remote filesystem in fsspec

* airflow.sdk.io.path: implement ObjectStoragePath via ProxyUPath

* airflow.sdk.io.path: provide a basic implementation for copy_into and move_into

* airflow.sdk.io.path: fix __str__ method

* airflow.sdk.io.path: docstring fixes

* update spelling_wordlist.txt
@ap-- ap-- mentioned this pull request Feb 1, 2026
Open
2 tasks
jason810496 pushed a commit to abhijeets25012-tech/airflow that referenced this pull request Feb 3, 2026
@ap-- @jason810496
Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyU…
bc99e78 
…Path (apache#60519)

* pyproject.toml: update universal-pathlib version boundaries

* task_sdk.io.path: fix test regarding relative_to

* tests: adjust the lazy_load test to reflect caching is done via STORE_CACHE

* tests: update tests to register fake remote filesystem in fsspec

* airflow.sdk.io.path: implement ObjectStoragePath via ProxyUPath

* airflow.sdk.io.path: provide a basic implementation for copy_into and move_into

* airflow.sdk.io.path: fix __str__ method

* airflow.sdk.io.path: docstring fixes

* update spelling_wordlist.txt
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Feb 3, 2026

I'd be happy to be added to the code owners file if that would enable automated tagging in issues as well.

Only committers can be added :(

And I'm still interested in finding a way of running the airflow test suite with a new universal-pathlib version before I make a new release.
Let me know if you have any suggestions.

This should be rather easy - make an rc release and make a PR to airflow with universal-pathlib>=your_rc_release

jhgoebbert pushed a commit to jhgoebbert/airflow_Owen-CH-Leung that referenced this pull request Feb 8, 2026
@ap-- @jhgoebbert
Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyU…
a6d3e3b 
…Path (apache#60519)

* pyproject.toml: update universal-pathlib version boundaries

* task_sdk.io.path: fix test regarding relative_to

* tests: adjust the lazy_load test to reflect caching is done via STORE_CACHE

* tests: update tests to register fake remote filesystem in fsspec

* airflow.sdk.io.path: implement ObjectStoragePath via ProxyUPath

* airflow.sdk.io.path: provide a basic implementation for copy_into and move_into

* airflow.sdk.io.path: fix __str__ method

* airflow.sdk.io.path: docstring fixes

* update spelling_wordlist.txt
choo121600 pushed a commit to choo121600/airflow that referenced this pull request Feb 22, 2026
@ap-- @choo121600
Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyU…
831316f 
…Path (apache#60519)

* pyproject.toml: update universal-pathlib version boundaries

* task_sdk.io.path: fix test regarding relative_to

* tests: adjust the lazy_load test to reflect caching is done via STORE_CACHE

* tests: update tests to register fake remote filesystem in fsspec

* airflow.sdk.io.path: implement ObjectStoragePath via ProxyUPath

* airflow.sdk.io.path: provide a basic implementation for copy_into and move_into

* airflow.sdk.io.path: fix __str__ method

* airflow.sdk.io.path: docstring fixes

* update spelling_wordlist.txt
@jason810496 jason810496 mentioned this pull request Mar 2, 2026
Merged
Subham-KRLX pushed a commit to Subham-KRLX/airflow that referenced this pull request Mar 4, 2026
@ap-- @Subham-KRLX
Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyU…
94bf042 
…Path (apache#60519)

* pyproject.toml: update universal-pathlib version boundaries

* task_sdk.io.path: fix test regarding relative_to

* tests: adjust the lazy_load test to reflect caching is done via STORE_CACHE

* tests: update tests to register fake remote filesystem in fsspec

* airflow.sdk.io.path: implement ObjectStoragePath via ProxyUPath

* airflow.sdk.io.path: provide a basic implementation for copy_into and move_into

* airflow.sdk.io.path: fix __str__ method

* airflow.sdk.io.path: docstring fixes

* update spelling_wordlist.txt
Ankurdeewan pushed a commit to Ankurdeewan/airflow that referenced this pull request Mar 15, 2026
@ap-- @Ankurdeewan
Update universal-pathlib to >=0.3.8 and use upath.extensions.ProxyU…
e44dd5e 
…Path (apache#60519)

* pyproject.toml: update universal-pathlib version boundaries

* task_sdk.io.path: fix test regarding relative_to

* tests: adjust the lazy_load test to reflect caching is done via STORE_CACHE

* tests: update tests to register fake remote filesystem in fsspec

* airflow.sdk.io.path: implement ObjectStoragePath via ProxyUPath

* airflow.sdk.io.path: provide a basic implementation for copy_into and move_into

* airflow.sdk.io.path: fix __str__ method

* airflow.sdk.io.path: docstring fixes

* update spelling_wordlist.txt
@vatsrahul1001 vatsrahul1001 mentioned this pull request Apr 2, 2026
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxil kaxil kaxil left review comments

@potiuk potiuk potiuk approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh is a code owner

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin

Assignees

No one assigned

Labels

all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs area:task-sdk full tests needed We need to run full set of tests for this PR to merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Rework universal-pathlib in ObjectStoragePath

3 participants

@ap-- @kaxil @potiuk