Repository navigation
Async S3 support needed #32174
Description
Description
We need async support for S3 facilities for use in our Triggers.
We see AwsBaseAsyncHook was introduced in this commit (PR 28850) merged on 2023-03-14.
However in #31610 , AwsBaseAsyncHook was marked Deprecated by @pankajastro and subsequently approved by @vincbeck .
I would like to know:
- on what basis do you determine this class is not used? There has been no discussions in the PR.
- from a user stance, providing a new functionality then removing it a few months later is not very good practice.
- is async S3 operations going to be supported ? We have code based on this and deprecating this will break our code.
Many thanks, particularly to the two gentlemen I named above.
Use case/motivation
No response
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jun 27, 2023
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
pankajastro commentedon Jun 27, 2023
We decided to use boto waiter for aws deferrable operator and not create a separate async hook.
You can check some examples here https://github.com/apache/airflow/tree/main/airflow/providers/amazon/aws/triggers how we are creating triggers. particular line
airflow/airflow/providers/amazon/aws/triggers/batch.py
Line 76 in 688f91b
yjdlau commentedon Jun 27, 2023
@pankajastro Could you please add context to your PR and discussion, particularly the link to the discussion. From your PR it looks like you're killing a feature without any discussion.
pankajastro commentedon Jun 27, 2023
Right, but keeping duplicate/unused code is not good. When I say unused code I mean not getting used in airflow code. This is a public interface that why I have deprecated it first.
yes, it will be supported.
pankajastro commentedon Jun 27, 2023
we have a discussion and PR #30032 where we decided to use the waiter and not create a separate hook async operator.
pankajastro commentedon Jun 27, 2023
FYI: I added AwsBaseAsyncHook hook
yjdlau commentedon Jun 27, 2023
Got it. That clears things up. Closing issue.