Repository navigation
Closed
Labels
kind:metaHigh-level information important to the communityHigh-level information important to the communitytesting statusStatus of testing releasesStatus of testing releases
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
The guidelines on how to test providers can be found in
Verify providers by contributors
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
Provider amazon: 8.13.0rc1
- Support IAM authentication for Redshift serverless (#35897): @hussein-awala
- Implement
is_authorized_variablein AWS auth manager (#35804): @vincbeck - Enhance DynamoDBValueSensor to support any attribute value in list (#35831): @flolas
- Fix handling of single quotes in
RedshiftToS3Operator(#35986): @vincbeck - Fix a bug in get_iam_token for Redshift Serverless (#36001): @hussein-awala
- Fix reraise outside of try block in
AthenaHook.get_output_location(#36008): @Taragolis - Fix a bug with accessing hooks in EKS trigger (#35989): @hussein-awala
- Fix a bug in method name used in GlacierToGCSOperator (#35978): @hussein-awala
- Fix EC2Hook get_instance for client_type api (#35960): @hussein-awala
- Avoid creating the hook in the EmrServerlessCancelJobsTrigger init (#35992): @hussein-awala
- Stop getting message from event after migrating EmrContainerTrigger to AwsBaseWaiterTrigger (#35892): @hussein-awala
- Fix for EksCreateClusterOperator deferrable mode (#36079): @syedahsn
- Update
boto3andbotocoreversions notes (#36073): @Taragolis - Improve typing hints for
only_client_typedecorator (#35997): @hussein-awala - Refactor some methods in EmrContainerHook (#35999): @hussein-awala
- Refactor
get_output_locationmethod in AthenaHook (#35996): @hussein-awala - Move RDS hook to a cached property in RDS trigger (#35990): @hussein-awala
- Replace default empty dict value by None in AzureBlobStorageToS3Operator (#35977): @hussein-awala
- Refacto CloudwatchTaskHandler set_context method (#35975): @hussein-awala
- Use S3 hook instead of AwsGenericHook in AWS S3 FS (#35973): @hussein-awala
- AWS auth manager: implement all
is_authorized_*methods (butis_authorized_dag) (#35928): @vincbeck - Remove setting a non-existing object param and use local var instead in S3Hook (#35950): @hussein-awala
Provider cncf.kubernetes: 7.11.0rc1
- fix: KPO typing env_vars (#36048): @raphaelauv
- Stop converting state to TaskInstanceState when it's None (#35891): @hussein-awala
- Feature pass dictionary configuration in application_file in SparkKubernetesOperator (#35848): @romsharon98
Provider docker: 3.9.0rc1
Provider elasticsearch: 5.3.0rc1
Provider google: 10.13.0rc1
- GCP Secrets Backend Impersonation (#36072): @nathadfield
- Add OpenLineage support to GCS operators (#35838): @kacpermuda
- Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel (#35136): @ginolegigot
- Fix DataprocSubmitJobOperator to retrieve failed job error message (#36053): @VladaZakharova
- Fix CloudRunExecuteJobOperator in deferrable mode (#36012): @VladaZakharova
- Fix gcs listing - ensure blobs are loaded (#34919): @atrbgithub
- allow multiple elements in impersonation chain (#35694): @melugoyal
- Deprecate
CloudComposerEnvironmentSensorin favor ofCloudComposerCreateEnvironmentOperatorwith defer mode (#35775): @VladaZakharova
Provider http: 4.8.0rc1
Provider jdbc: 4.2.0rc1
Provider microsoft.azure: 8.4.0rc1
- Fix reraise outside of try block in
AzureSynapsePipelineRunLink.get_fields_from_url(#36009): @Taragolis - Do not catch too broad exception in
WasbHook.delete_container(#36034): @Taragolis
Provider microsoft.mssql: 3.6.0rc1
Provider openai: 1.1.0rc1
Provider openlineage: 1.3.0rc1
- OpenLineage: Add parent_run_id for COMPLETE and FAIL events (#36067): @kacpermuda
- openlineage: add basic metrics to stats collector. (#35368): @JDarDagran
- Repair different run_id for OpenLineage START and FAIL events. (#36051): @kacpermuda
- openlineage: add ClassVar to Airflow facets. (#36084): @JDarDagran
Provider trino: 5.5.0rc1
Provider weaviate: 1.1.0rc1
- Add object methods in weaviate hook (#35934): @ephraimbuddy
- Add a cache for weaviate client (#35983): @ephraimbuddy
- Add more ways to connect to weaviate (#35864): @ephraimbuddy
All users involved in the PRs:
@Taragolis @vincbeck @romsharon98 @syedahsn @kacpermuda @raphaelauv @melugoyal @ginolegigot @atrbgithub @flolas @JDarDagran @nathadfield @potiuk @ephraimbuddy @Joffreybvn @pankajkoti @hussein-awala
@VladaZakharova @stavdav143 @eladkal
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Metadata
Metadata
Assignees
Labels
kind:metaHigh-level information important to the communityHigh-level information important to the communitytesting statusStatus of testing releasesStatus of testing releases
Activity
melugoyal commentedon Dec 8, 2023
#35694 working as expected
Updating setup.cfg with RC provider packages on apache/airflow#36117
Updating setup.cfg with RC provider packages on apache/airflow#36117
atrbgithub commentedon Dec 8, 2023
An issue has been raised with #34919
Unfortunately it looks like it has broken
GCSObjectsWithPrefixExistenceSensor.Testing myself I can confirm that the following code is able to create the error:
Please do not include this change.
Edit: I've raised a PR #36130 which appears to fix this issue for me when testing locally.
eladkal commentedon Dec 9, 2023
Docker provider will be excluded from rc1
hussein-awala commentedon Dec 9, 2023
I tested my change on the Kubernetes executor, and most of the changes in the Amazon provider, all work as expected.
pankajastro commentedon Dec 9, 2023
As @atrbgithub has mentioned above #34919 is breaking an existing DAG. so we might need to exclude Google provider
eladkal commentedon Dec 9, 2023
Yes. Google provider will also have RC2
potiuk commentedon Dec 10, 2023
Tested mssql - all looks good.
One more thing I noticed - the removed
daskexecutorhas not been prepared. We have this in the release process, so it's easy to miss:But maybye we will change it in the future - we have more removed providers coming. So possibly we shoudl automatically add removed providers to the list of providers prepared (both for documentation and package building - so that we do not have to remember to prepare them. That was probably bad idea to not add them in the first place.
@eladkal -> I guess we should include rc2 (even if there was no rc1) for
daskexecutor.potiuk commentedon Dec 10, 2023
One small update - as @pankajastro noticed in #32319 (comment) - there is a case where we are not handling properly the fact that
handlercan return either list of rows or row - in both ODBC and DAtabricks oviders, so IMHO they should also be moved to RC2 (cc: @Joffreybvn )potiuk commentedon Dec 10, 2023
Fix to the process to automatically generate removed providers here: #36160
Joffreybvn commentedon Dec 11, 2023
Fix for #32319 (comment) is here -> #36161
nathadfield commentedon Dec 11, 2023
#36072 has been tested and working as expected.
eladkal commentedon Dec 12, 2023
Providers: Google, ODBC, Databricks, Docker are excluded from RC1.
eladkal commentedon Dec 12, 2023
Thank you everyone.
Providers are released (excluding Google, ODBC, Databricks, Docker which we will have RC2 soon)
I invite everyone to help improve providers for the next release, a list of open issues can be found here.