Status of testing Providers that were prepared on October 13, 2023

@hussein-awala

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.6.0rc1

Consolidate hook management in AnalyticDBSparkSensor (#34435): @hussein-awala
Consolidate hook management in AnalyticDBSparkBaseOperator (#34434): @hussein-awala
Deprecate get_hook in OSSKeySensor and use hook instead (#34426): @hussein-awala

Provider amazon: 8.8.0rc1

Provider apache.beam: 5.3.0rc1

Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider apache.druid: 3.6.0rc1

Allow passing context to DruidDbApiHook (#34603): @saulius

Provider apache.flink: 1.2.0rc1

respect soft_fail argument when exception is raised for flink sensors (#34476): @Lee-W

Provider apache.hdfs: 4.2.0rc1

Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider apache.hive: 6.2.0rc1

Consolidate hook management in HiveOperator (#34430): @hussein-awala

Provider apache.kafka: 1.2.0rc1

Provider apache.livy: 3.6.0rc1

Consolidate hook management in LivyOperator (#34431): @hussein-awala

Provider arangodb: 2.3.0rc1

Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider celery: 3.4.0rc1

respect soft_fail argument when exception is raised for celery sensors (#34474): @Lee-W

Provider cncf.kubernetes: 7.7.0rc1

Provider common.sql: 1.8.0rc1

Provider databricks: 4.6.0rc1

Provider datadog: 3.4.0rc1

respect soft_fail argument when exception is raised for datadog sensors (#34472): @Lee-W

Provider dbt.cloud: 3.4.0rc1

Remove useless print from dbt operator (#34322): @hussein-awala

Provider docker: 3.8.0rc1

Provider elasticsearch: 5.1.0rc1

Provider github: 2.4.0rc1

Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider google: 10.10.0rc1

Provider http: 4.6.0rc1

respect soft_fail argument when exception is raised for http sensors (#34391): @Lee-W

Provider jenkins: 3.4.0rc1

Provider microsoft.azure: 8.0.0rc1

Provider openlineage: 1.1.1rc1

openlineage: adjust log levels (#34801): @JDarDagran

Provider papermill: 3.4.0rc1

Remove Airflow 2.3 compat from PapermillOperator (#34675): @Taragolis

Provider postgres: 5.7.0rc1

Add get_ui_field_behaviour() method to PostgresHook (#34811): @pubudu91

Provider presto: 5.2.0rc1

Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider samba: 4.3.0rc1

Feature: Add Google Cloud Storage to Samba operator (#34369): @ealebed

Provider slack: 8.2.0rc1

Slack: use default_conn_name by default (#34548): @Khrol

Provider snowflake: 5.1.0rc1

Provider trino: 5.4.0rc1

Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

All users involved in the PRs:
@Zhenye-Na @pubudu91 @pankajastro @krisfur @Taragolis @kahlstrm @Khrol @moiseenkov @utkarsharma2 @saulius @irajankumar @ealebed @eumiro @hussein-awala @cjonesy @dstandish @daxmahoney @Seokyun-Ha
@oleksiidav @aritra24 @cBiscuitSurprise @dkim010 @frankcash @hankehly @asherkhb @bkossakowska @Lee-W @Owen-CH-Leung @fdemiane @vandonr-amz @JDarDagran @dzhigimont @Adaverse @yanfangli85

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Updating setup.cfg with RC provider packages on apache/airflow#34930

Tested with our examples DAGs with only one expected fail due to azure provider breaking change.

apache-airflow-providers-samba==4.3.0rc1 has been tested with our examples DAGs (on staging Airflow and test Google Cloud Composer environments)

Tested the postgres provider 5.7.0rc1 locally. The fix done is available in it.

checked apache-airflow-providers-cncf-kubernetes==7.7.0rc1. The fix is working as expected.

Checked my changes, and they seem to be working fine. Thanks, Utkarsh Sharma

…

On Sun, Oct 15, 2023 at 1:01 PM Owen Leung ***@***.***> wrote: checked apache-airflow-providers-cncf-kubernetes==7.7.0rc1. The fix is working as expected. — Reply to this email directly, view it on GitHub <#34930 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDLAHIDCF7DE6NJ3B7OA73X7OGLPANCNFSM6AAAAAA57TO654> . You are receiving this because you were mentioned.Message ID: ***@***.***>

#33031 is working on amazon-8.8.0rc1 😄
Here are the commands I used to setup my environment / run my checks if it helps anyone.

breeze
pip install apache-airflow-providers-amazon==8.8.0rc1
pip list | grep amazon  # check version

# Run unit tests
pytest tests/providers/amazon/aws/transfers/test_gcs_to_s3.py

# Run system tests
export AWS_ACCESS_KEY_ID=***
export AWS_SECRET_ACCESS_KEY=***
export AWS_DEFAULT_REGION=***
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
export GCP_PROJECT_ID=***
pytest --system amazon tests/system/providers/amazon/aws/example_gcs_to_s3.py

Tested #32779, it works as expected.

Tested out #33321, works as expected

Tested #34603 in dev environment, works as expected.

#34545 and #34514 work as expected

#34801 tested, works as expected

Use airflow.exceptions.AirflowException in providers #34511: I've checked that packages in this release doesn't use lazy import AirflowException anymore
Remove Airflow 2.3 compat from PapermillOperator #34675: Check that operator doesn't have any serialization problem, as well as parameters templates after the changes
Include AWS Lambda execution logs to task logs #34692: works as expected
Allow setup endpoint_url per-service in AWS Connection #34593: works as expected, test on couple services include STS
Implements AwsBaseOperator and AwsBaseSensor #34784: No side effect found, this classes don't use right now
Extend hooks arguments into AwsBaseWaiterTrigger #34884: Couldn't found side effect in existed triggers, additional parameters doesn't provided from community Operators yet
Remove duplicate asgiref dependency in Amazon Provider #34580: Amazon Provider package's metadata contain asgiref as dependency only once

Use airflow.models.dag.DAG in Google Provider examples #34614: System test/examples only
Clarify Amazon Lambda invocation and sensing #34653: Doc only

Decode response in f-string #34518: Hard to check because I do not have access to any Databriks or Snowflake environment, should not be a problem since this code executed only on Hooks/Operator's fatal error

I trust the test coverage of #33472 since the changes are pretty trivial. Let's ship it 💪

I checked if all my changes are present, and I tested some of them, all looks good 👍

Confirming Amazon 8.8.0rc1 resolves the issues related to my changes 👍

slack: 8.2.0rc1

Tested locally. LGTM.

Thank you everyone.
Providers are released
I invite everyone to help improve providers for the next release, a list of open issues can be found here.

I Checked #34643 works as expected!

Sorry for late response 🙏

All good with #34627 (apologies for the very late response, holidays)

Status of testing Providers that were prepared on October 13, 2023 #34930

Description

Body

Provider alibaba: 2.6.0rc1

Provider amazon: 8.8.0rc1

Provider apache.beam: 5.3.0rc1

Provider apache.druid: 3.6.0rc1

Provider apache.flink: 1.2.0rc1

Provider apache.hdfs: 4.2.0rc1

Provider apache.hive: 6.2.0rc1

Provider apache.kafka: 1.2.0rc1

Provider apache.livy: 3.6.0rc1

Provider arangodb: 2.3.0rc1

Provider celery: 3.4.0rc1

Provider cncf.kubernetes: 7.7.0rc1

Provider common.sql: 1.8.0rc1

Provider databricks: 4.6.0rc1

Provider datadog: 3.4.0rc1

Provider dbt.cloud: 3.4.0rc1

Provider docker: 3.8.0rc1

Provider elasticsearch: 5.1.0rc1

Provider github: 2.4.0rc1

Provider google: 10.10.0rc1

Provider http: 4.6.0rc1

Provider jenkins: 3.4.0rc1

Provider microsoft.azure: 8.0.0rc1

Provider openlineage: 1.1.1rc1

Provider papermill: 3.4.0rc1

Provider postgres: 5.7.0rc1

Provider presto: 5.2.0rc1

Provider samba: 4.3.0rc1

Provider slack: 8.2.0rc1

Provider snowflake: 5.1.0rc1

Provider trino: 5.4.0rc1

Committer

Activity

Lee-W commented on Oct 14, 2023

ealebed commented on Oct 14, 2023

pubudu91 commented on Oct 15, 2023

Owen-CH-Leung commented on Oct 15, 2023

utkarsharma2 commented on Oct 15, 2023

hankehly commented on Oct 15, 2023

Adaverse commented on Oct 16, 2023

aritra24 commented on Oct 16, 2023

saulius commented on Oct 16, 2023

moiseenkov commented on Oct 16, 2023

JDarDagran commented on Oct 16, 2023

Taragolis commented on Oct 16, 2023

oleksiidav commented on Oct 16, 2023

hussein-awala commented on Oct 16, 2023

asherkhb commented on Oct 16, 2023

Khrol commented on Oct 17, 2023

eladkal commented on Oct 17, 2023

Seokyun-Ha commented on Oct 19, 2023

krisfur commented on Oct 19, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions