Repository navigation
Status of testing Providers that were prepared on August 26, 2023 #33751
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
The guidelines on how to test providers can be found in
Verify providers by contributors
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
Provider amazon: 8.6.0rc1
- Added Amazon SageMaker Notebook hook and operators (#33219): @ellisms
- Add
deferrableoption toLambdaCreateFunctionOperator(#33327): @vincbeck - Add Deferrable mode to GlueCatalogPartitionSensor (#33239): @syedahsn
- Add
sql_hook_paramsparameter toS3ToSqlOperator(#33427): @alexbegg - Add
sql_hook_paramsparameter toSqlToS3Operator(#33425): @alexbegg - Add parameter to pass role ARN to
GlueJobOperator(#33408): @erdos2n - Add new RdsStartExportTaskOperator parameters (#33251): @covidium
- Fix bug in task logs when using AWS CloudWatch. Do not set
start_time(#33673): @vincbeck - Fix AWS Batch waiter failure state (#33656): @yermalov-here
- [bugfix] fix AWS Appflow waiter (#33613): @vandonr-amz
- bugfix: strip tags when falling back to update in
SageMakerEndpointOperator(#33487): @vandonr-amz - Remove non-public interface usage in EcsRunTaskOperator (#29447): @Taragolis
- Upgrade botocore/aiobotocore minimum requirements (#33649): @potiuk
- always push ECS task ARN to xcom in
EcsRunTaskOperator(#33703): @vandonr-amz - Use
boto3.clientlinked to resource meta instead of create new one for waiters (#33552): @Taragolis
Provider apache.beam: 5.2.2rc1
Provider apache.hive: 6.1.5rc1
- Bring back hive support for Python 3.11 (#32607): @potiuk
- Avoid importing pandas and numpy in runtime and module level (#33483): @hussein-awala
Provider apache.pinot: 4.1.3rc1
Provider celery: 3.3.3rc1
- Fix dependencies for celery and opentelemetry for Python 3.8 (#33579): @potiuk
- Make auth managers provide their own airflow CLI commands (#33481): @vandonr-amz
- Refactor Sqlalchemy queries to 2.0 style (Part 7) (#32883): @phanikumv
Provider cncf.kubernetes: 7.5.0rc1
- Add istio test, use curl /quitquitquit to exit sidecar, and some othe… (#33306): @Owen-CH-Leung
- Add
active_deadline_secondsparameter toKubernetesPodOperator(#33379): @amoghrajesh - Make cluster_context templated in KubernetesPodOperator (#33604): @mati-o
- Fix KubernetesPodOperator duplicating logs when interrupted (#33500): @fdemiane
- Fix 2.7.0 db migration job errors (#33652): @LipuFei
- Inspect container state rather than last_state when deciding whether to skip (#33702): @SamWheating
- Introducing class constant to make worker pod log lines configurable (#33378): @amoghrajesh
- Adding typing for KPO SCC objects (#33381): @amoghrajesh
- Refactor Sqlalchemy queries to 2.0 style (Part 7) (#32883): @phanikumv
Provider databricks: 4.4.0rc1
- Add Service Principal OAuth for Databricks. (#33005): @johnbrandborg
- Update docs in databricks.py - we use 2.1 now (#32340): @zbstof
Provider elasticsearch: 5.0.1rc1
- Add backward compatibility for elasticsearch<8 (#33281): @sunank200
- Fix urlparse schemaless-behaviour on Python 3.9+ (#33289): @potiuk
- Upgrade Elasticsearch to 8 (#33135): @Owen-CH-Leung
Provider exasol: 4.2.4rc1
Provider influxdb: 2.2.2rc1
Provider microsoft.azure: 6.3.0rc1
- Add AzureBatchOperator example (#33716): @Lee-W
- feat(providers/microsoft): add DefaultAzureCredential support to AzureContainerInstanceHook (#33467): @Lee-W
- Add DefaultAzureCredential auth for ADX service (#33627): @pankajastro
- Fix updating account url for WasbHook (#33457): @ephraimbuddy
- Fix Azure Batch Hook instantation (#33731): @potiuk
- Bump azure-kusto-data>=4.1.0 (#33598): @pankajastro
- Set logging level to WARNING (#33314): @arjunanan6
Provider mysql: 5.3.0rc1
- Add
init_commandparameter toMySqlHook(#33359): @alexbegg - Add
local_infileparameter toS3ToMySqlOperator(#33459): @benchoncy
Provider openlineage: 1.0.2rc1
- openlineage: don't run task instance listener in executor (#33366): @mobuchowski
- openlineage: do not try to redact Proxy objects from deprecated config (#33393): @mobuchowski
- openlineage: defensively check for provided datetimes in listener (#33343): @mobuchowski
- openlineage, trino: add OpenLineage support for Trino. (#32910): @JDarDagran
Provider oracle: 3.7.3rc1
Provider salesforce: 5.4.2rc1
Provider sftp: 4.6.0rc1
Provider slack: 8.0.0rc1
- Remove deprecated parts from Slack provider (#33557): @Taragolis
- Replace deprecated slack notification in provider.yaml with new one (#33643): @potiuk
- Avoid importing pandas and numpy in runtime and module level (#33483): @hussein-awala
Provider snowflake: 5.0.0rc1
- Remove deprecated
S3ToSnowflakeandSnowflakeToSlackoperators (#33558): @Taragolis - Set snowflake_conn_id on Snowflake Operators to avoid error (#33681): @SamWheating
- Bump
apache-airflow-providers-snowflakedue to breaking changes (#33615): @Taragolis
Provider trino: 5.3.0rc1
Provider yandex: 3.4.0rc1
All users involved in the PRs:
@amoghrajesh @mobuchowski @zbstof @johnbrandborg @vincbeck @JDarDagran @benchoncy @ellisms @SamWheating @pankajastro @alexbegg @sunank200 @zinal @LipuFei @functicons @hussein-awala @ephraimbuddy @Lee-W
@arjunanan6 @erdos2n @vandonr-amz @fdemiane @Owen-CH-Leung @phanikumv @Taragolis @mati-o @syedahsn @tiranux @covidium @yermalov-here @potiuk
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
Updating setup.cfg with RC provider packages on apache/airflow#33751
alexbegg commentedon Aug 26, 2023
Hi,
#33427, #33425, #33359 works as expected, even tested using an external database and all looks good.
[DO NOT MERGE] Test RC provider packages for apache/airflow#33751
alexbegg commentedon Aug 26, 2023
I also tested #33601 and it no longer has the ImportError warning, as expected
arjunanan6 commentedon Aug 26, 2023
I have just tested #33314, works as expected.
LipuFei commentedon Aug 26, 2023
I haved tested #33652, looks good.
johnbrandborg commentedon Aug 26, 2023
I have tested #33005 this morning. Messages indicate Service Principal Token is being used, and data was retrieved from a Databricks Workspace.
potiuk commentedon Aug 26, 2023
Checked all my changes. All look good!
ephraimbuddy commentedon Aug 26, 2023
Tested #33457 and it works fine
Owen-CH-Leung commentedon Aug 26, 2023
Tested #33135 and it works as expected. For #33306 , it's not working as expected so I have created another PR which just got merged.
#33779
pankajastro commentedon Aug 26, 2023
#33598 and #33627 looks good. thanks!
hussein-awala commentedon Aug 26, 2023
Checked #33483 and it is present in the different providers.
Lee-W commentedon Aug 27, 2023
Tested #33716 and #33467. Both works fine. Thanks!
amoghrajesh commentedon Aug 27, 2023
Tested my changes:
active_deadline_secondsparameter toKubernetesPodOperator#33379Things do not seem to have regressed and it works as expected. Ran a few dags to target the cncf provider too. It looks good to me.
Thanks!
mobuchowski commentedon Aug 27, 2023
Tested my changes, everything looks good.
zinal commentedon Aug 28, 2023
Hi! #29811 looks fine for me, processed the test dag successfully.
phanikumv commentedon Aug 28, 2023
#32883 looks good
JDarDagran commentedon Aug 28, 2023
#32910 works as expected.
yermalov-here commentedon Aug 28, 2023
tested #33656 and it looks good
ellisms commentedon Aug 28, 2023
tested #33219 and it looks good.
fdemiane commentedon Aug 28, 2023
tested #33500, works as expected
vandonr-amz commentedon Aug 28, 2023
good for me
vincbeck commentedon Aug 28, 2023
#33327 and #33673 work as expected
eladkal commentedon Aug 29, 2023
Thank you everyone.
microsoft.azureandcncf.kubernetesare excluded from rc1. I will cut rc2 shortly.The rest of the providers are released.
I invite everyone to help improve providers for the next release, a list of open issues can be found here.