Repository navigation
Status of testing Providers that were prepared on November 30, 2021 #19895
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you help us to test the RC versions of the providers and let us know in the comment,
if the issue is addressed there.
Providers that need testing
Those are providers that require testing as there were some substantial changes introduced:
Provider amazon: 2.5.0rc1
- Adding support for using client_type API for interacting with EC2 and support for using filters (#9011): @VijayantSoni
- Do not check for S3 key before attempting download (#19504): @dstandish
- MySQLToS3Operator actually allow writing parquet files to s3. (#19094): @guotongfei
- Amazon provider remove deprecation, second try (#19815): @uranusjr
- Catch AccessDeniedException in AWS Secrets Manager Backend (#19324): @john-jac
Provider apache.druid: 2.1.0rc1
- Fix DruidOperator rendering problem (#19420): @humit0
- Adding missing
max_ingestion_timeto DruidOperator docstring (#18693): @josh-fell - Add guide for Apache Druid operators (#18527): @Jasonnor
Provider apache.hdfs: 2.2.0rc1
Provider apache.hive: 2.1.0rc1
Provider apache.spark: 2.0.2rc1
Provider cncf.kubernetes: 2.2.0rc1
- Added namespace as a template field in the KPO. (#19718): @ReadytoRocc
- Decouple name randomization from name kwarg (#19398): @ferruzzi
- Checking event.status.container_statuses before filtering (#19713): @ulisesojeda
- Coalesce
extraparams to None in KubernetesHook (#19694): @dstandish - Change to correct type in KubernetesPodOperator (#19459): @DorConnecteam
Provider databricks: 2.1.0rc1
- Databricks: add more methods to represent run state information (#19723): @alexott
- Databricks - allow Azure SP authentication on other Azure clouds (#19722): @alexott
- Databricks: allow to specify PAT in Password field (#19585): @alexott
- Databricks jobs 2.1 (#19544): @christophergrant
- Update Databricks API from 2.0 to 2.1 (#19412): @Mokubyow
- Authentication with AAD tokens in Databricks provider (#19335): @alexott
- Fixup string concatenations (#19099): @blag
The Databricks provider will be removed from this round due to #20036.
Provider google: 6.2.0rc1
- Added wait mechanizm to the DataprocJobSensor to avoid 509 errors when Job is not available (#19740): @lwyszomi
- Add support in GCP connection for reading key from Secret Manager (#19164): @keze
- Add dataproc metastore operators (#18945): @wojsamjan
- Add support of
pathparameter for GCloud Storage Transfer Service operators (#17446): @eskarimov - Move
bucket_namevalidation out of__init__in Google Marketing Platform operators (#19383): @josh-fell - Create dataproc serverless spark batches operator (#19248): @MaksYermak
- updates pipeline_timeout CloudDataFusionStartPipelineOperator (#18773): @matsavage
- Support impersonation_chain parameter in the GKEStartPodOperator (#19518): @lwyszomi
- Fix badly merged impersonation in GKEPodOperator (#19696): @potiuk
Provider influxdb: 1.1.0rc1
Provider microsoft.azure: 3.4.0rc1
- Remove unnecessary connection form customizations in Azure (#19595): @josh-fell
- Update Azure modules to comply with AIP-21 (#19431): @josh-fell
- Remove
hostfrom hidden fields inWasbHook(#19475): @josh-fell - use DefaultAzureCredential if login not provided for Data Factory (#19079): @sdebruyn
- Fix argument error in AzureContainerInstancesOperator (#19668): @chinwobble
Provider postgres: 2.4.0rc1
- 19489 - Pass client_encoding for postgres connections (#19827): @subkanthi
- Amazon provider remove deprecation, second try (#19815): @uranusjr
Provider salesforce: 3.4.0rc1
Provider sftp: 2.3.0rc1
Provider snowflake: 2.3.1rc1
Providers that do not need testing
We also had a number of providers that had documentation-only change (mostly around examples). Those need no testing.
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
harryplumer commentedon Nov 30, 2021
snowflake 2.3.1rc1 looks good!
eskarimov commentedon Dec 1, 2021
#17446 works as expected
lwyszomi commentedon Dec 1, 2021
google 6.2.0rc1 works fine
Aakcht commentedon Dec 1, 2021
#19711 and #19777 work as expected, so waiting for the release :)
raphaelauv commentedon Dec 1, 2021
Checking event.status.container_statuses before filtering (#19713)-> workchinwobble commentedon Dec 3, 2021
#19668 tested and working with
apache-airflow-providers-microsoft-azure==3.4.0rc1josh-fell commentedon Dec 3, 2021
Verified #18693, #19431, #19475, #19595, and #19383
Thanks again @potiuk!
potiuk commentedon Dec 3, 2021
Thanks for those who tested. Today is last day to do some testing :)
uranusjr commentedon Dec 3, 2021
cc @dimon222 for #19815
alexott commentedon Dec 3, 2021
I have capacity to test only over the weekend...
subkanthi commentedon Dec 4, 2021
Tested #19489 and #19356
potiuk commentedon Dec 4, 2021
@alexott - good we need to more votes so still some time :)
alexott commentedon Dec 4, 2021
Tested #19722, #19585, #19335, #19412.
Unfortunately there is a logical error in calculation of the expiration time that may lead to authentication error when executing long-running jobs, just submitted a very small PR to fix it: #20036 - it would be nice to include it into release
potiuk commentedon Dec 4, 2021
Thanks!
I can remove it from the release and make ad-hoc rc2 for databricks only. This is how we deal with it when we find problems with a candidate provider.
humit0 commentedon Dec 5, 2021
Tested #19420
potiuk commentedon Dec 6, 2021
Thanks everyone who helped. I am preparing Databricks relase separately now!