Repository navigation
Status of testing Providers that were prepared on June 18, 2021 #16456
Description
I have a kind request for all the contributors to the latest provider packages release.
Could you help us to test the RC versions of the providers and let us know in the comment,
if the issue is addressed there.
Providers that need testing
Those are providers that require testing as there were some substantial changes introduced:
Provider airbyte: 2.0.0rc2
- Add test_connection method to Airbyte hook (#16236): @msumit
- Fix hooks extended from http hook (#16109): @msumit
Provider amazon: 2.0.0rc2
- read timestamp from Cloudwatch events (#15173): @codenamestif
- remove retry for now (#16150): @zachliu
- Remove the
not-allow-trailing-slashrule on S3_hook (#15609): @Isaacwhyuenac - Add support of capacity provider strategy for ECSOperator (#15848): @codenamestif
- Update copy command for s3 to redshift (#16241): @sunki-hong
- Make job name check optional in SageMakerTrainingOperator (#16327): @olivermeyer
- Add AWS DMS replication task operators (#15850): @Visya
- Fix S3 Select payload join (#16189): @TAKEDA-Takashi
- Fix spacing in AwsBatchWaitersHook docstring (#15839): @jlaneve
- MongoToS3Operator failed when running with a single query (not aggregate pipeline) (#15680): @amatellanes
- fix: AwsGlueJobOperator change order of args for load_file (#16216): @avocadomaster
- Fix S3ToFTPOperator (#13796): @JavierLopezT
Provider apache.spark: 2.0.0rc2
Provider cncf.kubernetes: 2.0.0rc2
- Add KPO pod-template-file jinja template support. (#15942): @Dr-Denzy
- Save pod name to xcom for KubernetesPodOperator (#15755): @junnplus
- Bug Fix Pod-Template Affinity Ignored due to empty Affinity K8S Object (#15787): @jpyen
- Bug Pod Template File Values Ignored (#16095): @jpyen
- Fix issue with parsing error logs in the KPO (#15638): @dimberman
- Fix unsuccessful KubernetesPod final_state call when
is_delete_operator_pod=True(#15490): @MatthewRBruce
Provider dingding: 2.0.0rc2
Provider discord: 2.0.0rc2
Provider docker: 2.0.0rc2
Provider elasticsearch: 2.0.1rc1
- Remove support jinja templated log_id in elasticsearch (#16465): @jedcunningham
- Support remote logging in elasticsearch with filebeat 7 (#14625): @jedcunningham
- Support non-https elasticsearch external links (#16489): @jedcunningham
- Fix external elasticsearch logs link (#16357): @jedcunningham
- Fix elasticsearch external log link with json_format (#16467): @jedcunningham
Provider google: 4.0.0rc2
- Move plyvel to google provider extra (#15812): @dstandish
- Fixes AzureFileShare connection extras (#16388): @potiuk
- Add extra links for google dataproc (#10343): @yesemsanthoshkumar
- Add link to Oracle Connection Docs (#15632): @sunkickr
- pass wait_for_done parameter down to _DataflowJobsController (#15541): @dejii
- Update Google Ads hook (#15266): @jacobhjkim
- Implement BigQuery Table Schema Update Operator (#15367): @thejens
- Add BigQueryToMsSqlOperator (#15422): @subkanthi
- Fix: GCS To BigQuery source_object (#16160): @tegardp
- FIX: unnecessary downloads in GCSToLocalFilesystemOperator (#16171): @p-kachalov
- Fix bigquery type error when export format is parquet (#16027): @sunki-hong
- Fix argument ordering and type of bucket and object (#15738): @sjvanrossum
- Fix sql_to_gcs docstring lint error (#15730): @natanweinberger
- Ensure
mysql_to_gcsfully compatible with MySQL and BigQuery fordatetime-related values (#15026): @tianjianjiang - Fix deprecation warnings location in google provider (#16403): @ashb
Provider hashicorp: 2.0.0rc2
Provider http: 2.0.0rc2
- Update SimpleHttpOperator to take auth type object (#15605): @fredthomsen
- HttpHook. Use request factory and respect defaults (#14701): @ngaranko
Provider microsoft.azure: 3.0.0rc2
- Fixes AzureFileShare connection extras (#16388): @potiuk
- Add link to Oracle Connection Docs (#15632): @sunkickr
- Fix WasbHook.delete_file broken when using prefix (#15637): @monti-python
- Fix colon spacing in AzureDataExplorerHook docstring (#15841): @jlaneve
- fix wasb remote logging when blob already exists (#16280): @flolas
Provider odbc: 2.0.0rc2
- OdbcHook returns None. Related to #15016 issue. (#15510): @Goodkat
- Fix OdbcHook handling of port (#15772): @dstandish
Provider opsgenie: 2.0.0rc2
Provider oracle: 2.0.0rc2
- Add optional result handler to database hooks (#15581): @malthe
- [Oracle] Add port to DSN (#15589): @malthe
- Add link to Oracle Connection Docs (#15632): @sunkickr
Provider papermill: 2.0.0rc2
Provider plexus: 2.0.0rc2
Provider postgres: 2.0.0rc2
- PostgresHook: deepcopy connection to avoid mutating connection obj (#15412): @zhzhang
- Avoid passing
aws_conn_idas conn_args forpsycopg2.connect(#16100): @gabrielsyapse
Provider samba: 2.0.0rc2
Provider sftp: 2.0.0rc2
Provider slack: 4.0.0rc2
Provider snowflake: 2.0.0rc2
- Add
template_fieldstoS3ToSnowflakeoperator (#15926): @nlecoy - Allow S3ToSnowflakeOperator to omit schema (#15817): @uranusjr
- Added ability for Snowflake to attribute usage to Airflow by adding an application parameter (#16420): @sfc-gh-madkins
- fix: restore parameters support when sql passed to SnowflakeHook as str (#16102): @grassten
Provider ssh: 2.0.0rc2
Providers for which previous RC was fully tested (thanks @levyitay !)
Provider qubole: 2.0.0rc2
- Qubole Hook Does Not Support 'include_headers' (#15598): @levyitay
- Feature qubole hook support headers (#15683): @levyitay
- Feature qubole hook support headers (#15615): @levyitay
Providers that do not need testing
Those are providers that were either doc-only or had changes that do not require testing.
- Provider apache.beam: 3.0.0rc2
- Provider apache.cassandra: 2.0.0rc2
- Provider apache.druid: 2.0.0rc2
- Provider apache.hdfs: 2.0.0rc2
- Provider apache.hive: 2.0.0rc2
- Provider apache.kylin: 2.0.0rc2
- Provider apache.livy: 2.0.0rc2
- Provider apache.pig: 2.0.0rc2
- Provider apache.pinot: 2.0.0rc2
- Provider apache.sqoop: 2.0.0rc2
- Provider asana: 1.0.0rc2
- Provider celery: 2.0.0rc2
- Provider cloudant: 2.0.0rc2
- Provider databricks: 2.0.0rc2
- Provider datadog: 2.0.0rc2
- Provider exasol: 2.0.0rc2
- Provider facebook: 2.0.0rc2
- Provider ftp: 2.0.0rc2
- Provider grpc: 2.0.0rc2
- Provider imap: 2.0.0rc2
- Provider jdbc: 2.0.0rc2
- Provider jenkins: 2.0.0rc2
- Provider jira: 2.0.0rc2
- Provider microsoft.mssql: 2.0.0rc2
- Provider microsoft.winrm: 2.0.0rc2
- Provider mongo: 2.0.0rc2
- Provider mysql: 2.0.0rc2
- Provider neo4j: 2.0.0rc2
- Provider openfaas: 2.0.0rc2
- Provider pagerduty: 2.0.0rc2
- Provider presto: 2.0.0rc2
- Provider redis: 2.0.0rc2
- Provider salesforce: 3.0.0rc2
- Provider segment: 2.0.0rc2
- Provider sendgrid: 2.0.0rc2
- Provider singularity: 2.0.0rc2
- Provider sqlite: 2.0.0rc2
- Provider tableau: 2.0.0rc2
- Provider telegram: 2.0.0rc2
- Provider trino: 2.0.0rc2
- Provider vertica: 2.0.0rc2
- Provider yandex: 2.0.0rc2
- Provider zendesk: 2.0.0rc2
Reacted by Jacob Kim and duyet
Activity
potiuk commentedon Jun 15, 2021
This would be great it we could have confirmation for most of the providers by Friday afternoon, so that I could release them then :)
potiuk commentedon Jun 15, 2021
@jmelot Also a kind request for you - it's the first release of Asana provider, so if you could test it as well and let us know in the comment, it would be great: https://pypi.org/project/apache-airflow-providers-asana/1.0.0rc1
zhzhang commentedon Jun 15, 2021
Tested postgres connection changes on a Resdhift cluster, looks good to me.
Goodkat commentedon Jun 15, 2021
Tested odbc connection, it works as expected with boolean parameters now:
#15016 (comment)
jmelot commentedon Jun 15, 2021
Tested Asana provider, all seems to be well!
nlecoy commentedon Jun 16, 2021
I've been using the fix of #15926 for work and everything looks fine! Also, I tested #15817 and its working too.
pavelhlushchanka commentedon Jun 16, 2021
The package description says that the provider can be installed on top of existing airflow 2.*. I'm installing amazon provider and it requires airflow 2.1.
potiuk commentedon Jun 16, 2021
Yep. We have an enty in changelog about it with warning. Do you think it's not enough ?
pavelhlushchanka commentedon Jun 16, 2021
I tested #15173 and #15848. Works fine.
pavelhlushchanka commentedon Jun 16, 2021
@potiuk I believe it's a bit confusing. If such info is added, then it should be precise. But it's up to you.
potiuk commentedon Jun 16, 2021
I will ask others what they think. I am inclined to release new rc for that together with elasticsearch (but let's continue testing).
amatellanes commentedon Jun 16, 2021
I tested #15680 and it works fine 👍
levyitay commentedon Jun 16, 2021
tested Qubole changes, looks good
flolas commentedon Jun 16, 2021
Tested #16280 in prod env with k8s, it works fine.
9 remaining items
MatthewRBruce commentedon Jun 21, 2021
Tested #15490 - Fix unsuccessful KubernetesPod final_state call when is_delete_operator_pod=True. Works as expected.
olivermeyer commentedon Jun 21, 2021
#16327 works as expected.
monti-python commentedon Jun 21, 2021
#15637 works as expected as well
natanweinberger commentedon Jun 21, 2021
#15730 is just a whitespace linting fix 👍
However, #15728 and #15667 touched the same file around the same time (also for minor changes) and aren't listed here, should they be included?
potiuk commentedon Jun 21, 2021
Auto-apply is specifically excluded as it was a change that applied to all providers (and main reason for this backwards-incompatible release) but it has been tested before. The other one is a docstring only (and those are excluded from changelog/testing).
jedcunningham commentedon Jun 21, 2021
Tested elasticsearch 2.0.1rc1 (note: rc1, not 2 as linked in the description), looks good. Do note that some of the fixes/features only work with core changes coming in 2.1.1, but the new provider does still work with earlier 2.x versions.
potiuk commentedon Jun 21, 2021
Thanks @jedcunningham ! corrected the link to keep good track of it.
tianjianjiang commentedon Jun 22, 2021
@potiuk I've tried to allocate some resource for testing #15026 but unfortunately not happening soon enough. However, at least the situation I've experienced before is coded in https://github.com/apache/airflow/pull/15026/files#diff-ba677556962f02c90a0e6933525d9d3f795e2d44a3d2229de1db9d1b68a067eb and since the datetime-related change is abstract while its upstream has only one change set of docstring lint (#15730), it is highly likely #15026 will work.
potiuk commentedon Jun 22, 2021
Kind ping again. Last chance to do some testing before we release.
@JavierLopezT @mmenarguezpear @junnplus @codenamestif @gabrielsyapse @uranusjr @Dr-Denzy @jpyen @jacobhjkim @zachliu @thejens @p-kachalov @sunki-hong @sjvanrossum @subkanthi @avocadomaster @jlaneve @sunkickr @malthe @yesemsanthoshkumar @Visya @andrewgodwin @dimberman @msumit @grassten @pgillet @dstandish @sfc-gh-madkins @ngaranko @ashb @tegardp
JavierLopezT commentedon Jun 22, 2021
Hello. Regarding #13796, I did make a quick test before merging the PR (the file was there and was not empty) but I can't make a deeper test due to health reasons. I am sorry
ngaranko commentedon Jun 22, 2021
#14701 works as expected.
potiuk commentedon Jun 22, 2021
No worries! thanks for your contributions so far @JavierLopezT !
potiuk commentedon Jun 23, 2021
Callng for last vote so last chance to test before we release.
@mmenarguezpear @junnplus @codenamestif @gabrielsyapse @uranusjr @Dr-Denzy @jpyen @jacobhjkim @zachliu @thejens @p-kachalov @sunki-hong @sjvanrossum @subkanthi @avocadomaster @jlaneve @sunkickr @malthe @yesemsanthoshkumar @Visya @andrewgodwin @dimberman @msumit @grassten @pgillet @dstandish @sfc-gh-madkins @ashb @tegardp