Repository navigation
Status of testing Providers that were prepared on December 22, 2024 #45148
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
The guidelines on how to test providers can be found in
Verify providers by contributors
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
Provider airbyte: 5.0.0rc2
Provider alibaba: 3.0.0rc2
Provider amazon: 9.2.0rc2
- Add DMS Serverless Operators (#43988): @ellisms
- Add fail_on_file_not_exist option to SFTPToS3Operator (#44320): @Guaqamole
- Add wait_policy option to EmrCreateJobFlowOperator. (#44055): @adrian-adikteev
- Use
S3CopyObjectOperatorinexample_comprehend_document_classifier(#44160): @vincbeck - Remove unnecessary compatibility code in S3 asset import (#44714): @potiuk
- Remove AIP-44 from taskinstance (#44540): @potiuk
- Add do_xcom_push documentation in EcsRunTaskOperator (#44440): @leonidasefrem
- Move Asset user facing components to task_sdk (#43773): @Lee-W
- Set up JWT token authentication in Fast APIs (#42634): @vincbeck
- Bump to mypy-boto3-appflow and pass without
# type: ignore[arg-type](#44115): @jx2lee
Linked issues:
Provider apache.beam: 6.0.0rc2
- Removed deprecated code from provider Apach.beam (#44700): @ajitg25
Linked issues: - Fix deferrable mode for BeamRunPythonPipelineOperator (#44386): @MaksYermak
Provider apache.drill: 3.0.0rc2
- Remove provider deprecations in Apache Drill (#44575): @kunaljubce
Linked issues:
Provider apache.druid: 4.0.0rc2
- Remove provider deprecations in Apache Druid (#44765): @kunaljubce
Linked issues: - Rename execution_date to logical_date across codebase (#43902): @sunank200
Provider apache.hdfs: 4.7.0rc2
Provider apache.hive: 9.0.0rc2
- Remove deprecations from Apache hive Provider (#44715): @vatsrahul1001
- Rename execution_date to logical_date across codebase (#43902): @sunank200
Provider apache.kafka: 1.7.0rc2
Provider apache.livy: 4.0.0rc2
- Remove Provider Deprecations in Apache Livy (#44631): @jason810496
Linked issues:
Provider apache.spark: 5.0.0rc2
- Remove deprecated code from apache spark provider (#44567): @kunaljubce
Linked issues: - Fix failing mypy check on
main(#44191): @kaxil - spark-submit: replace
principlebyprincipal(#44150): @brouberol
Linked issues:
Provider apprise: 2.0.0rc2
Provider arangodb: 2.7.0rc2
- Added the ArangoDBCollectionOperator that executes collection operations in a ArangoDB database (#44676): @harjeevanmaan
Provider atlassian.jira: 3.0.0rc2
- Remove Provider Deprecations in Atlassian Jira (#44644): @jason810496
Linked issues:
Provider celery: 3.9.0rc2
- AIP-81 Move CLI Commands to directories according to Hybrid, Local and Remote (#44538): @bugraoz93
Linked issues: - Remove AIP-44 configuration from the code (#44454): @potiuk
Linked issues:
Provider cncf.kubernetes: 10.1.0rc2
- Add logging support for init containers in KubernetesPodOperator (#42498) (#43853): @mrk-andreev
- New Kubernetes Kueue Operators (#44568): @moiseenkov
- support grouping of log lines for KubernetesPodOperator (#44428): @karunpoudel
- Compare k8s executor against alias, not full ExecutorName repr (#44967): @o-nikolas
Linked issues: - Fix failing KubeExecutor tests due to #44710 (#44931): @amoghrajesh
Provider common.compat: 1.3.0rc2
- fix(providers/common/compat): add back add_input_dataset and add_output_dataset to NoOpCollector (#44681): @Lee-W
- Fix name of private function in compat provider (#44680): @jedcunningham
- Remove unnecessary compatibility code in S3 asset import (#44714): @potiuk
- Move Asset user facing components to task_sdk (#43773): @Lee-W
- Make AssetAliasEvent a class context.py (#44709): @uranusjr
- Move triggers to standard provider (#43608): @gopidesupavan
Linked issues:
Provider common.io: 1.5.0rc2
- feat: add OpenLineage support for transfer operators between gcs and local (#44417): @kacpermuda
Linked issues: - Move Asset user facing components to task_sdk (#43773): @Lee-W
- Migrate pickled data & change XCom value type to JSON (#44166): @kaxil
Provider common.sql: 1.21.0rc2
- [Added output_processor parameter to SQLQueryOperator and fixed bug with return_single_query_results handler when None is passed as split_statements (Added output_processor parameter to SQLQueryOperator and fixed bug with return_single_query_results handler when None is passed as split_statements #44781)](https://github.com/apache/airflow/
- Moved common SQL handler methods of common-sql-provider into dedicated module (#43747): @dabla
- Fix static checks in common SQL hooks (#44930): @shahar1
Linked issues: - Remove Provider Deprecations in Common SQL (#44645): @jason810496
Linked issues:
Provider databricks: 7.0.0rc2
- Remove deprecations from Databricks Provider (#44566): @vatsrahul1001
- [FIX] Fixed databricks repair run deferrable (#44213): @raghvendra-singh1
- fix(providers/databricks): remove additional argument passed to repair_run (#44140): @Lee-W
- Added job_clusters as a templated parameter to CreateDatabricksWorkflowOperator (#45022): @kunaljubce
Linked issues: - Upgrade databricks provider dependency (#43272): @dcmshi
- Rename execution_date to logical_date across codebase (#43902): @sunank200
Provider dbt.cloud: 4.0.0rc2
- Remove Provider Deprecations in DBT (#44638): @jason810496
Linked issues: - utilize map_index for deterministic generation of OpenLineage's run_id (#43936): @mobuchowski
- Remove commented breakpoint in dbt provider (#44163): @dstandish
- Rename execution_date to logical_date across codebase (#43902): @sunank200
Provider docker: 4.0.0rc2
- Remove Provider Deprecations in Docker (#44583): @jscheffl
- Fix docker documentation auth url (#44112): @gopidesupavan
Provider elasticsearch: 6.0.0rc2
- Remove Provider Deprecations in Elasticsearch (#44629): @jason810496
Linked issues: - Rename execution_date to logical_date across codebase (#43902): @sunank200
Provider google: 12.0.0rc2
- Google provider delete deprecated reaching removal date (December 2024) (#45084): @olegkachur-e
- Add Google Vertex AI Feature Store - Feature View Sync Operators, Sensor (#44891): @CYarros10
- Introduce GCP translation (V3), translate document providers (#44971): @olegkachur-e
- Introduce gcp advance (V3) API translate native models operators (#44627): @olegkachur-e
- Support multiple SQL queries in Dataproc SQL job (#44890): @amirmor1
- feat: add OpenLineage support for BigQuery Create Table operators (#44783): @kacpermuda
- feat: add OpenLineage support for S3ToGCSOperator (#44426): @kacpermuda
- feat: automatically inject OL info into spark job in DataprocSubmitJobOperator (#44477): @kacpermuda
- Implement AlloyDB operators: create/update/delete clusters (#45027): @moiseenkov
- Fix MetastoreHivePartitionSensor failing due to duplicate aliases (#45001): @CYarros10
- Fix failing OpenLineage emition for InsertBigQueryOperator (#44650): @spapi17
- BigQueryInsertJobOperator: log transient error and check job state before marking task as success (#44279): @pankajastro
- FIX make
CloudBatchSubmitJobOperatorfail when job fails (#44425): @SuccessMoses
Linked issues: - feat: add OpenLineage support for transfer operators between gcs and local (#44417): @kacpermuda
Linked issues: - chore: remove deprecated BigQuery facets from OpenLineage utils (#44838): @kacpermuda
- New Kubernetes Kueue Operators (#44568): @moiseenkov
- Fix system test for Dataform operators (#44729): @MaksYermak
- Deprecate VertexAI PaLM text generative model (#44719): @MaksYermak
- Show prominent warning for deprecations in docs (#44479): @omkar-foss
- chore: unify handling of gcs paths in OpenLineage processes (#44410): @kacpermuda
Provider hashicorp: 4.0.0rc2
- Removed deprecated code from hashicorp provider (#44598): @Prab-27
Linked issues:
Provider http: 5.0.0rc2
- Remove Provider Deprecations in HTTP (#44542): @jscheffl
- [Providers/HTTP] Add adapter parameter to HttpHook to allow custom requests adapters (#44302): @jieyao-MilestoneHub
Linked issues: - Rename execution_date to logical_date across codebase (#43902): @sunank200
- Bump
aiohttpto3.11.1(#44036): @kaxil
Provider jdbc: 5.0.0rc2
- Remove Provider Deprecations in JDBC (#44662): @jscheffl
- Made get_conn in JdbcHook threadsafe to avoid OSError: JVM is already started (#44718): @dabla
- Suppress JException when get_autocommit and set_autocommit methods aren't supported on JDBC driver (#43786): @dabla
Provider jenkins: 4.0.0rc2
- Remove Provider Deprecations in Jenkins (#44630): @jason810496
Linked issues:
Provider microsoft.azure: 12.0.0rc2
- Remove Provider Deprecations in Microsoft-AZURE (#44763): @vatsrahul1001
- Added test-case for callable values in path and query parameters of MSGraphAsyncOperator (#43799): @dabla
- PowerBIDatasetRefreshOperator should fail when refresh fails (#44696): @Ohashiro
- Move triggers to standard provider (#43608): @gopidesupavan
Linked issues: - Remove XCom pickling (#43905): @kaxil
- Update path of example dags in docs (#45069): @eladkal
- Avoid 1.1.8 version of msgraph-core (#45044): @potiuk
- Added MS Graph connection type (#45006): @dabla
Provider microsoft.mssql: 4.0.0rc2
- Remove Provider Deprecations in Microsoft-MSSQL (#44762): @jscheffl
- Support connection extra parameters in MsSqlHook (#44310): @jx2lee
Linked issues:- Linked Issue #43798: @ilarionkuleshov
Provider microsoft.psrp: 3.0.0rc2
Provider microsoft.winrm: 3.7.0rc2
Provider mongo: 5.0.0rc2
- Remove Provider Deprecations in Mongo (#44632): @jason810496
Linked issues:
Provider mysql: 6.0.0rc2
- Remove Provider Deprecations in MySQL (#44665): @jscheffl
- Fix error file not found. tmp file is deleted before inserting rows to DB in VerticaToMySQLOperator bulk (#44028): @bareketamir
- Add basic asyncio support (#43944): @dstandish
Provider openlineage: 2.0.0rc2
- Remove Provider Deprecations in OpenLineage (#44636): @jason810496
Linked issues: - add clear_number to OpenLineage's dagrun-level event run id generation (#44617): @mobuchowski
- utilize map_index for deterministic generation of OpenLineage's run_id (#43936): @mobuchowski
- feat: automatically inject OL info into spark job in DataprocSubmitJobOperator (#44477): @kacpermuda
- add basic system tests for OpenLineage (#43643): @mobuchowski
- Move Asset user facing components to task_sdk (#43773): @Lee-W
- Rename execution_date to logical_date across codebase (#43902): @sunank200
Provider opensearch: 1.6.0rc2
Provider oracle: 4.0.0rc2
Provider pagerduty: 4.0.0rc2
- Remove deprecated code from Pagerduty provider (#44653): @Prab-27
Linked issues:
Provider postgres: 6.0.0rc2
- Remove Provider Deprecations in Postgres (#44705): @jscheffl
- Add basic asyncio support (#43944): @dstandish
Provider presto: 5.8.0rc2
Provider redis: 4.0.0rc2
- Remove Provider Deprecations in Redis (#44633): @jason810496
Linked issues:
Provider sendgrid: 4.0.0rc2
- Remove Provider Deprecations in SendGrid (#44637): @jason810496
Linked issues:
Provider sftp: 5.0.0rc2
- Remove deprecations from SFTP Provider (#44740): @vatsrahul1001
- feat: retrieve sftp file attrs onces instead multiple time (#44625): @dondaum
- Add host_proxy_cmd parameter to SSHHook and SFTPHook (#44565): @ajitg25
Provider slack: 9.0.0rc2
Provider snowflake: 6.0.0rc2
- Remove deprecations from Snowflake Provider (#44756): @vatsrahul1001
- [Snowflake] enable client_store_temporary_credential for snowflake provider (#44431): @yzliao
- Allow
json_result_force_utf8_encodingspecification inproviders.snowflake.hooks.SnowflakeHookextra dict (#44264): @ttzhou - Make host/port configurable for Snowflake connections (#44079): @whummer
Provider sqlite: 4.0.0rc2
- Remove Provider Deprecations in Sqlite (#44707): @jscheffl
- Add basic asyncio support (#43944): @dstandish
Provider ssh: 4.0.0rc2
- Remove Provider Deprecations in SSH (#44544): @jscheffl
- Add host_proxy_cmd parameter to SSHHook and SFTPHook (#44565): @ajitg25
- Remove XCom pickling (#43905): @kaxil
Provider standard: 0.0.3rc2
- Remove Provider Deprecations in Standard (#44541): @jscheffl
- Add backward compatibility check for StartTriggerArgs import in filesystem sensor (#44458): @kunaljubce
Linked issues: - Remove Pydanitc models introduced for AIP-44 (#44552): @potiuk
Linked issues: - Deferrable sensors can implement sensor timeout (#33718): @dstandish
- Remove AIP-44 code from renderedtifields.py (#44546): @potiuk
Linked issues: - Remove AIP-44 from taskinstance (#44540): @potiuk
- Move
LatestOnlyOperatoroperator to standard provider. (#44309): @hardeybisey
Linked issues: - Remove AIP-44 configuration from the code (#44454): @potiuk
Linked issues: - Move external task sensor to standard provider (#44288): @kunaljubce
Linked issues: - Move triggers to standard provider (#43608): @gopidesupavan
Linked issues:
Provider tableau: 5.0.0rc2
Provider telegram: 4.7.0rc2
Provider teradata: 3.0.0rc2
Provider trino: 6.0.0rc2
- Remove Provider Deprecations in Trino (#44717): @Prab-27
Linked issues: - Rename execution_date to logical_date across codebase (#43902): @sunank200
Provider vertica: 4.0.0rc2
Provider weaviate: 3.0.0rc2
Provider yandex: 4.0.0rc2
All users involved in the PRs:
@kaxil @raghvendra-singh1 @uranusjr @Lee-W @mobuchowski @leonidasefrem @adrian-adikteev @dstandish @omkar-foss @HaKkaz @jedcunningham @shahar1 @potiuk @mrk-andreev @hardeybisey @o-nikolas @Tushar4432 @sunank200
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
jscheffl commentedon Dec 22, 2024
Checked the deprecations which were my main contributions and all seem to be OK.
potiuk commentedon Dec 22, 2024
All my removals checked. All looks good !
markhatch commentedon Dec 23, 2024
Tested WebHDFS hook with mTLS - looks good.
amoghrajesh commentedon Dec 23, 2024
#44931 works as expected, niko reworked it and even that fix works as expected.
moiseenkov commentedon Dec 23, 2024
Hi,
#44568 and #45027 work as expected
Ohashiro commentedon Dec 23, 2024
Hello
I think #44696 works well
hardeybisey commentedon Dec 23, 2024
Hi, #44309 works as expected.
kacpermuda commentedon Dec 23, 2024
All my changes look good and produce expected results with some sample DAGs.
dondaum commentedon Dec 23, 2024
Tested #44625. All good.
One thing. It seems that some contributors, including myself, are somehow not tagged correctly. That's why I didn't get a notification.
jx2lee commentedon Dec 23, 2024
#44115
#44310
works fine!
potiuk commentedon Dec 23, 2024
Interesting seems it's only @dondaum @yzliao @HaKkaz @hardeybisey - interesting why, as it's not something that we have control over.
potiuk commentedon Dec 23, 2024
Likely some intermittent GitHub Issue - now all the users are tagged properly when mentioned....
gopidesupavan commentedon Dec 23, 2024
Verified my changes all looks good.
yzliao commentedon Dec 23, 2024
Verified Snowflake provider and all LGTM. My change #44431 is good to go.
pankajastro commentedon Dec 24, 2024
#44279 looks good.
olegkachur-e commentedon Dec 24, 2024
HaKkaz commentedon Dec 26, 2024
#44040 looks good.
bugraoz93 commentedon Dec 26, 2024
I verified my changes (in #44204) and they look good. Thanks!
eladkal commentedon Dec 26, 2024
Thank you everyone. Providers are released.
I invite everyone to help improve providers for the next release, a list of open issues can be found here.
dabla commentedon Dec 27, 2024
Following have al been tested and are working as expected.
Moved common SQL handler methods of common-sql-provider into dedicated module (#43747): @dabla
Made get_conn in JdbcHook threadsafe to avoid OSError: JVM is already started (#44718): @dabla
Suppress JException when get_autocommit and set_autocommit methods aren't supported on JDBC driver (#43786): @dabla
Added test-case for callable values in path and query parameters of MSGraphAsyncOperator (#43799): @dabla
Added MS Graph connection type (#45006): @dabla
mrk-andreev commentedon Dec 28, 2024
Add logging support for init containers in KubernetesPodOperator (#42498) (#43853): @mrk-andreev verified
Apache Airflow _ Add logging support for init containers in KubernetesPodOperator.pdf