Repository navigation
Status of testing Providers that were prepared on May 01, 2024 #39346
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
The guidelines on how to test providers can be found in
Verify providers by contributors
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
Provider airbyte: 3.8.0rc1
Provider amazon: 8.21.0rc1
- Added logging_config,snapstart,ephemeral_storage parameters to aws lambdacreatefunctionoperator (#39300): @gopidesupavan
- Fix bug in GlueJobOperator where consecutive runs fail when a local script file is used (#38960): @moritzsanne
Linked issues: - Update
is_authorized_custom_viewfrom auth manager to handle custom actions (#39167): @vincbeck
Linked issues: - Updated logic to allow AWS Batch Hook
get_job_descriptionretries to be more effective (#38998): @shahar1
Linked issues: - Amazon Bedrock - Model Throughput Provisioning (#38850): @ferruzzi
- Adding MSGraphOperator in Microsoft Azure provider (#38111): @dabla
- Allow importing the aws executors with a shorter path (#39093): @o-nikolas
Linked issues: - Remove flag from AWS auth manager to use it (#39033): @vincbeck
Provider apache.beam: 5.7.0rc1
- Bugfix to correct GCSHook being called even when not required with BeamRunPythonPipelineOperator (#38716): @zstrathe
Linked issues:
Provider apache.livy: 3.8.0rc1
- Fix bug in LivyOperator when its trigger times out (#38916): @mateuslatrova
- Fix 'polling_interval' parameter docs in LivyOperator (#38979): @mateuslatrova
- Update docstring
LivyOperatorretry_args and deferrable docs (#39266): @boraberke
Linked issues:
Provider apache.spark: 4.8.0rc2
Provider cncf.kubernetes: 8.2.0rc1
- Add missing informative logs in KPO trigger until container has not finished (#37546): @pankajkoti
- Fix templated env_vars field in
KubernetesPodOperatorto allow for compatibility withXComArgs(#39139): @nyoungstudios - Fix SparkKubernetesOperator when using initContainers (#38119): @ShelRoman
- Refresh properties on KubernetesPodOperator when k8s fails due to token expiration (#39325): @paolo-moriello
- Remove unnecessary validation from cncf provider. (#39238): @VShkaberda
- Moves airflow import in deprecated pod_generator to local (#39062): @potiuk
Linked issues: - KPO xcom sidecar PodDefault usage (#38951): @jedcunningham
Provider common.sql: 1.13.0rc1
- Add
sqlalchemy_urlproperty toDbApiHookclass (#38871): @rawwar
Linked issues: - Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715): @dabla
- Fix
DbApiHook.insert_rowswhenrowsis a generator (#38972): @vincbeck - Fix
update-common-sql-api-stubspre-commit check (#38915): @Taragolis - openlineage, snowflake: do not run external queries for Snowflake when (#39113): @mobuchowski
Provider databricks: 6.4.0rc1
- Add
DatabricksNotebookOperator(#39178): @pankajkoti - Add notification settings paramaters (#39175): @SubhamSinghal
- [FEAT] raise exception with main notebook error in DatabricksRunNowDeferrableOperator (#39110): @gaurav7261
Provider datadog: 3.6.0rc1
Provider dbt.cloud: 3.8.0rc1
- fix wrong payload set when reuse_existing_run set to True in DbtCloudRunJobOperator (#39271): @Lee-W
- migrate to dbt v3 api for project endpoints (#39214): @Lee-W
Provider fab: 1.1.0rc1
- Remove plugins permissions from Viewer role (#39254): @csp33
Linked issues: - Update
is_authorized_custom_viewfrom auth manager to handle custom actions (#39167): @vincbeck
Linked issues:
Provider google: 10.18.0rc1
- Add templated fields for Google Cloud Generative AI Operators (#39174): @CYarros10
- Add logic to handle on_kill for BigQueryInsertJobOperator when deferrable=True (#38912): @sunank200
- Create CloudComposerRunAirflowCLICommandOperator operator (#38965): @MaksYermak
- Add the deferrable mode to the Dataflow sensors (#37693): @e-galan
- Deferrable mode for Custom Training Job operators (#38584): @e-galan
- Enhancement for SSL-support in CloudSQLExecuteQueryOperator (#38894): @moiseenkov
- Create GKESuspendJobOperator and GKEResumeJobOperator operators (#38677): @MaksYermak
- Add support for role arn for aws creds in storage transfer job operators (#38911): @wlinamchurch
- Fix deferrable mode for DataflowTemplatedJobStartOperator and DataflowStartFlexTemplateOperator (#39018): @e-galan
- Fix batching for BigQueryToPostgresOperator (#39233): @moiseenkov
- Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. (#39230): @sunank200
- Fix GCSObjectExistenceSensor operator to return the same XCOM value in deferrable and non-deferrable mode (#39206): @VladaZakharova
- Bugfix BigQueryToMsSqlOperator (#39171): @moiseenkov
- fix: add retry logic in case of google auth refresh credential error (#38961): @dondaum
Linked issues: - fix: BigQueryCheckOperator skip value and error check in deferrable mode (#38408): @kacpermuda
Linked issues: - [OpenLineage] Fix datasets in GCSDeleteObjectsOperator (#39059): @kacpermuda
Linked issues: - [OpenLineage] Use prefixes instead of file paths for datasets in GCSToGCSOperator (#39058): @kacpermuda
Linked issues: - [OpenLineage] Fix datasets in GCSTimeSpanFileTransformOperator (#39064): @kacpermuda
Linked issues: - Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup (#39130): @sunank200
- Adding MSGraphOperator in Microsoft Azure provider (#38111): @dabla
- Apply PROVIDE_PROJECT_ID mypy workaround across Google provider (#39129): @potiuk
- Handle ImportError for KubernetesDeleteJobOperator in google provider (#39036): @vatsrahul1001
Linked issues:
Provider hashicorp: 3.7.0rc2
- Add assume_role_kwargs in hashicorp backend config (#39279): @pankajastro
- Expose AWS IAM missing param in Hashicorp secret (#38536): @pankajastro
Provider http: 4.11.0rc1
Provider jdbc: 4.3.0rc1
- Suppress jaydebeapi.Error when setAutoCommit or getAutoCommit is unsupported by JDBC driver (#38707): @dabla
- Add stacklevel into the
suppress_and_warnwarning (#39263): @Taragolis
Provider microsoft.azure: 10.1.0rc1
- add dns_config and diagnostics parameters to AzureContainerInstancesOperator (#39156): @Kylea650
Linked issues: - Add stacklevel into the
AzureSynapsePipelineHookdeprecation warnings (#39192): @Taragolis - Adding MSGraphOperator in Microsoft Azure provider (#38111): @dabla
- Pass through more parameters to AzureBlobFileSystem to mirror handlin… (#38747): @tomrutter
Linked issues: - Implement run-method on KiotaRequestAdapterHook and move logic away from triggerer to hook (#39237): @dabla
- Fix: Only quote the keys of the query_parameters in MSGraphOperator (#39207): @dabla
Provider odbc: 4.6.0rc1
- Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715): @dabla
Provider openai: 1.2.0rc1
- OpenAI Chat & Assistant hook functions (#38736): @nathadfield
- OpenAI Files & Vector Store Hooks (#39248): @nathadfield
Provider openlineage: 1.7.1rc1
Provider pagerduty: 3.7.0rc2
Provider pinecone: 2.0.0rc1
- Pinecone provider support for
pinecone-client>=3 (#37307): @rawwar
Linked issues: - Bump ruff to 0.3.3 (#38240): @hussein-awala
- Add comment about versions updated by release manager (#37488): @potiuk
Provider postgres: 5.11.0rc1
- Fix schema assigment in PostgresOperator (#39264): @eladkal
- Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715): @dabla
Provider qdrant: 1.1.0rc1
Provider salesforce: 5.7.0rc2
Provider slack: 8.7.0rc1
Provider snowflake: 5.5.0rc1
Provider teradata: 2.1.0rc1
- Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715): @dabla
All users involved in the PRs:
@mobuchowski @josix @mateuslatrova @kacpermuda @dabla @anush008 @jedcunningham @boraberke @pankajastro @Taragolis @gopidesupavan @hussein-awala @wlinamchurch @nathadfield @pankajkoti @potiuk @Kylea650
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
nathadfield commentedon May 1, 2024
@eladkal #39248 and #38736 tested and working correctly.
dabla commentedon May 1, 2024
I know there is still an issue with the MSGraphSensor in that sense that it’s not working correctly in an async way and sometimes an event loop closed error can occur. This PR will fix it so dunno how to proceed for the msgraph part with this RC?
wlinamchurch commentedon May 1, 2024
@eladkal #38911 tested and working correctly.
gopidesupavan commentedon May 1, 2024
@eladkal #39300 tested and working correctly.
anush008 commentedon May 1, 2024
Tested. Works as expected.
dondaum commentedon May 1, 2024
#38961 is good. ✅
Updating setup.cfg with RC provider packages on apache/airflow#39346
dabla commentedon May 2, 2024
Following 3 PR suffer from the same issues with the MSGraphSenor which has been fixed in #39304:
#38111
#39237
#39207
eladkal commentedon May 2, 2024
It's not merged. Only merged PR before release are included.
dabla commentedon May 2, 2024
I know Elad I just wanted to make you guys aware of it.
pankajastro commentedon May 2, 2024
Tested #39279 and #38536 looks good
eladkal commentedon May 2, 2024
If there is regression we can make RC2 then it will be included. What do you recommend ?
paolo-moriello commentedon May 2, 2024
#39325 tested 👍
Kylea650 commentedon May 2, 2024
Anybody with an Azure account able to test #39156? I can provide an example DAG I just don’t have access to my work Azure account right now and don’t want to block the release.
rawwar commentedon May 2, 2024
Pineconepinecone: 2.0.0rc1
PR: #37307 is has bugs.
Fix is being implemented in #39365
tomrutter commentedon May 2, 2024
Happy with #38747
dabla commentedon May 2, 2024
Yes an RC2 would be great.
pankajkoti commentedon May 2, 2024
Verified #39178 , works all fine
boraberke commentedon May 2, 2024
#39266 looks good.
zstrathe commentedon May 2, 2024
#38716 tested and verified #38713 is resolved
Updating setup.cfg with RC provider packages on apache/airflow#39346
nyoungstudios commentedon May 3, 2024
Tested #39139 and looks good!
Taragolis commentedon May 4, 2024
Tests my changes (most of them just cosmetic), look good
Lee-W commentedon May 6, 2024
for #38912, we might introduce a potential bug as commented in #36090 (comment)
pankajkoti commentedon May 6, 2024
Similar risk for #39130 and #39230
eladkal commentedon May 6, 2024
Thank you everyone. Providers are released.
Google, microsoft.azure and pinecone are excluded from this wave.
I invite everyone to help improve providers for the next release, a list of open issues can be found here.