Improve Databricks operators with query tags

[zhaorui2022]

Description

Databricks supports query tags which allows us to track where the source of a query is on Databricks UI. Reference: https://docs.databricks.com/aws/en/sql/user/queries/query-tags

Currently when a query is sent to Databricks, on Databricks UI, the only info we can see is the query is from Airflow, but there is no other metadata like airflow dag id, task id, run id and etc.

Use case/motivation

With query tags instrumented, it will be easy for organizations to track and do Databricks cost contribution across different Airflow DAGs.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[eladkal]

cc @moomindani @mwojtyczka I think this should be easy to implement?

[shashbha14]

I'd like to work on this if it's still available.

[eladkal]

It is. If you have experience with databricks go for it.
This needs to be tested and validated against real databricks env

[Shaan-alpha]

I would like to work on this issue! I will submit a PR with the required changes soon.