Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup

Apache Airflow version

2.10.4

If "Other Airflow 2 version" selected, which one?

No response

What happened?

While creating workflow in Databricks using DatabricksWorkflowTaskGroup, I have a situation to pass some parameters to spark submit which are passed as jinja template.

Sample task creation code with DatabricksWorkflowTaskGroup,

databricks_workflow = DatabricksWorkflowTaskGroup(
            group_id="test_wf",
            databricks_conn_id=DATABRICKS_CONNECTION_ID,
            add_suffix_on_collision=True,
            job_clusters=job_clusters_spec,
        )
        
with databricks_workflow:
    task1 = build_task_1(dag)
    task2 = build_task_2(dag)
    task1 >> task2

From the above sample workflow creation code, build_task_1 and build_task_2 returns task which is my custom build operator on extending DatabricksTaskBaseOperator.

Sample code for custom build operator,

class CustomDatabricksWorkflowTaskOperator(DatabricksTaskBaseOperator):
    """Custom operator wrapping DatabricksTaskOperator"""

    template_fields = (
        "application_args",
        "spark_conf",
        "driver_jvm_props",
    )

    def __init__(
            self,
            *args,
            class_name: str = "",
            spark_conf: dict[str, str] | str = None,
            application_args: List[str] | str | None = None,
            driver_jvm_props: dict[str, str] | None = None,
            application_conf_file: str = "test.conf",
            libraries: List[dict[str, str]] = None,
            **kwargs,
    ):
        self.class_name = class_name
        self.application_conf_file = application_conf_file
        self.library = libraries if libraries is not None else _get_default_libraries()
        self.application_args = application_args
        super().__init__(*args, **kwargs)

        self.driver_jvm_props = driver_jvm_props
        self.spark_conf = spark_cone

    def _get_task_base_json(self) -> dict[str, Any]:
         return {
            "run_if": self.databricks_trigger_rule,
            'spark_jar_task': {
                'main_class_name': self.class_name,
                parameters": [
                     self.application_args,
                     "{{ get_test_template_value(ti,'test_key') }}" // I have hard-coded here for ref. In ideal case, this comes as part of self.application_args which has been added in template_fields
               ] 
            },
            'libraries': self.library,
        }

    def execute(self, context):
        super().execute(context)

From the above code sample, {{ get_test_template_value(ti,'test_key') }} is not getting resolved at the time of launch task from DatabricksWorkflowTaskGroup. But the same value got resolved at the time of task execution in airflow.

Sample json from launch task:

"tasks": [
    {
      "task_key": "task1",
      "depends_on": [],
      "run_if": "NONE_FAILED",
      "spark_jar_task": {
        "main_class_name": "com.sample.task1",
        "parameters": [
          "app.conf",
          "{{ get_test_template_value(ti,'test_key') }}"
        ]
      },
      "libraries": [
        {
          "jar": "test.jar"
        }
      ],
      "job_cluster_key": "job_cluster"
    },

Note: get_test_template_value function is added as a macro while DAG creation.

Though I have included the application_args as template_fields section in the CustomDatabricksWorkflowTaskOperator, it is unable to resolve the template.

What you think should happen instead?

Expectation is to jinja template resolution should happen while the launch tasks trigger API requests to Databricks to create the workflow. But since the template resolution is happening lazily here i.e., at the time of task execution in airflow which is late in this case as the respective task is already created in Databricks at the time of launch and Databricks doesn't have to way to get the template resolved value.

How to reproduce

Create a DAG with DatabricksWorkflowTaskGroup
Add couple of tasks
Create a custom Databricks Task operator by extending DatabricksTaskBaseOperator
Add one of the constructor parameter in template_fields section which needs to be tested
Override _get_task_base_json method to return a templated field/value as of the return dictionary
Trigger DAG
In launch task, you can notice from logs that the parameter property will pass the string as-is without resolving the actual value.
In the actual task execution, the actual value can be seen in the rendered template section.

Operating System

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)", NAME="Debian GNU/Linux"

Versions of Apache Airflow Providers

NA

Deployment

Other Docker-based deployment

Deployment details

We are building our own docker image with base image as apache/airflow:2.10.4-python3.10 and use the same for deployment.

Anything else?

Other approaches which I tried inside _get_task_base_json method of the custom operation which is responsible for returning the task JSON.

Tried to force the render by using super().render_template(self.application_args, context) and super().render_template_fields(context) which unfortunately was not possible as the context was not part of the _get_task_base_json method scope.
As this is an abstract method and I had to override, unable to pass context object to the step1.

Due to the above constraint, render templating as well as pulling from xcom also is not supported. (In other words, all the capabilities which we get from using context object is not possible).

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Experienced the same behaviors, very annoying. I got around it with a hacky fix that uses notebook_params, which do get templated properly.

I got around this issue by adding extra_job_params to template fields of _CreateDatabricksWorkflowOperator and passing them using workflow parameters

in plugins file:

from airflow.providers.databricks.operators import databricks_workflow


class Patched_CreateDatabricksWorkflowOperator(_CreateDatabricksWorkflowOperator):
    template_fields = (
        "notebook_params",
        "extra_job_params",
    )


databricks_workflow._CreateDatabricksWorkflowOperator = (
    Patched_CreateDatabricksWorkflowOperator
)


class DatabricksProviderPatcherPlugin(AirflowPlugin):
    name = "databricks_provider_patcher_plugin"

DatabricksWorkflowTaskGroup

...
                with DatabricksWorkflowTaskGroup(
                    databricks_conn_id="dataricks_default",
                    group_id="workflow",
                    extra_job_params={
                        "parameters": [
                            {
                                "name": "date",
                                "default": "{{ tomorrow_ds }}",
                            },
                        ],
                    },
                ) as workflow_tg:
...

DatabricksTaskOperator

        task = DatabricksTaskOperator(
            task_id="task",
            task_config={
                "spark_python_task": {
                    "python_file": "...",
                    "parameters": [
                        "--date",
                        "{{ job.parameters.date }}",
                    ],
                },
            },
        )

Having the same issue using other jinja templates default to airflow
{{ ds }}
and
{{ param.date }}

    data_preparation = DatabricksTaskOperator(
        dag=dag,
        task_id=...
        databricks_conn_id='databricks_conn',
        task_config=dict(
            python_wheel_task=dict(
                package_name=...
                entry_point=...
                parameters=[
                    '--date-range-min',  '{{ ds }}',
                ]
            )
        )
    )

@moomindani

cc @moomindani maybe you'll have time to take a look?

@eladkal

Thanks for the cc @eladkal. I took a look at this.

The reported behavior looks plausible based on the code. The root cause would be that child task operators inside DatabricksWorkflowTaskGroup never have their template fields rendered before being converted to the Databricks workflow JSON.

Here's what happens:

_CreateDatabricksWorkflowOperator (the "launch" task) stores references to child task operator instances in tasks_to_convert.
When the launch task executes, Airflow's task runner renders only the launch operator's own template_fields before calling execute().
Inside execute(), create_workflow_json() calls _convert_to_databricks_workflow_task() → _get_task_base_json() on each child task — but no one has called render_template_fields() on those child operators, so any Jinja expressions ({{ ds }}, {{ params.x }}, custom macros, etc.) would be sent to Databricks as raw strings.

Additionally, even if rendering were triggered, some key fields are missing from template_fields:

DatabricksTaskOperator.template_fields does not include task_config
DatabricksNotebookOperator.template_fields does not include notebook_path, source, or notebook_packages

The workarounds shared by @cshaddox and @aldouslpzchr are consistent with this analysis — they work because notebook_params and extra_job_params are rendered as part of the launch operator's own template_fields, bypassing the unrendered child tasks.

To fix this, create_workflow_json() (or _convert_to_databricks_workflow_task()) should call render_template_fields(context) on each child task before extracting the JSON, and the missing fields above should be added to template_fields.

I'm taking a quick peek at this as well, but just want to call out the following.

DatabricksTaskOperator.template_fields does not include task_config.

This has been taken care of by #65858.

@moomindani

@moomindani, why do you think this?

DatabricksNotebookOperator.template_fields does not include notebook_path, source, or notebook_packages.

I've created an initial draft PR for this. I need to do a bit more testing, but this should take care of the issue.

#65869

PR is updated, should be ready to go (pending passage of checks)!

@jroachgolf84

@jroachgolf84 Good catch on #65858, thanks!

On DatabricksNotebookOperator — looking at it again, the strongest case is notebook_path (users may want to use Jinja expressions like {{ ds }} in the path). source and notebook_packages are less likely to need templating, so I overstated that part.

No worries - I'll go ahead and open up another PR for that.

Navigation Menu

Repository navigation

Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup #46563

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup #46563

Description

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

boring-cyborg commented on Feb 7, 2025

cshaddox commented on Jun 10, 2025

aldouslpzchr commented on Sep 3, 2025

gabrielbckrsc commented on Oct 28, 2025

eladkal commented on Apr 9, 2026

moomindani commented on Apr 9, 2026

jroachgolf84 commented on Apr 26, 2026

jroachgolf84 commented on Apr 26, 2026

jroachgolf84 commented on Apr 26, 2026

jroachgolf84 commented on Apr 26, 2026

moomindani commented on Apr 28, 2026

jroachgolf84 commented on Apr 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions