Repository navigation
Conversation
…pache#46563) When DatabricksWorkflowTaskGroup creates or updates a Databricks workflow job, the launch operator (_CreateDatabricksWorkflowOperator) calls create_workflow_json(context=context), which converts member tasks to workflow task JSON via task._convert_to_databricks_workflow_task. However, DatabricksTaskBaseOperator._convert_to_databricks_workflow_task never rendered the child operator's declared template_fields before extracting its task JSON. As a result, Jinja template expressions (such as {{ ds }}, {{ params }}, etc.) were transmitted to Databricks unrendered as raw strings. Fix this by: - Rendering declared template_fields on each child task operator using the execution context in DatabricksTaskBaseOperator._convert_to_databricks_workflow_task (excluding workflow_run_metadata, which is an unresolved XComArg at workflow creation time). - Adding notebook_path and source to DatabricksNotebookOperator.template_fields. - Documenting the template rendering contract in DatabricksWorkflowTaskGroup and the Databricks provider documentation (run-scoped vs task-scoped expressions). - Adding comprehensive regression tests for DatabricksTaskOperator, DatabricksNotebookOperator, custom DatabricksTaskBaseOperator subclasses, and context=None fallback. Closes: apache#46563 Related: apache#65869
…pache#46563) When tasks are defined within a DatabricksWorkflowTaskGroup, member tasks are converted to Databricks REST API workflow tasks via _convert_to_databricks_workflow_task(). Previously, declared template_fields (such as DatabricksTaskOperator.task_config) were not rendered against the launch task's execution context, causing unrendered Jinja templates to be sent to Databricks. In addition, DatabricksNotebookOperator omitted notebook_path and source from template_fields. This change: 1. Adds notebook_path and source to DatabricksNotebookOperator.template_fields. 2. Updates DatabricksTaskBaseOperator._convert_to_databricks_workflow_task to render declared template_fields using an isolated shallow proxy (excluding workflow_run_metadata), preventing in-place attribute mutation across repeated DAG runs. 3. Updates DatabricksNotebookOperator._convert_to_databricks_workflow_task to perform TaskGroup package and parameter merges on an operator proxy, preserving original task parameters. 4. Adds documentation on run-scoped vs. member task-scoped templating semantics. 5. Adds regression tests covering built-in operators, custom subclasses, repeated-run immutability, and unresolved XComArg preservation. Closes apache#46563
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When tasks are placed inside a
DatabricksWorkflowTaskGroup, the group creates a_CreateDatabricksWorkflowOperator("launch task") that constructs a Databricks Workflow job definition via REST API by calling_convert_to_databricks_workflow_task()on each member task.Previously, member task template fields (e.g.
DatabricksTaskOperator.task_configor custom operator template attributes) were not rendered against the launch task's execution context before building the payload. In addition,DatabricksNotebookOperatoromittednotebook_pathandsourcefrom its declaredtemplate_fields.This PR:
notebook_pathandsourcetoDatabricksNotebookOperator.template_fields.DatabricksTaskBaseOperator._convert_to_databricks_workflow_taskto render declaredtemplate_fieldsagainst the launch task's execution context using an isolated shallow proxy (copy.copy(self)), safely excludingworkflow_run_metadatato avoid resolvingXComArgprematurely and preventing in-place attribute mutation across repeated DAG runs.DatabricksNotebookOperator._convert_to_databricks_workflow_taskto merge parent TaskGroup packages and notebook parameters on a shallow proxy, preserving the original operator instance's parameters and packages across runs.workflow.rstand theDatabricksWorkflowTaskGroupdocstring clarifying that run-scoped Jinja expressions ({{ ds }},{{ dag_run }},{{ params }}) are rendered during workflow definition, whereas member task-scoped variables (e.g.{{ ti }}or task-specific XCom pulls) are not supported because member task instances do not exist when the launch task executes.context=None, immutability of operator attributes across repeated runs, and preservation of unresolvedworkflow_run_metadataXComArgwiring.Closes: #46563
Use Case / Motivation
Users submitting Spark jobs, JAR applications, or notebooks via
DatabricksWorkflowTaskGroupoften parameterize script paths, arguments, or job configurations with execution date or run metadata (such as{{ ds }}). Rendering member task template fields during workflow JSON creation ensures the Databricks REST API receives the evaluated parameters.Related Issues
Closes: #46563
Was generative AI tooling used to co-author this PR?
Generated-by: Antigravity following the guidelines