cluster config generation using cluster config YAML file.

Description

Add cluster config generation from yaml file which is generated using gcloud dataproc clusters export command.

Use case / motivation

We are generating quite a lot of clusters using a cluster config which keeps on changing depending upon project to project. I wrote custom operator which reads yaml config and converts it to dict and then uses the same config to genreate cluster. Wondering if it will be useful if cluster config generator operator can accept a yaml config or can create a new operator.

Are you willing to submit a PR?

I do have a working code for converting yaml stored on gcs to dict.

Related Issues

There is no issue as such, it is a functionality which allows users to specify cluster config as yaml file.

Thanks for opening your first issue here! Be sure to follow the issue template!

Feel free to make a PR!

config generator

~~DataprocCreateClusterOperator~~ClusterGenerator class is a deprecated and we are not adding new features to it.

I think we can extend DataprocCreateClusterOperator to accept the file path in cluster_config parameter like Cloud Build operator.

airflow/airflow/providers/google/cloud/operators/cloud_build.py

Lines 172 to 175 in d268016

    
               :param body: The build config with instructions to perform with CloudBuild. 
        
                   Can be a dictionary or path to a file type like YAML or JSON. 
        
                   See: https://cloud.google.com/cloud-build/docs/api/reference/rest/v1/projects.builds 
        
               :type body: dict or string

@mik-laj

you mean airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator or you mean airflow.contrib.operators.dataproc_operator.DataprocClusterCreateOperator ? I thought this is going to be in providers package.
@mik-laj

@iostreamdoth

@iostreamdoth ClusterGenerator from providers is legacy/deprecated method to generate dataproc cluster configuration. You should use cluster_config parameter. For details, see:

airflow/airflow/providers/google/cloud/operators/dataproc.py

Lines 553 to 559 in 026ffe6

    
               "Passing cluster parameters by keywords to `{}` " 
        
               "will be deprecated. Please provide cluster_config object using `cluster_config` parameter. " 
        
               "You can use `airflow.dataproc.ClusterGenerator.generate_cluster` method to " 
        
               "obtain cluster object.".format(type(self).__name__), 
        
               DeprecationWarning, 
        
               stacklevel=1, 
        
           )

@mik-laj

Got it, I got confused with the Operator name earlier.
Thanks @mik-laj

@iostreamdoth

@iostreamdoth do you plan to finish the stale PR?

@mik-laj

Hi! Triaging older issues — I think this one may be answerable with the existing API, and could potentially be closed.

The use case described (loading a YAML-exported Dataproc cluster config and feeding it to DataprocCreateClusterOperator) is already supported via the cluster_config parameter, which accepts a dict shaped like the google.cloud.dataproc_v1.types.ClusterConfig protobuf message:

providers/google/src/airflow/providers/google/cloud/operators/dataproc.py#L602-L605 — docstring: "If a dict is provided, it must be of the same form as the protobuf message ClusterConfig".
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py#L656 — signature: cluster_config: dict | Cluster | None = None.
The original reporter acknowledged this back in 2021 ("Got it, I got confused with the Operator name earlier") after @mik-laj pointed out cluster_config was the recommended path.

In practice this means a gcloud dataproc clusters export ... > cfg.yaml output can be loaded with yaml.safe_load(...) in the DAG file and passed directly to cluster_config=... — no dedicated YAML-loading operator needed.

If I'm missing something (a YAML-specific feature beyond what dict-via-yaml.safe_load covers, or a deprecation I haven't spotted), happy to dig further. Otherwise this looks closeable.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current main branch.)

Navigation Menu

Repository navigation

cluster config generation using cluster config YAML file. #17089

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	:param body: The build config with instructions to perform with CloudBuild.
	Can be a dictionary or path to a file type like YAML or JSON.
	See: https://cloud.google.com/cloud-build/docs/api/reference/rest/v1/projects.builds
	:type body: dict or string

	"Passing cluster parameters by keywords to `{}` "
	"will be deprecated. Please provide cluster_config object using `cluster_config` parameter. "
	"You can use `airflow.dataproc.ClusterGenerator.generate_cluster` method to "
	"obtain cluster object.".format(type(self).__name__),
	DeprecationWarning,
	stacklevel=1,
	)

cluster config generation using cluster config YAML file. #17089

Description

Activity

boring-cyborg commented on Jul 19, 2021

potiuk commented on Jul 19, 2021

mik-laj commented on Jul 19, 2021

iostreamdoth commented on Jul 19, 2021

mik-laj commented on Jul 19, 2021

iostreamdoth commented on Jul 19, 2021

eladkal commented on Nov 28, 2021

jbbqqf commented on May 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions