Skip to content

Inconsistencies with Dataproc Operator parameters #15622

Closed

Description

@leahecole

I'm looking at the GCP Dataproc operator and noticed that the DataprocCreateClusterOperator and DataprocDeleteClusterOperator require a region parameter, but other operators, like the DataprocSubmitJobOperator require a location parameter instead. I think it would be best to consistently enforce the parameter as region, because that's what is required in the protos for all of the cluster CRUD operations and for job submission. location also feels too ambiguous imho, because it implies we could also pass a GCE zone, which in this case is either unnecessary or not supported (I can't remember which and it's too late in my Friday for me to double check 🙃 )

This might be similar to #13454 but I'm not 100% sure. If we think this is worth working on, I could maybe take this on as a PR, but it would be low priority for me, and if someone else wants to take it on, they should feel free to 😄

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.

-->

Apache Airflow version: 2.0.1

Environment:

running locally on a Mac with SQLite backend only running a unit test that makes sure the dag is compiled. Installed with Pip and the provided constraints file.

What happened:

if you pass location in the DataprocCreateClusterOperator the DAG won't compile and will throw an error airflow.exceptions.AirflowException: Argument ['region']

As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.

--->

Activity

  1. boring-cyborg commented on Apr 30, 2021

    @boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template!

  2. mik-laj commented on May 2, 2021

    @mik-laj
    Member

    Hi @leahecole

    Are you willing to submit a PR? I'm happy to help with the review.

  3. Daniel-Han-Yang commented on May 16, 2021

    @Daniel-Han-Yang
    Contributor

    Can I work on this @leahecole @mik-laj ?

  4. mik-laj commented on May 16, 2021

    @mik-laj
    Member

    @Daniel-Han-Yang feel free. Assigned.

  5. Daniel-Han-Yang commented on May 30, 2021

    @Daniel-Han-Yang
    Contributor

    @mik-laj, I've put in a PR for this change, appreciate it if you let me know what needs to be added/changed. Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions