You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Inconsistencies with Dataproc Operator parameters #15622
I'm looking at the GCP Dataproc operator and noticed that the DataprocCreateClusterOperator and DataprocDeleteClusterOperator require a region parameter, but other operators, like the DataprocSubmitJobOperator require a location parameter instead. I think it would be best to consistently enforce the parameter as region, because that's what is required in the protos for all of the cluster CRUD operations and for job submission. location also feels too ambiguous imho, because it implies we could also pass a GCE zone, which in this case is either unnecessary or not supported (I can't remember which and it's too late in my Friday for me to double check 🙃 )
This might be similar to #13454 but I'm not 100% sure. If we think this is worth working on, I could maybe take this on as a PR, but it would be low priority for me, and if someone else wants to take it on, they should feel free to 😄
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
Apache Airflow version: 2.0.1
Environment:
running locally on a Mac with SQLite backend only running a unit test that makes sure the dag is compiled. Installed with Pip and the provided constraints file.
What happened:
if you pass location in the DataprocCreateClusterOperator the DAG won't compile and will throw an error airflow.exceptions.AirflowException: Argument ['region']
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
Activity
boring-cyborg commentedon Apr 30, 2021
Thanks for opening your first issue here! Be sure to follow the issue template!
mik-laj commentedon May 2, 2021
Hi @leahecole
Are you willing to submit a PR? I'm happy to help with the review.
Daniel-Han-Yang commentedon May 16, 2021
Can I work on this @leahecole @mik-laj ?
mik-laj commentedon May 16, 2021
@Daniel-Han-Yang feel free. Assigned.
Daniel-Han-Yang commentedon May 30, 2021
@mik-laj, I've put in a PR for this change, appreciate it if you let me know what needs to be added/changed. Thank you