Repository navigation
Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator #10596
Description
Description
GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator do not support direct impersonation of a service account in Google services.
Use case / motivation
GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator, in contrary to other Google operators, do not use Credentials class for authentication, so they require individual approach to support direct impersonation in them.
In case of GKEStartPodOperator it seems it should suffice to add --impersonate-service-account to the gcloud container clusters get-credentials command. However, this way we will not be able to use chain of service accounts, like in the rest of Google operators.
In case of DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator, some changes in provide_gcp_credential_file_as_context will probably be needed.
Related Issues
Activity
rajatsri28 commentedon Oct 5, 2020
Hi @olchas ,
Can I pick this up?
mik-laj commentedon Oct 5, 2020
@rajatsri28 Do you have experience with Google Cloud? This task is trivial from an Airflow perspective, but requires above-average knowledge of the Google Cloud platform. I don't want you to have a bad experience after contributing to this project.
rajatsri28 commentedon Oct 5, 2020
Hi @mik-laj , Yes I have some experience with Google Cloud. At Twitter, we migrated from on-premise offering to provide Airflow on GKE (GCP) so I can somewhat relate to this issue as well.
mik-laj commentedon Oct 5, 2020
@rajatsri28 Awesome! I assigned you to this ticket. 🐈
rajatsri28 commentedon Oct 15, 2020
Hi,
For
GKEStartPodOperator, I did not find any way to use a chain of service accounts for impersonation. It seems it's only available via the API. So I am thinking of following what is suggested here, i.e. to add--impersonate-service-accountto thegcloud container clusters get-credentialscommand.For
DataflowCreateJavaJobOperatorandDataflowCreatePythonJobOperator, I am a bit confused since there are not many options to update ADC. Are we looking for something like runninggcloud config set auth/impersonate_service_accountcommand to declare the impersonation? However, we will not be able to specify a chain of accounts then.mik-laj commentedon Oct 19, 2020
@rajatsri28 I think support for a single service account is sufficient in most cases.
gcloud config set auth/impersonate_service_accountThe gcloud command should work for all operators that use gcloud, so all you have to do is add account name forwarding inairflow.providers.google.common.hooks.base_google.GoogleBaseHook.provide_authorized_gcloudmethod.gcloud config set auth/impersonate_service_accountcommand and--impersonate-service-accountargument behave similarly, but I think the first method will be simpler to implement. I am not sure about this and think it is worth checking for all the gcloud supported authentication methods.abhishekbafna commentedon Aug 27, 2021
The DataflowCreateJavaJobOperator is documented as deprecated and users are suggested to use the BeamRunJavaPipelineOperator. I tried that and I believe, I was not able to impersonate the different service account in that as well. The reason could be that the underline logic might be the same between the two operators. We should add these Beam operators also into the above list.
shahar1 commentedon Jan 31, 2023
@eladkal Please assign me
shahar1 commentedon Feb 11, 2023
@eladkal Please reassign, I unassigned by accident
shahar1 commentedon Mar 3, 2023
I've been trying to resolve this issue, and it seems that:
GKEStartPodOperatorcurrently supports an impersonation chain. (Support impersonation_chain parameter in the GKEStartPodOperator #19518)DataflowCreateJavaJobOperatorandDataflowCreatePythonJobOperatorare both deprecated in favor ofBeamRunPythonPipelineOperator. (Add Apache Beam operators #12814)BeamRunPythonPipelineOperatorsupports the impersonation chain by passingDataflowConfigurationthat includesimpersonation_chainparameter. (Improve Apache Beam operators - refactor operator - common Dataflow logic #14094)Therefore, this issue could be closed as it is already resolved.