SparkKubernetesOperator to support arguments

Need an option ,when triggering a dag via airflow web ui, to pass JSON arguments to
SparkKubernetesOperator.

Currently I don't see a way to do it as, SparkKubernetesOperator accepts application.yaml only.

Would you be interested in submitting a PR? It would be nice if you could use the issue template in the future as well.

Incase this has not yet been picked up, I would like to work on this issue 😄

@goodhamgupta

@potiuk

I think I don't understand the issue well enough. I see that for the SparkKubernetesOperator, the following code segment:

class SparkKubernetesOperator(BaseOperator):
    """
    """

    template_fields = ['application_file', 'namespace']
    template_ext = ('.yaml', '.yml', '.json')
    ui_color = '#f4a460'

I think this should already allow .json type file to be uploaded from the UI? I would appreciate it if you could provide me with any pointers on how to start this issue. 😅

Thanks!

@potiuk

@potiuk, @uranusjr Sorry for the noise, but I would appreciate any hints you might be able to provide 😄

If you meant template_ext contains .json, that only affects the listed templated_fields. I think OP wants to be able to pass a path from the “Trigger DAG” page’s param field, which is a completely different thing.

@goodhamgupta

@goodhamgupta are you still working on this issue?

@eladkal

@eladkal Unfortunately I wasn't able to work on this issue for now.

Answering earlier question (which I missed):

The scope of this change is mainly to make sure that you can pass the CRD .yaml file (possibly in json form) through a Param of the kind that will be easy to enter/ validate via UI and document it.

I do not think there is a lot (maybe no) code is to be written on that one - mostly to see if the current Param .json type will nicely fit there and provide an example of defining such param in the task and testing it.

Most likely the output of the task will be to add an example dag and UI screenshot to the documentation describing how to do it.

UPDATE: possibly it will need "application_file_content" optional field to be added (or simillar)

Navigation Menu

Repository navigation

SparkKubernetesOperator to support arguments #16728

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SparkKubernetesOperator to support arguments #16728

Description

Activity

uranusjr commented on Jun 30, 2021

goodhamgupta commented on Aug 3, 2021

potiuk commented on Aug 3, 2021

goodhamgupta commented on Aug 6, 2021

goodhamgupta commented on Sep 13, 2021

uranusjr commented on Sep 13, 2021

eladkal commented on Dec 19, 2021

goodhamgupta commented on Dec 19, 2021

potiuk commented on Sep 9, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions