Adding Support for Yarn queue and other extras in SparkSubmit Operator and Hook #35911

Activity

1. pateash

   added

   kind:featureFeature RequestsFeature Requests

   needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

   on Nov 28, 2023
2. pateash commented on Nov 28, 2023

   

   pateash

   on Nov 28, 2023

   Last edited by pateash

   ContributorAuthor

   More actions

   Please assign this to me, this is a quick fix
   I will try to complete this ASAP.
3. hussein-awala

   added

   area:providers

   provider:apache-spark

   and removed

   needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

   on Nov 28, 2023
4. hussein-awala

   assigned

   pateash

   on Nov 28, 2023
5. hussein-awala commented on Nov 28, 2023

   

   hussein-awala

   on Nov 28, 2023

   Member

   More actions

   @pateash assigned you.

   👍React with 👍1Reacted by Ashish Patel
6. pateash commented on Dec 2, 2023

   

   pateash

   on Dec 2, 2023

   ContributorAuthor

   More actions

   Currently it's possible to set this option inside Spark connection, but we should have the ability to override it if needed using operator arguments.
   
7. eladkal commented on Dec 2, 2023

   

   eladkal

   on Dec 2, 2023

   Contributor

   More actions

   Currently it's possible to set this option inside Spark connection, but we should have the ability to override it if needed using operator arguments.

   So do we have a task to complete here?
8. pateash commented on Dec 3, 2023

   

   pateash

   on Dec 3, 2023

   ContributorAuthor

   More actions

   @eladkal,
   I am thinking of adding it as an argument in SparkSubmitOperator if user wants to override it from connection extras.
   let me know what you think.
9. eladkal commented on Dec 3, 2023

   

   eladkal

   on Dec 3, 2023

   Contributor

   More actions

   We normally allow to override connection extra from the operator level. For example with Snowflake you can set different warehouse than the default one set in the connection.

   👍React with 👍1Reacted by Ashish Patel
10. pateash commented on Dec 7, 2023

    

    pateash

    on Dec 7, 2023

    ContributorAuthor

    More actions

    Yes, it's done using args AFAIK,
    Let me add an argument to override this.
11. eladkal

    added

    good first issue

    on Dec 9, 2023
12. pateash

    changed the title [-]Adding Support for Yarn queue in SparkSubmit Operator[/-] [+]Adding Support for Yarn queue and other extras in SparkSubmit Operator and Hook[/+] on Dec 12, 2023
13. josh-fell

    closed this as completedin #36151on Dec 13, 2023
14. databius commented on Mar 18, 2024

    

    databius

    on Mar 18, 2024

    Last edited by databius

    More actions

    I think the update caused the new issue since SparkSubmitOperator accidentally override the Base Operator's parameter.

    https://github.com/apache/airflow/blob/main/airflow/providers/apache/spark/operators/spark_submit.py#L76

    :param queue: The name of the YARN queue to which the application is submitted.
    (will overwrite any yarn queue defined in the connection's extra JSON)
    

    https://github.com/apache/airflow/blob/main/airflow/models/baseoperator.py#L587

    :param queue: which queue to target when running this job. Not
     all executors implement queue management, the CeleryExecutor
     does support targeting specific queues.
    

    So if you are using CeleryKubernetesExecutor, you cannot decide the Operator run on KubernetesExecutor by setting the queue to kubernetes_queue

    😕React with 😕1Reacted by Daniel Juárez
15. djuarezg commented on Mar 25, 2024

    

    djuarezg

    on Mar 25, 2024

    Last edited by djuarezg

    More actions

    Since 4.6.0 where this was introduced, onwards, using SparkSubmitOperator is broken for us, as the queue can no longer be taken into account due to this change.

    Before, this queue parameter overrode BaseOperator value where a queue is an Airflow queue