Repository navigation
Spark Connection with k8s in URL not mapped correctly #21638
Description
Official Helm Chart version
1.2.0
Apache Airflow version
2.1.4
Kubernetes Version
v1.21.6+bb8d50a
Helm Chart configuration
I defined a new Connection String for AIRFLOW_CONN_SPARK_DEFAULT in values.yaml like the following section (base64 encoded or with correct string (spark://k8s://100.68.0.1:443?deploy-mode=cluster):
extraSecrets:
'{{ .Release.Name }}-airflow-connections':
data: |
AIRFLOW_CONN_SPARK_DEFAULT: 'c3Bhcms6Ly9rOHM6Ly8xMDAuNjguMC4xOjQ0Mz9kZXBsb3ktbW9kZT1jbHVzdGVy'
in Section extraEnvFrom i defined the following:
extraEnvFrom: |
- secretRef:
name: '{{ .Release.Name }}-airflow-connections'
Docker Image customisations
added apache-airflow-providers-apache-spark to base Image
What happened
Airflow Connection mapped wrong because of the k8s:// within the url. if i ask for the connection with cmd "airflow connections get spark_default" then host=k8s and schema=/100.60.0.1:443 which is wrong
What you expected to happen
the spark Connection based on k8s (spark://k8s://100.68.0.1:443?deploy-mode=cluster) should be parsed correctly
How to reproduce
define in values.yaml
extraSecrets:
'{{ .Release.Name }}-airflow-connections':
data: |
AIRFLOW_CONN_SPARK_DEFAULT: 'c3Bhcms6Ly9rOHM6Ly8xMDAuNjguMC4xOjQ0Mz9kZXBsb3ktbW9kZT1jbHVzdGVy'
extraEnvFrom: |
- secretRef:
name: '{{ .Release.Name }}-airflow-connections'
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by Kamil Breguła
Reacted by Invis00
Activity
boring-cyborg commentedon Feb 17, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
potiuk commentedon Feb 17, 2022
Would you lke to make a PR for that @Scope0910 - seems that it should not be difficult and you have an environment that you could test it in.
mik-laj commentedon Feb 17, 2022
@potiuk This is a bug in the URL parser in the Python built-in library
Scope0910 commentedon Feb 17, 2022
i agree with @mik-laj it isn't quite a bug of the helm chart himself. urllib parser in python is quite pain. Maybe it should handled in models/connection.py ?
potiuk commentedon Feb 17, 2022
Yeah. Doesn't really matter who's bug it is - Maybe we can simply workaround it in airflow ? Maybe we can fix some world around us rather than garbage-in -> garbage-out :)
I do not say we have to - but if possible - why not?
github-actions commentedon Mar 13, 2023
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
amoghrajesh commentedon Mar 21, 2023
If nobody is actively working on it, I'd like to try my hand at it. Looks like an interesting issue
potiuk commentedon Mar 21, 2023
feel free
amoghrajesh commentedon May 18, 2023
@potiuk @Scope0910 As per the documentation, the connection format cannot be the way specified by the issue: https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#uri-format
But I think this is something we should be fixing in our code to support spark on k8s urls.