Repository navigation
Description
Apache Airflow Provider(s)
opensearch
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.6.1
apache-airflow-providers-celery==3.10.6
apache-airflow-providers-cncf-kubernetes==10.4.3
apache-airflow-providers-common-compat==1.6.0
apache-airflow-providers-common-io==1.5.4
apache-airflow-providers-common-messaging==1.0.0
apache-airflow-providers-common-sql==1.26.0
apache-airflow-providers-docker==4.3.1
apache-airflow-providers-elasticsearch==6.2.2
apache-airflow-providers-fab==2.0.1
apache-airflow-providers-ftp==3.12.3
apache-airflow-providers-git==0.0.2
apache-airflow-providers-google==15.1.0
apache-airflow-providers-grpc==3.7.3
apache-airflow-providers-hashicorp==4.1.1
apache-airflow-providers-http==5.2.2
apache-airflow-providers-microsoft-azure==12.3.1
apache-airflow-providers-mysql==6.2.2
apache-airflow-providers-odbc==4.9.2
apache-airflow-providers-openlineage==2.2.0
apache-airflow-providers-opensearch==1.6.3
apache-airflow-providers-postgres==6.1.3
apache-airflow-providers-redis==4.0.2
apache-airflow-providers-sendgrid==4.0.1
apache-airflow-providers-sftp==5.2.1
apache-airflow-providers-slack==9.0.5
apache-airflow-providers-smtp==2.0.3
apache-airflow-providers-snowflake==6.2.2
apache-airflow-providers-ssh==4.0.1
apache-airflow-providers-standard==1.0.0
apache-airflow-task-sdk==1.0.0
Apache Airflow version
3.0.0
Operating System
Debian GNU/Linux 12
Deployment
Official Apache Airflow Helm Chart
Deployment details
Deployed the latest Airflow helm chart with airflowVersion: "3.0.0" and KubernetesExecutor for executor on EKS. The logs persistent is diabled and chose Opensearch as an instance of opensearch is running on the same eks cluster.
opensearch:
# Enable opensearch task logging
enabled: true
connection:
scheme: https
user: admin
pass: pass
host: opensearch-cluster-master.opensearch.svc.cluster.local
port: 9200
What happened
Seeing this error when trying to access task logs in the API server pod logs
opensearchpy.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPSConnection object at 0x7efce05bef50>: Failed to resolve 'https://admin:pass@opensearch-cluster-master.opensearch.svc.cluster.local:9200' ([Errno -2] Name or service not known)) caused by: NameResolutionError(<urllib3.connection.HTTPSConnection object at 0x7efce05bef50>: Failed to resolve 'https://admin:pass@opensearch-cluster-master.opensearch.svc.cluster.local:9200' ([Errno -2]
What you think should happen instead
I should be able to view the loggging in the UI which should be retrived from opensearch.
How to reproduce
Enabled Opensearch option for remote logging in the helm chart and give valid connection setting for the same.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Apr 23, 2025
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
sonu1311 commentedon Apr 24, 2025
Suppose I opt for persistence logs like EFS on EKS and stay with KubernetesExecutor. In that case, it tries to use a service running on port 8793, but with this type of executor, that pod/service, and I suspect it's trying to connect with the worker pod, which is not running and will run, I guess, when I choose the Celery-based executor.
Am I missing any configuration here?
Thanks for any help on this.
sonu1311 commentedon Apr 24, 2025
The issue with logging to EFS has been resolved. It was a permission error when the task pods wrote to the EFS volume.
The OpenSearch integration is still an issue.
eladkal commentedon Mar 31, 2026
cc @Owen-CH-Leung
Owen-CH-Leung commentedon Mar 31, 2026
Yes I have a WIP MR that restores it.
#64364
I'll complete it this weekend