Apache Airflow version
3.0.4
If "Other Airflow 2 version" selected, which one?
No response
What happened?
We have Airflow deployed on Kubernetes, with logs being forwarded to Elasticsearch via Fluentd. After upgrading from version 2.10.4 to 3.0.4, we encountered an issue where task logs are no longer visible in stdout, which previously worked as expected in 2.10.4.
We attempted to use custom logging configurations, but the problem persists. Notably, writing logs to files still works correctly.
We would like to continue using Fluentd to send logs to Elasticsearch, rather than writing directly to Elasticsearch.
Do you have any insights into the root cause of this issue or suggestions for resolving it?
Apache Airflow
version | 3.0.4
executor | LocalExecutor,KubernetesExecutor
task_logging_handler | airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler
[elasticsearch]
write_stdout = True
write_to_es = False
[logging]
remote_logging = True
What you think should happen instead?
Tasks logs should be visible in stdout
How to reproduce
create a simple task which prints to stdout
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
apache-airflow-providers-amazon 9.9.0
apache-airflow-providers-apache-kafka 1.9.0
apache-airflow-providers-celery 3.12.0
apache-airflow-providers-cncf-kubernetes 10.6.0
apache-airflow-providers-common-compat 1.7.1
apache-airflow-providers-common-io 1.6.0
apache-airflow-providers-common-messaging 1.0.3
apache-airflow-providers-common-sql 1.27.2
apache-airflow-providers-docker 4.4.0
apache-airflow-providers-elasticsearch 6.3.0
apache-airflow-providers-fab 2.2.1
apache-airflow-providers-ftp 3.13.0
apache-airflow-providers-git 0.0.3
apache-airflow-providers-google 16.0.0
apache-airflow-providers-grpc 3.8.0
apache-airflow-providers-hashicorp 4.3.0
apache-airflow-providers-http 5.3.1
apache-airflow-providers-microsoft-azure 12.4.1
apache-airflow-providers-mongo 5.2.0
apache-airflow-providers-mysql 6.3.1
apache-airflow-providers-odbc 4.10.0
apache-airflow-providers-openlineage 2.4.0
apache-airflow-providers-postgres 6.2.0
apache-airflow-providers-redis 4.1.0
apache-airflow-providers-sendgrid 4.1.1
apache-airflow-providers-sftp 5.3.1
apache-airflow-providers-slack 9.1.1
apache-airflow-providers-smtp 2.1.0
apache-airflow-providers-snowflake 6.4.0
apache-airflow-providers-ssh 4.1.0
apache-airflow-providers-standard 1.3.0
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Apache Airflow version
3.0.4
If "Other Airflow 2 version" selected, which one?
No response
What happened?
We have Airflow deployed on Kubernetes, with logs being forwarded to Elasticsearch via Fluentd. After upgrading from version 2.10.4 to 3.0.4, we encountered an issue where task logs are no longer visible in stdout, which previously worked as expected in 2.10.4.
We attempted to use custom logging configurations, but the problem persists. Notably, writing logs to files still works correctly.
We would like to continue using Fluentd to send logs to Elasticsearch, rather than writing directly to Elasticsearch.
Do you have any insights into the root cause of this issue or suggestions for resolving it?
Apache Airflow
version | 3.0.4
executor | LocalExecutor,KubernetesExecutor
task_logging_handler | airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler
[elasticsearch]
write_stdout = True
write_to_es = False
[logging]
remote_logging = True
What you think should happen instead?
Tasks logs should be visible in stdout
How to reproduce
create a simple task which prints to stdout
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
apache-airflow-providers-amazon 9.9.0
apache-airflow-providers-apache-kafka 1.9.0
apache-airflow-providers-celery 3.12.0
apache-airflow-providers-cncf-kubernetes 10.6.0
apache-airflow-providers-common-compat 1.7.1
apache-airflow-providers-common-io 1.6.0
apache-airflow-providers-common-messaging 1.0.3
apache-airflow-providers-common-sql 1.27.2
apache-airflow-providers-docker 4.4.0
apache-airflow-providers-elasticsearch 6.3.0
apache-airflow-providers-fab 2.2.1
apache-airflow-providers-ftp 3.13.0
apache-airflow-providers-git 0.0.3
apache-airflow-providers-google 16.0.0
apache-airflow-providers-grpc 3.8.0
apache-airflow-providers-hashicorp 4.3.0
apache-airflow-providers-http 5.3.1
apache-airflow-providers-microsoft-azure 12.4.1
apache-airflow-providers-mongo 5.2.0
apache-airflow-providers-mysql 6.3.1
apache-airflow-providers-odbc 4.10.0
apache-airflow-providers-openlineage 2.4.0
apache-airflow-providers-postgres 6.2.0
apache-airflow-providers-redis 4.1.0
apache-airflow-providers-sendgrid 4.1.1
apache-airflow-providers-sftp 5.3.1
apache-airflow-providers-slack 9.1.1
apache-airflow-providers-smtp 2.1.0
apache-airflow-providers-snowflake 6.4.0
apache-airflow-providers-ssh 4.1.0
apache-airflow-providers-standard 1.3.0
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct