Skip to content

[v3.0.4] Task logs are no longer visible in stdout only in files #54501

@alex1404

Description

@alex1404

Apache Airflow version

3.0.4

If "Other Airflow 2 version" selected, which one?

No response

What happened?

We have Airflow deployed on Kubernetes, with logs being forwarded to Elasticsearch via Fluentd. After upgrading from version 2.10.4 to 3.0.4, we encountered an issue where task logs are no longer visible in stdout, which previously worked as expected in 2.10.4.

We attempted to use custom logging configurations, but the problem persists. Notably, writing logs to files still works correctly.

We would like to continue using Fluentd to send logs to Elasticsearch, rather than writing directly to Elasticsearch.

Do you have any insights into the root cause of this issue or suggestions for resolving it?

Apache Airflow
version | 3.0.4
executor | LocalExecutor,KubernetesExecutor
task_logging_handler | airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler

[elasticsearch]
write_stdout = True
write_to_es = False

[logging]
remote_logging = True

What you think should happen instead?

Tasks logs should be visible in stdout

How to reproduce

create a simple task which prints to stdout

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon 9.9.0
apache-airflow-providers-apache-kafka 1.9.0
apache-airflow-providers-celery 3.12.0
apache-airflow-providers-cncf-kubernetes 10.6.0
apache-airflow-providers-common-compat 1.7.1
apache-airflow-providers-common-io 1.6.0
apache-airflow-providers-common-messaging 1.0.3
apache-airflow-providers-common-sql 1.27.2
apache-airflow-providers-docker 4.4.0
apache-airflow-providers-elasticsearch 6.3.0
apache-airflow-providers-fab 2.2.1
apache-airflow-providers-ftp 3.13.0
apache-airflow-providers-git 0.0.3
apache-airflow-providers-google 16.0.0
apache-airflow-providers-grpc 3.8.0
apache-airflow-providers-hashicorp 4.3.0
apache-airflow-providers-http 5.3.1
apache-airflow-providers-microsoft-azure 12.4.1
apache-airflow-providers-mongo 5.2.0
apache-airflow-providers-mysql 6.3.1
apache-airflow-providers-odbc 4.10.0
apache-airflow-providers-openlineage 2.4.0
apache-airflow-providers-postgres 6.2.0
apache-airflow-providers-redis 4.1.0
apache-airflow-providers-sendgrid 4.1.1
apache-airflow-providers-sftp 5.3.1
apache-airflow-providers-slack 9.1.1
apache-airflow-providers-smtp 2.1.0
apache-airflow-providers-snowflake 6.4.0
apache-airflow-providers-ssh 4.1.0
apache-airflow-providers-standard 1.3.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions