Skip to content

fix: forward task logs to stdout in LocalExecutor#63960

Closed
taranlu-houzz wants to merge 1 commit intoapache:mainfrom
taranlu-houzz:fix-local-executor-logs-to-stdout
Closed

fix: forward task logs to stdout in LocalExecutor#63960
taranlu-houzz wants to merge 1 commit intoapache:mainfrom
taranlu-houzz:fix-local-executor-logs-to-stdout

Conversation

@taranlu-houzz
Copy link
Copy Markdown

LocalExecutor's _execute_work() calls supervise() without passing
subprocess_logs_to_stdout=True, so task logs are only written to log
files and never reach the container's stdout. This breaks log collection
in deployments that rely on container stdout for log aggregation.

The containerized executor (execute_workload.py) already passes this
flag (added in #47731), but LocalExecutor was not updated.

closes: #54501
related: #49863


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Anthropic)

Generated-by: Claude Code following the guidelines

@boring-cyborg
Copy link
Copy Markdown

boring-cyborg bot commented Mar 20, 2026

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@boring-cyborg boring-cyborg bot added the area:Executors-core LocalExecutor & SequentialExecutor label Mar 20, 2026
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 20, 2026

@taranlu-houzz This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

  • Other failing CI checks: Failing: Postgres tests: core / DB-core:Postgres:14:3.10:Core...Serialization, MySQL tests: core / DB-core:MySQL:8.0:3.10:Core...Serialization, Sqlite tests: core / DB-core:Sqlite:3.10:Core...Serialization, Low dep tests:core / All-core:LowestDeps:14:3.10:Core...Serialization. Run prek run --from-ref main locally to reproduce. See static checks docs.

What to do next:

  • The comment informs you what you need to do.
  • Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
  • There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
  • Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

@potiuk potiuk marked this pull request as draft March 20, 2026 11:43
@taranlu-houzz taranlu-houzz force-pushed the fix-local-executor-logs-to-stdout branch from 1af4a04 to 1a722cb Compare March 20, 2026 17:08
@taranlu-houzz
Copy link
Copy Markdown
Author

taranlu-houzz commented Mar 20, 2026

The CI failures are all from test_execution_api_server_url_config (5 parametrized variants × 4 backend jobs = 20 failures) — the test mocks supervise() and asserts exact kwargs, so the new subprocess_logs_to_stdout=True parameter caused the mismatch.

Pushed a fix that updates the test assertion to expect the new parameter.

Edit: Claude sent this message. Still trying to figure out the best way to be transparent about working via an agent.

@taranlu-houzz taranlu-houzz marked this pull request as ready for review March 20, 2026 17:24
@eladkal eladkal added this to the Airflow 3.1.9 milestone Mar 22, 2026
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Mar 22, 2026
LocalExecutor's `_execute_work()` calls `supervise()` without passing
`subprocess_logs_to_stdout=True`, so task logs are only written to log
files and never reach the container's stdout. This breaks log collection
in Kubernetes deployments that rely on container stdout for aggregation
(e.g., Fluentd, Coralogix, Datadog).

The containerized executor (`execute_workload.py`) already passes this
flag (added in apache#47731), but LocalExecutor was not updated.

Closes apache#54501
@eladkal
Copy link
Copy Markdown
Contributor

eladkal commented Mar 22, 2026

Closing in favor of #64067

@eladkal eladkal closed this Mar 22, 2026
Subham-KRLX added a commit to Subham-KRLX/airflow that referenced this pull request Apr 1, 2026
…tor log forwarding in Airflow 3

This restores the ElasticsearchTaskHandler in airflow_local_settings.py and ensures LocalExecutor forwards task logs to stdout by passing subprocess_logs_to_stdout=True to the supervisor.

closes: apache#63960

closes: apache#49863

closes: apache#54501
eladkal pushed a commit that referenced this pull request Apr 1, 2026
…4067)

This restores the ElasticsearchTaskHandler in airflow_local_settings.py and ensures LocalExecutor forwards task logs to stdout by passing subprocess_logs_to_stdout=True to the supervisor.

closes: #63960
closes: #49863
closes: #54501
github-actions bot pushed a commit that referenced this pull request Apr 1, 2026
…rwarding (#64067)

This restores the ElasticsearchTaskHandler in airflow_local_settings.py and ensures LocalExecutor forwards task logs to stdout by passing subprocess_logs_to_stdout=True to the supervisor.
(cherry picked from commit f1495c2)

Co-authored-by: Subham <subhamsangwan26@gmail.com>
closes: #63960
closes: #49863
closes: #54501
github-actions bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Apr 1, 2026
…rwarding (apache#64067)

This restores the ElasticsearchTaskHandler in airflow_local_settings.py and ensures LocalExecutor forwards task logs to stdout by passing subprocess_logs_to_stdout=True to the supervisor.
(cherry picked from commit f1495c2)

Co-authored-by: Subham <subhamsangwan26@gmail.com>
closes: apache#63960
closes: apache#49863
closes: apache#54501
eladkal pushed a commit that referenced this pull request Apr 1, 2026
…rwarding (#64067) (#64592)

This restores the ElasticsearchTaskHandler in airflow_local_settings.py and ensures LocalExecutor forwards task logs to stdout by passing subprocess_logs_to_stdout=True to the supervisor.
(cherry picked from commit f1495c2)


closes: #63960
closes: #49863
closes: #54501

Co-authored-by: Subham <subhamsangwan26@gmail.com>
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Subham-KRLX added a commit to Subham-KRLX/airflow that referenced this pull request Apr 3, 2026
…ache#64067)

This restores the ElasticsearchTaskHandler in airflow_local_settings.py and ensures LocalExecutor forwards task logs to stdout by passing subprocess_logs_to_stdout=True to the supervisor.

closes: apache#63960
closes: apache#49863
closes: apache#54501
Suraj-kumar00 pushed a commit to Suraj-kumar00/airflow that referenced this pull request Apr 7, 2026
…ache#64067)

This restores the ElasticsearchTaskHandler in airflow_local_settings.py and ensures LocalExecutor forwards task logs to stdout by passing subprocess_logs_to_stdout=True to the supervisor.

closes: apache#63960
closes: apache#49863
closes: apache#54501
vatsrahul1001 added a commit that referenced this pull request Apr 8, 2026
…rwarding (#64067) (#64592)

This restores the ElasticsearchTaskHandler in airflow_local_settings.py and ensures LocalExecutor forwards task logs to stdout by passing subprocess_logs_to_stdout=True to the supervisor.
(cherry picked from commit f1495c2)


closes: #63960
closes: #49863
closes: #54501

Co-authored-by: Subham <subhamsangwan26@gmail.com>
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Executors-core LocalExecutor & SequentialExecutor type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3.0.4] Task logs are no longer visible in stdout only in files

3 participants