Log URL 404 for older retry attempts (hostname overwritten)

[pedro-cf]

Apache Airflow version

2.11.X

If "Other Airflow 3 version" selected, which one?

No response

What happened?

When a task has multiple retry attempts and each attempt ran on a different Celery worker, the log URLs for older attempts return 404.

For example, if attempt 1 ran on worker-a and attempt 2 ran on worker-b, clicking "Attempt 1" logs results in a 404 because the log server request is routed to worker-b (the current hostname stored on the TaskInstance).

This only affects setups using local/file-based logging (i.e. FileTaskHandler). Remote logging backends (S3, GCS, etc.) are not affected since they don't route log requests through the worker's log server.

What you think should happen instead?

Each retry attempt's log URL should route to the worker that actually executed that attempt. Attempt 1 logs should be fetched from worker-a, attempt 2 from worker-b, etc.

How to reproduce

1. Set up Airflow with Celery executor, at least 2 workers, and local/file-based logging (default FileTaskHandler, no remote logging backend configured)
2. Create a DAG with retries=2 where the task will fail on first attempts
3. Ensure each retry runs on a different worker (can be forced by scaling/restarting workers between retries)
4. After the task reaches a terminal state, navigate to its log view
5. Click on the log for attempt 1
6. Observe a 404 error from the log server

Operating System

Ubuntu 24.04 LTS

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[kalluripradeep]

Hi I would like to work on this