fix: gracefully handle 404 from worker log server for historical retry attempts

[kalluripradeep]

Backport of #62475 to v2-11-test.

Original PR: #62475

What changed

Added 404 handling in _read_from_logs_server in file_task_handler.py with a local filesystem fallback and a clear message when logs aren't accessible.

How I fixed it

Added an elif response.status_code == 404 branch that:

1. First tries reading from local filesystem via _read_from_local — covers cases where logs are on a shared drive
2. If that also finds nothing, shows the user a clear message instead of a raw 404: "Log file not found on worker '{hostname}'. This attempt may have run on a different worker whose logs are no longer accessible. Consider configuring remote logging (S3, GCS, etc.) for log persistence."

Fixes #62372

[kalluripradeep]

@eladkal — backport PR for v2-11-test as requested.

[eladkal]

@kalluripradeep can you verify if the failures in test_trigger_dag_and_wait_for_result relates to the changes in the PR?

[kalluripradeep]

@kalluripradeep can you verify if the failures in test_trigger_dag_and_wait_for_result relates to the changes in the PR?

@eladkal — confirmed these failures are unrelated to this PR. The failing tests are test_integration_run_dag (timeout >300s) and test_integration_run_dag_with_scheduler_failure in TestCeleryAndLocalExecutor — both are DAG execution/scheduling tests.

This PR only modifies file_task_handler.py (log reading logic) and adds unit tests for it. No changes to DAG triggering, scheduling, or executor code.

[potiuk]

Yes. That was a problem with Celery Provider 3.17.0 #63043