Repository navigation
Airflow webserver UI unable to access worker node logs #26201
Description
Apache Airflow version
2.3.4
What happened
We have set up a multi node Airflow configuration. We have the below setup.
- 4 nodes
- node 1: webserver, rabbitmq, flower
- node 2: meta db
- node 3: worker 1
- node 4: worker 2
Although the setup is working fine and DAGs are getting distributed among the worker nodes, however the logs generated in the worker nodes are not reflecting in the webserver UI in the master node.
Below is the error:
***** Log file does not exist: /home/dvafadm/airflow/logs/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** Fetching from: http://:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log'
For more information check: https://httpstatuses.com/403**
We have checked all the folder permission, secret_key, timezone but still the problem persist.
can anyone help here?
What you think should happen instead
No response
How to reproduce
No response
Operating System
Linux Red Hat
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Sep 7, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
nsAstro commentedon Sep 8, 2022
Can you see if there are any files in the directory '/home/dvafadm/airflow/logs/' and if so, can you find the ones corresponding the the above DAG and list the file/directory structure? I'm curious if this is a name mismatch or if the logs just don't exist.
devashish-parmar commentedon Sep 9, 2022
yes, the directory/file does exist. the below command runs fine from the worker node
less "/home/dvafadm/airflow/logs/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log"
jledoux-sonergia commentedon Sep 16, 2022
We've ran into the same issue, worker's gunicorn log webserver does not start properly on 2.3.4, downgrading to 2.3.3 fix this issue.
I think it's related to this PR #24846
And saimon46's comment explains it: #24846 (comment)
potiuk commentedon Sep 19, 2022
@jledoux-sonergia - The 2.4.0 is out today and It should have the IPV6 problem fixed. @devashish-parmar - if the request is forbidden, you should have looked at the webeserver logs and it should tell you the exact reason for the problem you have
Timezone is not an issue. You should check if TIME is synchronized on all your machines.
Converting that into a discussion as either some of the problems mentioned here (IPV6) have been fixed in 2.4.0 or they need more looking at the deployment side and checking the webserver logs.