Repository navigation
Log server on celery worker does not work in IPv6-only setup #24755
Description
Apache Airflow version
2.2.5
What happened
I deployed the Airflow helm chart in a Kubernetes cluster that only allows IPv6 traffic.
When I want to look at a task log in the UI there is this message:
*** Fetching from: http://airflow-v1-worker-0.airflow-v1-worker.airflow.svc.cluster.local:8793/log/my-dag/my-task/2022-06-28T00:00:00+00:00/1.log
*** Failed to fetch log file from worker. [Errno 111] Connection refused
So the webserver cannot fetch the logfile from the worker.
This happens in my opinion because the gunicorn application listens to 0.0.0.0 (IPv4), see code or worker log below, and the inter-pod communication in my cluster is IPv6.
~ » k logs airflow-v1-worker-0 -c airflow-worker -p
[2022-06-30 14:51:52 +0000] [49] [INFO] Starting gunicorn 20.1.0
[2022-06-30 14:51:52 +0000] [49] [INFO] Listening at: http://0.0.0.0:8793 (49)
[2022-06-30 14:51:52 +0000] [49] [INFO] Using worker: sync
[2022-06-30 14:51:52 +0000] [50] [INFO] Booting worker with pid: 50
[2022-06-30 14:51:52 +0000] [51] [INFO] Booting worker with pid: 51
-------------- celery@airflow-v1-worker-0 v5.2.3 (dawn-chorus)
--- ***** -----
-- ******* ---- Linux-5.10.118-x86_64-with-glibc2.28 2022-06-30 14:51:53
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f73b8d23d00
- ** ---------- .> transport: redis://:**@airflow-v1-redis-master.airflow.svc.cluster.local:6379/1
- ** ---------- .> results: postgresql://airflow:**@airflow-v1-pgbouncer.airflow.svc.cluster.local:6432/airflow_backend_db
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. airflow.executors.celery_executor.execute_command
What you think should happen instead
The gunicorn webserver should (configurably) listen to IPv6 traffic.
How to reproduce
No response
Operating System
Debian GNU/Linux 10 (buster)
Versions of Apache Airflow Providers
No response
Deployment
Other 3rd-party Helm chart
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
potiuk commentedon Jun 30, 2022
Would you like to fix it - you seem to know, where and how and you have nice environment to test the fix in. And you will not have to wait for someone to pick it up - this is one of the best ways you can contribute back to the software you get for free and join the community of more than 2100 contributors.
zartstrom commentedon Jul 2, 2022
-- edited --
Hi @potiuk, I'll try to craft a PR, thanks for the inspiration :)
I have one question, maybe you can help or point me to the right source:
What is the best way to create a airflow docker image with a change in the source code?
Answer: Look in the documentation for building the image and there for section building from github.
You can run the build with build arg
What I tried before: I did a tiny code change (hardcoded the log server to some other number,
8888), and then built an image withBut this builds from the latest pypi version of airflow, doh.
When I deployed an image built with the that command with a helm chart to our Kubernetes Cluster nothing changed and my code change was not visible of course.
potiuk commentedon Jul 2, 2022
Yeah. This is good observation. The
Dockerfiledefaults are meant to be useful for people who want to customize the image and use the "released" version of Airflow. What you should look at as a "developer" is breeze. Breeze has abuild-prod-imagecommand that uses local sources to build the image.If you are curious - you might run it with
--dry-runto understand which build-args should be passed to get thesource build- and you can also look more at https://airflow.apache.org/docs/docker-stack/build-arg-ref.html to understand all the parameters passed. one of them isAIRFLOW_SOURCES_FROMset to.to use current sources, but there are few others.Bind log server on worker to IPv6 address (#24755) (#24846)
Bind log server on worker to IPv6 address (#24755) (#24846)