Airflow Helm Chart - CeleryExecutor issue?

[avidalco]

Hello Airflow Community!

I am currently facing an issue with my Airflow setup and would greatly appreciate any guidance or insights you could provide.

I have Airflow 2.9.1 installed on a Kubernetes cluster using the official Helm chart. Recently, I have observed frequent restarts of the scheduler, and consequently, some tasks fail to initiate. In the logs, I notice errors related to Celery and the database connection. Specifically, when executing the following query:

SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback

The final reported error is:

psycopg2.OperationalError: server closed the connection unexpectedly

and further indicates:

ForkPoolWorker-8] Failed operation _store_result.

It seems to be an issue involving Celery and the external database (PostgreSQL 14, in our case), but I'm uncertain which parameter might be relevant to resolve this error. It's worth mentioning that we use three workers with CeleryExecutor and run approximately 100 tasks daily at various times.

I would greatly appreciate your advice on potential solutions or adjustments that could help mitigate this issue.

Thank you for your time and assistance.

[lohyenshen]

we are encountering the same issue too.

[potiuk]

I suggest you:

• upgrade to latest version of airflow (it has newer celery version bound)
• or upgrade your celery to latest verion

There were some stability issues reported in earlier celery versions - you might take a look at their changelogs, but generally this is all celery related and by upgrading to newer versions (either celery or airflow) you are taking advantage of all bugfixes they implemented.

[lohyenshen]

hi @potiuk , could you enlighten us what might the common reported issues be?

we are using airflow with git tag of helm-chart/1.16.0
apache-airflow = {version = "2.10.5"
apache-airflow-providers-celery="3.12.2" (which is the latest stable celery version)

May I know in which explicit airflow version has this issue resolved?

[potiuk]

No idea, And I have no idea if a) there is an issue b) whether it solved.

Same as you, you might look through changelog - of both Airlfow and Celery and see if there is a similar issue, but generally speaking upgrading to latest versions of both Airflow and Celery increases your chances that it is solved (if that is a celery issue that is). But - since this is a celery issue, I suggest you ask them - maybe they might have better answers and know things by heart.

[lohyenshen]

thanks @potiuk , appreciate it.

[lohyenshen]

hi @avidalco,
any chance that you were able to resolve this issue?

[avidalco]

Hi lohyenshen,

We've upgraded to Airflow 2.10.5 using Helm chart version 1.18.0, but the issue is still there.
Right now, we're looking into the database performance because our Airflow installation is inconsistent.

[avidalco]

Hi lohyenshen,

We were able to resolve the issue after improving the database storage performance.