Celery: Ignore redelivered message for already-running task

[anishgirianish]

Catch TaskAlreadyRunningError from the supervisor and raise Celery Ignore() to prevent the broker redelivery from being recorded as a task failure.

related: #58441

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Summary

Catch TaskAlreadyRunningError from the supervisor and raise Celery Ignore() to prevent a broker redelivery from being recorded as a task failure.

In Airflow 3.x, the scheduler processes executor events for RUNNING tasks (not just QUEUED as in 2.x). When Celery redelivers a message for a task that's already running, the worker hits a 409 from the API server. Without this fix, Celery records it as FAILURE, and the scheduler marks the still-running task as failed.

related: #58441
depends on: #60855

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[jscheffl]

Assume will fail unless #60855 is merged... which is fair. Can be re-based once the other PR is merged.

[potiuk]

@anishgirianish This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ mypy (type checking): Failing: CI image checks / MyPy checks (mypy-providers). Run prek --stage manual mypy-providers --all-files locally to reproduce. You need breeze ci-image build --python 3.10 for Docker-based mypy. See mypy (type checking) docs.
• ❌ Provider tests: Failing: Postgres tests: providers / DB-prov:Postgres:14:3.10:-amazon,celer...standard, MySQL tests: providers / DB-prov:MySQL:8.0:3.10:-amazon,celer...standard, Sqlite tests: providers / DB-prov:Sqlite:3.10:-amazon,celer...standard, Special tests / Min SQLAlchemy test: providers / DB-prov:MinSQLAlchemy-Postgres:14:3.10:-amazon,celer...standard, Special tests / Latest SQLAlchemy test: providers / DB-prov:LatestSQLAlchemy-Postgres:14:3.10:-amazon,celer...standard (+2 more). Run provider tests with breeze run pytest <provider-test-path> -xvs. See Provider tests docs.

Note: Your branch is 258 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

[jscheffl]

@potiuk This is an exception where your AI tooling is not right - all the errors are caused by the explicit split PR #60855 not being merged (yet). Once this is done a rebase most probably will fix all errors. Re-converted to "Ready for review".

[eladkal]

@anishgirianish can you resolve conflicts and rebase?
@jscheffl the code in this PR should work regardless of the change in #60855 as the change will be in airflow 3.1.x and the celery provider must work with airflow 2.11 as well

[anishgirianish]

@anishgirianish can you resolve conflicts and rebase? @jscheffl the code in this PR should work regardless of the change in #60855 as the change will be in airflow 3.1.x and the celery provider must work with airflow 2.11 as well

@eladkal rebased and updated thanks