Repository navigation
Fix handling AirflowFailException on_retry_callback #60172
Description
Description
This was previously possible in Airflow 2 using this solution. However, in Airflow 3 RuntimeTaskInstanceProtocol does not expose a method for modifying the task state directly.
My suggestion would be to create a new exception type that can be raised from on_retry_callback to request a state change to either success or failure.
Update: While this is not specified, it seema a bug that it does not work that way: AirflowFailException raised in on_retry_callback or any other state-change callback should immediately fail the task.
Use case/motivation
We have some long-running tasks (>24hr) that sometimes fail fast due to transient issues. We want to be able to retry whenever these fast fails occur. However, if a failure occurs hours into the task, we would prefer not to retry, as the task must be completed by a certain time and failures of this sort typically represent an actual issue we want to look into.
Related issues
Just the discussion item here from years ago as far as I can tell.
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jan 6, 2026
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
potiuk commentedon Jan 6, 2026
There is already exception for that:
There is no need to have "success" exception, you just exit from the method when it happens
potiuk commentedon Jan 6, 2026
Described in the docs here https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#special-exceptions
ndellosa95 commentedon Jan 7, 2026
@potiuk I think you are misunderstanding my request.
AirflowFailExceptiononly works when called from within operator/task code. When you call it within anon_retry_callbackit simply fails the callback and retries as normal. See the example here, which I linked above, for what I mean.potiuk commentedon Jan 7, 2026
So I guess you are saying that it's a bug in handling this exception - because there is no reason why raising AirflowFailException in on_retry_callback should continue retrying. If that's the case, then this is a bug IMHO
[-]Enable conditional retry logic in `on_retry_callback` in Airflow 3[/-][+]Fix handling AirflowFailException `on_retry_callback` [/+]5 remaining items
potiuk commentedon Mar 2, 2026
@ndellosa95 We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
IvanildoBarauna commentedon Mar 25, 2026
I working it
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Honor AirflowFailException raised inside on_retry_callback
Stop retries when retry callbacks raise AirflowFailException