Repository navigation
ALL_DONE trigger rule not respected in TaskFlow API for upstream failures #23942
Description
Apache Airflow version
2.3.0
What happened
Hello!
I'm using taskflow api for my dag and was trying to run the last task even if the previous task failed, I used @task(trigger_rule=TriggerRule.ALL_DONE) but i still got the ending task failed too.
The UI confirms that I declared the trigger_rule "all_done"
What you think should happen instead
No response
How to reproduce
This is the whole code
import logging
import pendulum
from airflow.decorators import task, dag
from airflow.utils.trigger_rule import TriggerRule
@dag(
schedule_interval=None,
start_date=pendulum.datetime(2021, 5, 10, tz="UTC"),
catchup=False,
)
def taskflow_trigger():
@task()
def first_task():
return "first_task"
@task
def second_task(value):
return "second_task"
@task
def task_to_fail(value):
data = {"test": 1}
val = data["not_here"]
return val
@task(trigger_rule=TriggerRule.ALL_DONE)
def end(value):
return "yes"
op_1 = first_task()
op_3 = task_to_fail(op_1)
op_2 = second_task(op_1)
end([op_3, op_2])
taskflow_trigger = taskflow_trigger()
Operating System
"Debian GNU/Linux 11 (bullseye)
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon May 26, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
josh-fell commentedon May 26, 2022
IMO this should be expected. The
endtask is expecting an input from thetask_to_failoutput but the latter task has failed and doesn't produce anXComArg. Theendtask does actually run so the trigger rule is respected, but fails with:What would you like the behavior to be in this scenario?
ZouhirSarrar commentedon May 26, 2022
i want the end task to continue working
[-]Trigger_rule don't work while using task decorator[/-][+]`ALL_DONE` trigger rule not respected in TaskFlow API[/+][-]`ALL_DONE` trigger rule not respected in TaskFlow API[/-][+]`ALL_DONE` trigger rule not respected in TaskFlow API for upstream failures[/+]josh-fell commentedon Jun 13, 2022
A similar issue was raised when upstream tasks are skipped. The point of "the downstream should not run if any of its depended upstream XCom values fail to resolve" (in this comment) was brought up again.
@uranusjr curious for your take on this issue generally.
I feel that, as this issue is written, the consensus is that a downstream TaskFlow task should not execute if a dependent, upstream
XComcannot be found.uranusjr commentedon Jun 14, 2022
Since an XCom push can never be None1, we can probably just set the value to None in this case. Combining this with #24401, we probably need to do some additional checks in
XComArg.resolve()to take account the current task’s trigger rule, and resolve to None when appropriate.Footnotes
A design decision I didn’t agree with but that ship sailed long ago. ↩
ashb commentedon Jul 6, 2022
Duplicate of #24338 (It's the same underlying cause)