Skip to content

Handle TaskNotFound in triggerer to prevent stale triggers from blocking batch - #69852

Closed
hanumanraj07 wants to merge 4 commits into
apache:mainfrom
hanumanraj07:fix-stale-trigger-blocking-workloads
Closed

hanumanraj07 wants to merge 4 commits into
apache:mainfrom
hanumanraj07:fix-stale-trigger-blocking-workloads

Conversation

@hanumanraj07

Copy link
Copy Markdown

What

Catches TaskNotFound in the triggerer supervisor's _create_workload method so that one stale task-associated trigger cannot abort the entire batch of healthy trigger workloads.

Closes: #69841

Why

The triggerer supervisor loads the pinned serialized Dag for every task-associated trigger to check whether the task uses start_from_trigger. If the serialized Dag exists but no longer contains the deferred task, get_task() raises TaskNotFound. The exception escapes the per-batch loop in build_trigger_workloads, so one stale task instance prevents unrelated healthy triggers in the same batch from starting and can repeatedly restart the triggerer supervisor.

How

  • Wrapped get_task() in _create_workload with a try/except TaskNotFound block
  • When caught, logs a warning and sets task = None, falling through to the context-free RunTrigger return (matching behavior when serialized_dag_model is None)
  • Changed task.start_from_trigger check to task is not None and task.start_from_trigger
  • Added unit test verifying the fix

@potiuk potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jul 18, 2026

@steveahnahn steveahnahn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I verified the test fails on main and passes with the fix. Good use of TaskNotFound rather than a blanket except, LGTM.

@kaxil

kaxil commented Oct 1, 2026

Copy link
Copy Markdown
Member

Closing this. #69988 made the same change on main: the triggerer wraps get_task() in try/except TaskNotFound and only treats the task as start_from_trigger when it was found.

@kaxil kaxil closed this Oct 1, 2026
@hanumanraj07
hanumanraj07 deleted the fix-stale-trigger-blocking-workloads branch October 2, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Triggerer ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prevent stale task triggers from blocking healthy workloads

4 participants