Skip to content

Fix ForeignKeyViolation crash when rescheduling sensors during failure - #71966

Closed
saitejabandaru-in wants to merge 1 commit into
apache:mainfrom
saitejabandaru-in:fix-71923-reschedule-fk-crash
Closed

saitejabandaru-in wants to merge 1 commit into
apache:mainfrom
saitejabandaru-in:fix-71923-reschedule-fk-crash

Conversation

@saitejabandaru-in

Copy link
Copy Markdown

Closes #71923.

Motivation

When a sensor fails, the scheduler calls TaskInstance.fetch_handle_failure_context. This method clears older task_reschedule records and mutates ti.id to a new UUID by calling ti.prepare_db_for_next_try, and then it calls on_task_instance_failed listener hooks before flushing the updated ti.id to the database.

If a listener hook blocks for seconds (e.g. OpenLineage timing out), the transaction window is extended. Concurrently, an API server might process a reschedule request from the supervisor and insert a new task_reschedule row using the old ti.id. When the listener finishes, the scheduler flushes UPDATE task_instance SET id = <new_id> WHERE id = <old_id>. But since the newly inserted task_reschedule points to old_id, PostgreSQL rejects the update with a ForeignKeyViolation, causing the scheduler to crash.

Changes

  • Moved ti.prepare_db_for_next_try(session) in fetch_handle_failure_context to run after the listener hooks.
  • This reduces the race window between the task_reschedule delete and the ti.id update flush from seconds down to microseconds.
  • Bonus correctness fix: Listener hooks now receive the TaskInstance with the old (actual) UUID that experienced the failure, rather than the prematurely rolled new UUID that hasn't run yet.

This moves the `prepare_db_for_next_try` call in `fetch_handle_failure_context` to execute after the listener hook (`on_task_instance_failed`). Previously, a slow listener could extend the window between deleting old `task_reschedule` rows and assigning a new `ti.id`, allowing a concurrent reschedule insert (using the old `ti.id`) to arrive and block the subsequent `ti.id` update flush with a ForeignKeyViolation. This closes the gap to microseconds and ensures listeners see the correct (old) `ti.id` that actually failed. Fixes apache#71923.
@kaxil

kaxil commented Oct 1, 2026

Copy link
Copy Markdown
Member

Closing this. Main already notifies listeners before allocating the next try: fetch_handle_failure_context calls ti.notify_failure(error) and only then ti.prepare_db_for_next_try(session) (#73554). The block this PR edits no longer exists, and it adds no test.

@kaxil kaxil closed this Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants