Repository navigation
Bound and harden the task-UUID scan fallback #71489
Description
GlueJobOperator._find_job_run_id_by_task_uuid (the task-UUID scan fallback, used when neither task_state_store nor a cached XCom id has a prior run) has a few pre-existing quality issues, carried over verbatim from main, that are worth cleaning up:
- Unbounded pagination. It's a
while Trueoverget_job_runs(MaxResults=50)with no page cap and no age cutoff. The no-match case — the prior attempt died before it ever calledStartJobRun, which is the common retry shape — is exactly the one that walks the job's entire run history before giving up. - Failure is silently swallowed. The two
except Exceptionblocks around this scan (and the XCom lookup before it) log atwarninglevel and returnNone, after which the operator submits fresh — straight intoConcurrentRunsExceededExceptionagainst a run that's still alive. There's no error-level signal telling anyone why. - The
except Exceptionis broader than it needs to be. It should be narrowed toexcept ClientError, so a real bug in the surrounding code doesn't get silently absorbed the same way as a transient AWS-side issue. - Missing IAM documentation. Anyone relying on this fallback (via
resume_glue_job_on_retry=Truetoday, or explicitdurable=Trueon Airflow <3.3 going forward) needsglue:GetJobRunsin their task policy, on top ofStartJobRun/GetJobRun. The docs don't currently call that out.
None of this is new — it's present on main today, gated behind the opt-in resume_glue_job_on_retry flag. Raised during review of #71211, initially flagged as more urgent because that PR was going to make the scan reachable by default; the default-flip was scoped back to explicit opt-in for Airflow <3.3 (and to a narrow crash-recovery window on 3.3+) in the same PR, so the original urgency no longer applies — but the underlying issues are still real for anyone who does hit this path.
Fix direction: bound the walk (a page cap, or stop once a run's StartedOn predates the DAG run), narrow the exception handling, log the fallthrough at error level with the actual reason, and document the IAM requirement.
See #71211 (comment) for the original discussion.
Activity
[-]GlueJobOperator: bound and harden the task-UUID scan fallback (_find_job_run_id_by_task_uuid)[/-][+]Bound and harden the task-UUID scan fallback[/+]MannXo commentedon Aug 12, 2026
I'd like to give this a shot if it's unassigned.
atanishka308 commentedon Aug 12, 2026
I would like to work on this issue.
Johnkothapalli commentedon Aug 13, 2026
I'm working on this. I'll keep the change to the issue's defined scope: bound the Glue job-run fallback scan, narrow AWS handling to ClientError with actionable logging, document glue:GetJobRuns, and add focused mocked pagination/error tests. I'll link the PR here once the provider checks pass.
Drafted-by: Codex (GPT-5) (no human review before posting)
Johnkothapalli commentedon Aug 13, 2026
I found that #71546 was already open before my comment and covers this scope, so I won't create a duplicate implementation. Sorry for the extra notification.
Drafted-by: Codex (GPT-5) (no human review before posting)
jasperjonkhans commentedon Aug 30, 2026
Hello, I would like to work on this, if still available.
LeonxLJX commentedon Aug 31, 2026
Hi! I'd like to work on bounding and hardening the task-UUID scan fallback. Thanks!
LeonxLJX commentedon Sep 1, 2026
Hi! I'd like to work on this. I'll reproduce, locate the root cause, implement a minimal fix with a regression test, and verify against the module's test suite. Please let me know if I can proceed / be assigned.
neevmodh commentedon Sep 22, 2026
Hi @amoghrajesh, I'd like to work on this. Plan per your fix direction:
_find_job_run_id_by_task_uuid(page cap and/or stop once a run'sStartedOnpredates the DAG run)except Exceptiontoexcept ClientErroraround the scan/XCom lookuperrorlevel with the actual reason instead of swallowing atwarningglue:GetJobRunsIAM requirement alongside the existingStartJobRun/GetJobRundocsWill add/extend tests covering the bounded walk and narrowed exception handling. Will open a PR.
Narrow exception handling in Glue job run fallback scan