Skip to content

Propagate OpenLineage parent and root job types to triggered Dags - #75303

Merged
mobuchowski merged 6 commits into
apache:mainfrom
kacpermuda:fix-trigger-dagrun-openlineage-root-job-type
Oct 26, 2026
Merged

mobuchowski merged 6 commits into
apache:mainfrom
kacpermuda:fix-trigger-dagrun-openlineage-root-job-type

Conversation

@kacpermuda

@kacpermuda kacpermuda commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Add lineage_job_type and lineage_root_job_type macros to the OpenLineage provider, and have TriggerDagRunOperator inject parentJobType and rootParentJobType into the triggered Dag run conf. When present and complete in the conf, the job types are added as the jobType facet of the parent and root jobs in the ParentRunFacet. Root job type is only inherited when the conf root is the one actually used, so a type is never paired with a different job.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Sonnet 5.5)

Generated-by: Claude Code (Sonnet 5.5) following the guidelines

@kacpermuda
kacpermuda marked this pull request as ready for review October 21, 2026 11:34
@kacpermuda
kacpermuda force-pushed the fix-trigger-dagrun-openlineage-root-job-type branch from 6ed9e41 to 7684b09 Compare October 21, 2026 11:42
@kacpermuda kacpermuda changed the title Propagate rootParentJobType through TriggerDagRunOperator Propagate OpenLineage parent and root job types to triggered Dags Oct 21, 2026
@kacpermuda
kacpermuda force-pushed the fix-trigger-dagrun-openlineage-root-job-type branch from f362269 to 0ab02e1 Compare October 21, 2026 15:40
kacpermuda and others added 6 commits October 26, 2026 14:23
When a Dag is triggered by TriggerDagRunOperator, the OpenLineage parent
info injected into the triggered Dag run conf dropped rootParentJobType.
The root job type set by an upstream caller was lost, so downstream
OpenLineage events could not report the correct root job type.

Co-Authored-By: Claude <noreply@anthropic.com>
…Dags

Reading rootParentJobType straight from the Dag run conf ignored the
validation the OpenLineage macros apply to root identifiers, so a job
type belonging to a rejected root could be paired with a different job.
Resolving the types in the OpenLineage provider keeps them consistent
with the root identifiers they describe.

Co-Authored-By: Claude <noreply@anthropic.com>
Job types forwarded through the Dag run conf were only exposed by the
macros, so events of a triggered Dag did not describe the type of their
parent and root jobs. Only complete job type information is accepted
from the conf, as a partial JobTypeJobFacet would not be a reliable
description of the job.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Task event parent facets and SQL child events did not describe the
type of their parent and root jobs even when it was known, and the
review of the job type handling found a few places where a type could
be paired with a different job or dropped.

Co-Authored-By: Claude <noreply@anthropic.com>
@kacpermuda
kacpermuda force-pushed the fix-trigger-dagrun-openlineage-root-job-type branch from 32787d1 to d258e85 Compare October 26, 2026 13:23
@mobuchowski
mobuchowski merged commit eb7a359 into apache:main Oct 26, 2026
148 checks passed
@kacpermuda
kacpermuda deleted the fix-trigger-dagrun-openlineage-root-job-type branch October 26, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants