Skip to content

Surface errors from OpenLineage database info lookups without losing lineage - #74805

Merged
mobuchowski merged 7 commits into
apache:mainfrom
kacpermuda:ol-dont-swallow-db-info-attribute-error
Oct 27, 2026
Merged

mobuchowski merged 7 commits into
apache:mainfrom
kacpermuda:ol-dont-swallow-db-info-attribute-error

Conversation

@kacpermuda

@kacpermuda kacpermuda commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

SQL OpenLineage extraction called the hook's get_openlineage_* helpers inside except AttributeError blocks meant to detect hooks that don't implement them. They also swallowed AttributeErrors raised inside those methods (e.g. a broken optional import in SnowflakeHook.get_uri()), logging only at debug level, so lineage silently came out empty with nothing in the task logs explaining why.

Now a missing method is detected with getattr() and treated as "not supported" (silent). An error raised inside get_openlineage_database_info, get_openlineage_database_dialect, get_openlineage_default_schema or get_openlineage_database_specific_lineage is caught at the call site, logged at WARNING with the connection id (traceback at debug), and lineage falls back to whatever can still be built. This applies to SQLExecuteQueryOperator, sqlparser.get_openlineage_facets_with_sql, SQL hook lineage and the common.compat fallback. A failing get_sqlalchemy_engine() in the operator no longer drops lineage either.


Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code)

Generated-by: Claude Code following the guidelines


🤖 Generated with Claude Code

@kacpermuda
kacpermuda force-pushed the ol-dont-swallow-db-info-attribute-error branch from 8513859 to 71e6b8f Compare October 26, 2026 16:42
@kacpermuda kacpermuda added the skip common compat check Skips common compat provider modification check label Oct 26, 2026
@kacpermuda
kacpermuda marked this pull request as draft October 26, 2026 16:51
@kacpermuda
kacpermuda marked this pull request as ready for review October 26, 2026 16:51

@mobuchowski mobuchowski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description still says the method is "called outside any try/except, so real errors propagate (the extractor already turns them into a warning)". After the latest commits it works the other way round: errors from the four get_openlineage_* hook helpers are caught at each call site, logged at WARNING, and lineage falls back so partial lineage is kept. The change also now covers the dialect, default schema and database-specific lineage lookups and the common.compat fallback, not just get_openlineage_database_info. Worth updating the description, and probably the title too (it's no longer only about AttributeError), so reviewers and the changelog reflect the final behaviour.

@kacpermuda kacpermuda changed the title Don't silence AttributeError from OpenLineage database info lookup Surface errors from OpenLineage database info lookups without losing lineage Oct 26, 2026
kacpermuda and others added 6 commits October 27, 2026 14:06
SQL OpenLineage extraction called hook.get_openlineage_database_info()
inside `except AttributeError` blocks, meant to detect hooks that don't
implement the method. The same handler also swallowed AttributeErrors
raised from inside the method (e.g. a broken optional import in
SnowflakeHook.get_uri()), logging only at debug level. Lineage silently
came out without inputs/outputs and nothing in the task logs said why.

Look the method up with getattr() instead and call it outside any
try/except, so real errors propagate (the extractor already turns them
into a warning and still emits events). Also log failures in SQL hook
lineage at warning level, with details at debug.

Co-Authored-By: Claude <noreply@anthropic.com>
Reviewers pointed out that letting AttributeError escape lost all lineage
for the task, including GCS outputs and query ids that don't depend on the
failing method, and that hooks without the method were warned about.

A missing method is now detected with getattr and stays silent. An error
raised inside the method is logged as a warning and lineage falls back to
what can still be built. The same applies to the dialect, default schema
and database specific lineage lookups.

Co-Authored-By: Claude <noreply@anthropic.com>
…lback

The fallback used when the OpenLineage provider is too old to ship
get_openlineage_facets_with_sql had the same except AttributeError blocks,
so an error raised inside the hook's database info, dialect or default
schema lookup was reported as a hook without those methods.

Co-Authored-By: Claude <noreply@anthropic.com>
A failing get_sqlalchemy_engine() in SQLExecuteQueryOperator still dropped
all lineage of the task, and the new warnings named the hook object and the
wrong method, which made failures hard to attribute to a connection.

Co-Authored-By: Claude <noreply@anthropic.com>
… lineage

A hook whose dialect or default schema lookup fails would log the same
warning for every SQL extra of a task. The SQLParser is now built once per
hook and reused, and the SQL parsing warning names the connection.

Co-Authored-By: Claude <noreply@anthropic.com>
@kacpermuda
kacpermuda force-pushed the ol-dont-swallow-db-info-attribute-error branch from 3b8399c to 0d19f01 Compare October 27, 2026 13:06
Co-Authored-By: Claude <noreply@anthropic.com>
@mobuchowski
mobuchowski merged commit ae50f95 into apache:main Oct 27, 2026
107 checks passed
@kacpermuda
kacpermuda deleted the ol-dont-swallow-db-info-attribute-error branch October 27, 2026 16:13
@shahar1 shahar1 mentioned this pull request Nov 4, 2026
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants