Repository navigation
Surface errors from OpenLineage database info lookups without losing lineage - #74805
Merged
mobuchowski merged 7 commits intoOct 27, 2026
Merged
Conversation
kacpermuda
marked this pull request as ready for review
October 15, 2026 15:35
mobuchowski
requested changes
Oct 26, 2026
kacpermuda
force-pushed
the
ol-dont-swallow-db-info-attribute-error
branch
from
October 26, 2026 16:42
8513859 to
71e6b8f
Compare
kacpermuda
marked this pull request as draft
October 26, 2026 16:51
kacpermuda
marked this pull request as ready for review
October 26, 2026 16:51
mobuchowski
reviewed
Oct 26, 2026
mobuchowski
left a comment
Contributor
There was a problem hiding this comment.
The description still says the method is "called outside any try/except, so real errors propagate (the extractor already turns them into a warning)". After the latest commits it works the other way round: errors from the four get_openlineage_* hook helpers are caught at each call site, logged at WARNING, and lineage falls back so partial lineage is kept. The change also now covers the dialect, default schema and database-specific lineage lookups and the common.compat fallback, not just get_openlineage_database_info. Worth updating the description, and probably the title too (it's no longer only about AttributeError), so reviewers and the changelog reflect the final behaviour.
mobuchowski
reviewed
Oct 27, 2026
SQL OpenLineage extraction called hook.get_openlineage_database_info() inside `except AttributeError` blocks, meant to detect hooks that don't implement the method. The same handler also swallowed AttributeErrors raised from inside the method (e.g. a broken optional import in SnowflakeHook.get_uri()), logging only at debug level. Lineage silently came out without inputs/outputs and nothing in the task logs said why. Look the method up with getattr() instead and call it outside any try/except, so real errors propagate (the extractor already turns them into a warning and still emits events). Also log failures in SQL hook lineage at warning level, with details at debug. Co-Authored-By: Claude <noreply@anthropic.com>
Reviewers pointed out that letting AttributeError escape lost all lineage for the task, including GCS outputs and query ids that don't depend on the failing method, and that hooks without the method were warned about. A missing method is now detected with getattr and stays silent. An error raised inside the method is logged as a warning and lineage falls back to what can still be built. The same applies to the dialect, default schema and database specific lineage lookups. Co-Authored-By: Claude <noreply@anthropic.com>
…lback The fallback used when the OpenLineage provider is too old to ship get_openlineage_facets_with_sql had the same except AttributeError blocks, so an error raised inside the hook's database info, dialect or default schema lookup was reported as a hook without those methods. Co-Authored-By: Claude <noreply@anthropic.com>
A failing get_sqlalchemy_engine() in SQLExecuteQueryOperator still dropped all lineage of the task, and the new warnings named the hook object and the wrong method, which made failures hard to attribute to a connection. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
… lineage A hook whose dialect or default schema lookup fails would log the same warning for every SQL extra of a task. The SQLParser is now built once per hook and reused, and the SQL parsing warning names the connection. Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda
force-pushed
the
ol-dont-swallow-db-info-attribute-error
branch
from
October 27, 2026 13:06
3b8399c to
0d19f01
Compare
mobuchowski
approved these changes
Oct 27, 2026
Co-Authored-By: Claude <noreply@anthropic.com>
7 of 36 tasks
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SQL OpenLineage extraction called the hook's
get_openlineage_*helpers insideexcept AttributeErrorblocks meant to detect hooks that don't implement them. They also swallowedAttributeErrors raised inside those methods (e.g. a broken optional import inSnowflakeHook.get_uri()), logging only at debug level, so lineage silently came out empty with nothing in the task logs explaining why.Now a missing method is detected with
getattr()and treated as "not supported" (silent). An error raised insideget_openlineage_database_info,get_openlineage_database_dialect,get_openlineage_default_schemaorget_openlineage_database_specific_lineageis caught at the call site, logged at WARNING with the connection id (traceback at debug), and lineage falls back to whatever can still be built. This applies toSQLExecuteQueryOperator,sqlparser.get_openlineage_facets_with_sql, SQL hook lineage and thecommon.compatfallback. A failingget_sqlalchemy_engine()in the operator no longer drops lineage either.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code following the guidelines
🤖 Generated with Claude Code