Repository navigation
Snowflake, Databricks: emit query lineage via OpenLineage public API - #73655
Merged
Conversation
boring-cyborg
Bot
added
area:providers
provider:databricks
provider:openlineage
kacpermuda
marked this pull request as ready for review
September 24, 2026 11:18
kacpermuda
force-pushed
the
kacpermuda/o11y-3149-fix-snowflake-and-databricks-ol-query-extraction-in-airflow
branch
from
September 24, 2026 13:31
d453143 to
a9ca6f4
Compare
mobuchowski
reviewed
Sep 28, 2026
kacpermuda
force-pushed
the
kacpermuda/o11y-3149-fix-snowflake-and-databricks-ol-query-extraction-in-airflow
branch
from
October 13, 2026 09:08
a9ca6f4 to
774b48c
Compare
kacpermuda
force-pushed
the
kacpermuda/o11y-3149-fix-snowflake-and-databricks-ol-query-extraction-in-airflow
branch
from
October 13, 2026 09:24
774b48c to
f9f12d1
Compare
mobuchowski
requested changes
Oct 13, 2026
kacpermuda
force-pushed
the
kacpermuda/o11y-3149-fix-snowflake-and-databricks-ol-query-extraction-in-airflow
branch
from
October 13, 2026 12:11
f9f12d1 to
6c16eee
Compare
mobuchowski
approved these changes
Oct 13, 2026
Contributor
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
Snowflake and Databricks each duplicated their own logic for building and emitting per-query OpenLineage lineage events, instead of using the OpenLineage provider's own public API for that. This replaces both with calls to the provider's `emit_query_lineage` API. This also fixes a bug where the START and COMPLETE events for a query shared a single mutable Run/Job object, letting a facet added while emitting one event leak into, or get corrupted alongside, the other.
The hook-lineage query sub-job name changed from "...query.N" to "...hll_query.N" to avoid colliding with the "...query.N" naming used by the Snowflake/Databricks query-lineage emitters added in this PR. The policy/hook-lineage e2e fixture still expected the old name.
# Conflicts: # uv.lock
kacpermuda
force-pushed
the
kacpermuda/o11y-3149-fix-snowflake-and-databricks-ol-query-extraction-in-airflow
branch
from
October 14, 2026 09:23
6c16eee to
b5bc5de
Compare
kacpermuda
deleted the
kacpermuda/o11y-3149-fix-snowflake-and-databricks-ol-query-extraction-in-airflow
branch
October 14, 2026 15:26
1 task done
This was referenced Oct 20, 2026
This was referenced Oct 29, 2026
Merged
1 task done
potiuk
pushed a commit
that referenced
this pull request
Nov 4, 2026
…73655) (#76450) * Snowflake, Databricks: emit query lineage via OpenLineage public API Snowflake and Databricks each duplicated their own logic for building and emitting per-query OpenLineage lineage events, instead of using the OpenLineage provider's own public API for that. This replaces both with calls to the provider's `emit_query_lineage` API. This also fixes a bug where the START and COMPLETE events for a query shared a single mutable Run/Job object, letting a facet added while emitting one event leak into, or get corrupted alongside, the other. * Fix expected-events fixture for the SQL hook-lineage job rename The hook-lineage query sub-job name changed from "...query.N" to "...hll_query.N" to avoid colliding with the "...query.N" naming used by the Snowflake/Databricks query-lineage emitters added in this PR. The policy/hook-lineage e2e fixture still expected the old name. * Bump openlineage version in pyproject.toml * pre commit # Conflicts: # uv.lock * address review comments (cherry picked from commit 22f86cb) Co-authored-by: Kacper Muda <mudakacper@gmail.com>
vatsrahul1001
added a commit
that referenced
this pull request
Nov 4, 2026
…73655) (#76450) * Snowflake, Databricks: emit query lineage via OpenLineage public API Snowflake and Databricks each duplicated their own logic for building and emitting per-query OpenLineage lineage events, instead of using the OpenLineage provider's own public API for that. This replaces both with calls to the provider's `emit_query_lineage` API. This also fixes a bug where the START and COMPLETE events for a query shared a single mutable Run/Job object, letting a facet added while emitting one event leak into, or get corrupted alongside, the other. * Fix expected-events fixture for the SQL hook-lineage job rename The hook-lineage query sub-job name changed from "...query.N" to "...hll_query.N" to avoid colliding with the "...query.N" naming used by the Snowflake/Databricks query-lineage emitters added in this PR. The policy/hook-lineage e2e fixture still expected the old name. * Bump openlineage version in pyproject.toml * pre commit # Conflicts: # uv.lock * address review comments (cherry picked from commit 22f86cb) Co-authored-by: Kacper Muda <mudakacper@gmail.com>
vatsrahul1001
added a commit
that referenced
this pull request
Nov 4, 2026
…73655) (#76450) * Snowflake, Databricks: emit query lineage via OpenLineage public API Snowflake and Databricks each duplicated their own logic for building and emitting per-query OpenLineage lineage events, instead of using the OpenLineage provider's own public API for that. This replaces both with calls to the provider's `emit_query_lineage` API. This also fixes a bug where the START and COMPLETE events for a query shared a single mutable Run/Job object, letting a facet added while emitting one event leak into, or get corrupted alongside, the other. * Fix expected-events fixture for the SQL hook-lineage job rename The hook-lineage query sub-job name changed from "...query.N" to "...hll_query.N" to avoid colliding with the "...query.N" naming used by the Snowflake/Databricks query-lineage emitters added in this PR. The policy/hook-lineage e2e fixture still expected the old name. * Bump openlineage version in pyproject.toml * pre commit # Conflicts: # uv.lock * address review comments (cherry picked from commit 22f86cb) Co-authored-by: Kacper Muda <mudakacper@gmail.com>
vatsrahul1001
added a commit
that referenced
this pull request
Nov 4, 2026
…73655) (#76450) * Snowflake, Databricks: emit query lineage via OpenLineage public API Snowflake and Databricks each duplicated their own logic for building and emitting per-query OpenLineage lineage events, instead of using the OpenLineage provider's own public API for that. This replaces both with calls to the provider's `emit_query_lineage` API. This also fixes a bug where the START and COMPLETE events for a query shared a single mutable Run/Job object, letting a facet added while emitting one event leak into, or get corrupted alongside, the other. * Fix expected-events fixture for the SQL hook-lineage job rename The hook-lineage query sub-job name changed from "...query.N" to "...hll_query.N" to avoid colliding with the "...query.N" naming used by the Snowflake/Databricks query-lineage emitters added in this PR. The policy/hook-lineage e2e fixture still expected the old name. * Bump openlineage version in pyproject.toml * pre commit # Conflicts: # uv.lock * address review comments (cherry picked from commit 22f86cb) Co-authored-by: Kacper Muda <mudakacper@gmail.com>
vatsrahul1001
added a commit
that referenced
this pull request
Nov 5, 2026
…73655) (#76450) * Snowflake, Databricks: emit query lineage via OpenLineage public API Snowflake and Databricks each duplicated their own logic for building and emitting per-query OpenLineage lineage events, instead of using the OpenLineage provider's own public API for that. This replaces both with calls to the provider's `emit_query_lineage` API. This also fixes a bug where the START and COMPLETE events for a query shared a single mutable Run/Job object, letting a facet added while emitting one event leak into, or get corrupted alongside, the other. * Fix expected-events fixture for the SQL hook-lineage job rename The hook-lineage query sub-job name changed from "...query.N" to "...hll_query.N" to avoid colliding with the "...query.N" naming used by the Snowflake/Databricks query-lineage emitters added in this PR. The policy/hook-lineage e2e fixture still expected the old name. * Bump openlineage version in pyproject.toml * pre commit # Conflicts: # uv.lock * address review comments (cherry picked from commit 22f86cb) Co-authored-by: Kacper Muda <mudakacper@gmail.com>
vatsrahul1001
added a commit
that referenced
this pull request
Nov 5, 2026
…73655) (#76450) * Snowflake, Databricks: emit query lineage via OpenLineage public API Snowflake and Databricks each duplicated their own logic for building and emitting per-query OpenLineage lineage events, instead of using the OpenLineage provider's own public API for that. This replaces both with calls to the provider's `emit_query_lineage` API. This also fixes a bug where the START and COMPLETE events for a query shared a single mutable Run/Job object, letting a facet added while emitting one event leak into, or get corrupted alongside, the other. * Fix expected-events fixture for the SQL hook-lineage job rename The hook-lineage query sub-job name changed from "...query.N" to "...hll_query.N" to avoid colliding with the "...query.N" naming used by the Snowflake/Databricks query-lineage emitters added in this PR. The policy/hook-lineage e2e fixture still expected the old name. * Bump openlineage version in pyproject.toml * pre commit # Conflicts: # uv.lock * address review comments (cherry picked from commit 22f86cb) Co-authored-by: Kacper Muda <mudakacper@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Snowflake and Databricks each duplicated their own logic for building and emitting per-query OpenLineage lineage events, instead of using the OpenLineage provider's own public API for it. This replaces both with calls to the provider's
emit_query_lineageAPI.This also fixes a bug where the START and COMPLETE events for a query shared a single mutable Run/Job object, letting a facet added while emitting one event leak into, or get corrupted alongside, the other.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code following the guidelines