[Khairajani]

Describe your changes:

Fixes #37258

Query and view column lineage now reaches the fields of STRUCT and ARRAY columns. Before this change it stopped at the top-level column, so tag and description propagation over lineage never reached nested fields.

• Struct selected as-is (SELECT customer FROM t): the customer → customer edge now also carries one edge per nested field the two columns share by name, at every depth (customer.address.zip → customer.address.zip).

• Field paths (SELECT customer.*, SELECT customer.email AS e): the parser reads customer as a table that the query never reads. These entries are now resolved against the source table's STRUCT. Before, they produced no lineage.

• No guessing: the parser drops a field path's qualifier, so a path is resolved only when nothing else can own its name:

  • exactly one source table;
  • the source is a table, not a table-valued function or stage;
  • no intermediate table;
  • no subquery or CTE;
  • exactly one STRUCT of that name in the table.

  Otherwise the path gets no lineage instead of lineage from the wrong column. This rule is recorded in ADR:2026-11-06-sql-column-lineage-follows-nested-fields.

Type of change:

• Bug fix

Behaviour change / breaking changes:

• Not breaking. Edges are only added. Every edge produced today is still produced, and no schema, API or config changes.

• On by default, with no flag. It fills in lineage the SQL already implies, so there is nothing to opt into.

• Applies to every connector that ingests nested columns. Covered by tests: BigQuery, Databricks, Spark, Hive, Trino, Athena and Snowflake. The next lineage run adds the nested edges; tables with wide STRUCTs get noticeably more column edges.

• Propagation reaches nested fields. Tag and description propagation now reaches nested fields in downstream views, which is the point of the change. Users who propagate tags will see them land on those fields after the next lineage run.

• Still no field-level lineage for:

  • field paths in joins, over subqueries or CTEs;
  • UNNEST-based array access;
  • Snowflake col:field paths.

  These behave the same as before the change.

High-level design:

N/A — small change in sql_lineage.py, plus a LineageParser.has_subqueries property in parser.py.

Tests:

Use cases covered

• A STRUCT and an ARRAY selected as-is map every nested field, at every depth.
• struct.* (unqualified and aliased) maps each field to its view column.
• struct.field AS alias and struct.inner.field AS alias map to the alias.
• The same field paths resolve in the Databricks, Spark, Hive, Trino, Athena and Snowflake dialects.
• These cases get no field lineage, never a wrong edge:
  • a join;
  • a subquery or CTE that could rename the STRUCT;
  • a multi-statement query;
  • two STRUCTs sharing the name.
• An unqualified table name keeps its own column lineage next to a field path.

Unit tests

• I added unit tests for the new/changed logic.
• Files added: ingestion/tests/unit/lineage/test_nested_column_lineage.py (22 tests). Each of them fails on main or fails when its guard is removed.
• Affected unit tests: every test that imports sql_lineage or parser, plus tests/unit/lineage/. 1013 passed, 0 failed (now also covers tests/unit/test_query_parser.py).
  • Not run: the two Snowflake test files, which need the Snowflake package that isn't installed locally.
  • Skipped: test_pgspider_lineage_unit.py, because the local env has no pgspider SQLAlchemy dialect.
  • The wider set (all lineage-related topology tests) fails exactly the same tests with and without this change, all from missing connector packages.
• Coverage: every changed statement in sql_lineage.py is executed by the tests.
• Static checks: I ran basedpyright 1.39.3 (CI's version) in CI's baseline mode on sql_lineage.py and parser.py; it reports 0 new errors. The ruff suppression check also passes.

Backend integration tests

• Not applicable (no backend API changes).

Ingestion integration tests

• Ran tests/integration/postgres/test_lineage.py against a local server: 4 passed.
• Pending: tests/integration/sql_server/test_lineage.py (needs the SQL Server container; it runs in the merge queue). The profiler and classifier integration tests on the affected list don't exercise column lineage.

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

End to end, against a local server, I created a BigQuery-type service with a users table holding customer STRUCT<email, address STRUCT<zip, city>> and events ARRAY<STRUCT<ip>>, with PII.Sensitive on customer.email, customer.address.zip and events.ip. I added three views and ran view lineage (get_view_lineage), then lineage-based tag propagation from users. Results, before → after:

• SELECT id, customer, events:
  • Lineage: 3 → 8 column edges.
  • Tags: none → PII.Sensitive on customer.email, customer.address.zip and events.ip.
• SELECT id, customer.*:
  • Lineage: 1 → 5 column edges.
  • Tags: none → PII.Sensitive on email and address.zip.
• SELECT customer.email AS customer_email, customer.address.zip AS zip:
  • Lineage: 0 → 2 column edges.
  • Tags: none → PII.Sensitive on both columns.

The untagged fields (id, address.city) stayed untagged. The server stored every nested column FQN, since lineage validation already resolves nested children.

UI screen recording:

Not applicable — ingestion-only fix with no UI change.

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: N/A, no schema change.
• For a UI feature or feature task: N/A, no UI change.
• I have added tests (unit / integration / Playwright as applicable) and listed them above.
• Self-review: I reviewed the final diff against REVIEW.md and fixed every 🔴 finding. Reviewed at 17c0321ee96; the bot follow-up is fae55f39005.
  • First pass, two 🔴, both fixed:
    • guards that no test could fail now have tests;
    • a subquery or CTE could rename a STRUCT and cause lineage from the wrong field, so field paths now skip those queries.
  • Bot review: a table-valued function or stage as the single source could expand into several tables. Fixed in fae55f39005; field paths now need a table source.
  • Second pass: two 🟡 test-strength nits, both fixed.
• Decision check: added ADR:2026-11-06-sql-column-lineage-follows-nested-fields.
• I have added a test that covers the exact scenario we are fixing.

🤖 Generated with Claude Code

[github-actions]

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[gitar-bot]

Code Review ✅ Approved 1 closed / 1 findings

🟡 Medium risk · Adding nested-field lineage enables tag and description propagation, and an incorrect edge could attach metadata to the wrong downstream fields.

Maps query and view column lineage through STRUCT and ARRAY<STRUCT> fields so tag and description propagation reaches nested columns, including field paths like customer.* and customer.email AS e. A field path is resolved only when exactly one source table and one matching STRUCT can own its name. The review's finding on STRUCT field paths through a UDF source getting lineage from the wrong table was resolved. No open issues found.

✅ 1 closed

✅ Edge Case: STRUCT field paths through a UDF source can get lineage from the wrong table

📄 ingestion/src/metadata/ingestion/lineage/sql_lineage.py:515 📄 ingestion/src/metadata/ingestion/lineage/sql_lineage.py:435-438
_key_struct_field_parents checks len(lineage_parser.source_tables) != 1, and a table-valued function (DataFunction) counts as that single source. get_source_table_names then expands the UDF into its underlying tables (__process_udf_es_results), and _create_lineage_by_table_name calls get_column_lineage once per underlying table. The new loop at lines 435-438 applies every <struct>.* key to whatever from_entity it receives, without checking from_table_raw_name. So in SELECT customer.email AS e FROM my_udf(), every table the UDF reads that has a customer STRUCT gets an edge to e. That happens even when the UDF joins several such tables or renames another column to customer. This is exactly the kind of guessed lineage that ADR:2026-11-06-sql-column-lineage-follows-nested-fields says must not be produced, because it would push tags onto the wrong columns. Fix: only resolve field paths when the one source is an actual table.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

_{For more verification details, comment gitar display:verbose.}

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
89.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud