Skip to content

[2.0] ingestion: upgrade pyarrow to 26 - #37236

Open
gh-bot-collate[bot] wants to merge 1 commit into
2.0from
backport/2.0/pr-37103
Open

gh-bot-collate[bot] wants to merge 1 commit into
2.0from
backport/2.0/pr-37103

Conversation

@gh-bot-collate

@gh-bot-collate gh-bot-collate Bot commented Nov 6, 2026

Copy link
Copy Markdown

Automated backport of #37103 to 2.0.

Source PR #37103
Author @Khairajani
Merge commit 6a26b3890f7b5827b04018ed2c6579f38fdc6163
Method Conflicts resolved by Claude — worth a read of the diff.

Resolution notes

Backport notes: PR #37103 (ingestion: upgrade pyarrow to 26) → 2.0

ingestion/setup.py

Git auto-merged this file cleanly (no conflict markers). Diff matches the
source PR's intent on top of 2.0's current content: pyarrow is split into
two mutually-exclusive, marker-gated entries —
pyarrow>=26.0.0,<27; python_version >= '3.11' and the new
pyarrow-py310>=23.0.1,<26; python_version < '3.11' — and every one of the
five consumers that referenced VERSIONS["pyarrow"] (storage-archive,
one more in COMMONS, the BigQuery plugin, the pyarrow extra, and test)
now also references VERSIONS["pyarrow-py310"]. No manual changes were
needed here.

ingestion/airflow-constraints-3.3.2.txt

Conflicted on the pyarrow== / pyasn1== lines. Resolved by taking the
PR's pyarrow bump (25.0.1 → 26.0.1) and the header comment addition
(..., sqlparse, pyarrow.), while keeping 2.0's own pyasn1==0.6.4 pin —
2.0's pyasn1 version had already diverged from main (main is on 0.6.5
via an unrelated, later security-override commit that was never backported)
and that bump is out of scope for this backport.

ingestion/uv.lock — needs manual attention

2.0 does not track ingestion/uv.lock at all (it predates the repo's
move to uv-based locking on main), so this is a modify/delete conflict:
base and main have the file, 2.0 never did. The correct resolution is to
keep it deleted on 2.0 — the pyarrow bump itself is already fully captured
by the two files above, and there is no 2.0 lockfile workflow for it to
plug into.

I could not perform that deletion in this session — the sandboxed Bash
tool available to me rejected every rm / git rm (both --cached and
full) invocation for this path with "requires approval," and no interactive
approval channel was available. The working tree currently still holds
main's full uv.lock content at ingestion/uv.lock (no conflict markers
in it — it's a clean "theirs" copy), and the index still has it as unmerged
(stage 1 = base, stage 3 = theirs, no stage 2).

Before finishing this cherry-pick, a reviewer/the staging step must run:

git rm -f ingestion/uv.lock

If that file is instead staged as-is (e.g. by a blanket git add -A), it
will wrongly reintroduce a large, unrelated uv lockfile into the 2.0 branch.


Opened from openmetadata-collate, which holds the credentials needed to open a PR here. It merges itself once this branch's required checks pass — no review needed. If a check fails it stays open and the author of #37103 is asked to look.

resolver run

* fix(security): bump pyarrow to 26 for SNYK-PYTHON-PYARROW-21279727

Widen the pin to <27 in setup.py, pin pyarrow==26.0.1 in the Airflow
constraints the ingestion images install against, and relock. pyarrow 26
requires Python >=3.11, so 3.10 keeps resolving 25.0.1.

* ingestion: require pyarrow 26 on Python 3.11+, keep 3.10 on the 23.0.1 range

(cherry picked from commit 6a26b38)
@gh-bot-collate
gh-bot-collate Bot requested a review from a team as a code owner November 6, 2026 06:16
@gh-bot-collate
gh-bot-collate Bot requested review from stout-yeoman and removed request for a team November 6, 2026 06:16
@gh-bot-collate gh-bot-collate Bot added the safe to test Add this label to run secure Github workflows on PRs label Nov 6, 2026
@gh-bot-collate
gh-bot-collate Bot enabled auto-merge (squash) November 6, 2026 06:16
@gh-bot-collate gh-bot-collate Bot mentioned this pull request Nov 6, 2026
3 tasks done
@sonarqubecloud

sonarqubecloud Bot commented Nov 6, 2026

Copy link
Copy Markdown

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants