Skip to content

Fixed concatKeepingSchema turning ColumnGroups into DataColumns#1764

Merged
koperagen merged 1 commit intomasterfrom
arrow-read-concat-struct-batches
Mar 24, 2026
Merged

Fixed concatKeepingSchema turning ColumnGroups into DataColumns#1764
koperagen merged 1 commit intomasterfrom
arrow-read-concat-struct-batches

Conversation

@koperagen
Copy link
Copy Markdown
Collaborator

Fixes #1763

arrow cannot yet read framecolumns, so fix only for ColumnGroup. Maybe in the future we should remove code duplication, but let's wait for benchmark result first. For now i'd like to keep this optimized version

@koperagen koperagen self-assigned this Mar 23, 2026
Copy link
Copy Markdown
Collaborator

@Jolanrensen Jolanrensen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, I see, it would probably indeed be better to simply call .concat() and not have to fix the logic in two places. But let's await #1761 indeed. For now it's okay :)

@koperagen koperagen added the bug Something isn't working label Mar 24, 2026
@koperagen koperagen added this to the 1.0.0-Beta5 milestone Mar 24, 2026
@koperagen koperagen merged commit 42b7f90 into master Mar 24, 2026
4 checks passed
@koperagen koperagen changed the title Fix concatKeepingSchema turning ColumnGroups into DataColumns Fixed concatKeepingSchema turning ColumnGroups into DataColumns Mar 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reading Struct field from an Arrow file with multiple batches leads to incomplete schema

2 participants