Skip to content

fix: overflow error when concatenating more than 128 regular arrays#3568

Merged
ianna merged 5 commits intomainfrom
ikrommyd/fix-regulararray-concatenation
Jul 15, 2025
Merged

fix: overflow error when concatenating more than 128 regular arrays#3568
ianna merged 5 commits intomainfrom
ikrommyd/fix-regulararray-concatenation

Conversation

@ikrommyd
Copy link
Collaborator

@ikrommyd ikrommyd commented Jul 5, 2025

Now actually finalize #3310

Instantiate prototype in the same way we do for tags_cls.
Without it, we get

  File "/Users/nmangane/scikit-hep-dev-3/.pixi/envs/default/lib/python3.13/site-packages/awkward/operations/ak_concatenate.py", line 262, in action
    prototype[start : start + size] = tag
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
OverflowError: Python integer 128 out of bounds for int8

This error occurred while calling

    ak.concatenate(
        [<Array [[False], [False], ..., [False]] type='20 * [1 * bool, parame...
        axis = 1
    )

@ikrommyd ikrommyd changed the title concatenate more than 128 regular arrays fix: overflow error when concatenating more than 128 regular arrays Jul 5, 2025
@ikrommyd ikrommyd requested a review from ianna July 5, 2025 22:49
Copy link
Member

@ianna ianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikrommyd - Looks good! Thanks for providing the fix so quickly! This is the right solution since all the kernels involved have specialization for int64 and both UnionArray functions regular_index and simplified allow Index64 as tags. It would help if we add a test where simplified fails to reduce to no more than 128 variants. Please see the comments. Thanks!

# Note: to help merge more than 128 arrays, tags *can* have type ak.index.Index64.
# This is only supported when index is also Index64,
# and all indexed contents are also Index64.
# We still require that this reduces to no more than 128 variants.

Copy link
Member

@ianna ianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikrommyd - Thanks for fixing it!

@ianna ianna enabled auto-merge (squash) July 15, 2025 18:00
@ianna ianna merged commit 0fbc936 into main Jul 15, 2025
43 checks passed
@ianna ianna deleted the ikrommyd/fix-regulararray-concatenation branch July 15, 2025 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants