Core: Coalesce consecutive position deletes into range inserts (Iceberg V2)#16052
Draft
Baunsgaard wants to merge 1 commit intoapache:mainfrom
Draft
Core: Coalesce consecutive position deletes into range inserts (Iceberg V2)#16052Baunsgaard wants to merge 1 commit intoapache:mainfrom
Baunsgaard wants to merge 1 commit intoapache:mainfrom
Conversation
Baunsgaard
force-pushed
the
coalesce-deletes-range-consumer
branch
from
06105f9 to
8130781
Compare
Baunsgaard
force-pushed
the
coalesce-deletes-range-consumer
branch
3 times, most recently
from
fa10273 to
24545db
Compare
Add PositionDeleteRangeConsumer that coalesces runs of consecutive positions into a single delete(start, end) call, and use it from Deletes.toPositionIndex() so sorted position delete files are inserted into the bitmap as ranges instead of one position at a time.
Baunsgaard
force-pushed
the
coalesce-deletes-range-consumer
branch
from
24545db to
8b75033
Compare
Baunsgaard
added a commit
to Baunsgaard/iceberg-cpp
that referenced
this pull request
May 6, 2026
Add ForEachPositionDelete, the C++ equivalent of Java's
PositionDeleteRangeConsumer. When DeleteLoader buffers matching
positions for a data file, it now hands them to ForEachPositionDelete
instead of calling PositionDeleteIndex::Delete(pos) per position.
The implementation sniffs the first 1024 positions to estimate boundary
density (fraction of adjacent pairs where pos[i] != pos[i-1] + 1) and
dispatches to one of two optimized paths:
* Coalesce: walk the input emitting closed-interval runs to
PositionDeleteIndex::Delete(start, end), letting CRoaring's
addRange collapse contiguous runs.
* Bulk addMany: group by high-32-bit key and flush each group via a
private RoaringPositionBitmap::AddManyForKey hook over CRoaring's
bulk addMany. Used when boundary density exceeds 10%.
Below 64 positions the sniff is skipped entirely; coalescing wins for
small inputs because the bulk path's fixed overhead (thread-local
buffer, per-key grouping) exceeds the work to be done.
Rework DeleteLoader::LoadPositionDelete to read Arrow batches via
nanoarrow ArrowArrayView directly instead of the row-oriented
ArrowArrayStructLike wrapper. The loader reuses the ArrowArrayView
across batches and exposes the int64 pos column as a raw buffer. When
the delete file's referenced_data_file matches the target data file
(Iceberg V2 writer hint, the common case), positions are passed to
ForEachPositionDelete as a zero-copy span directly from the Arrow
buffer. Otherwise a per-batch staging vector is used with
ArrowArrayViewGetStringUnsafe + memcmp to filter by path, still much
faster than the Scalar variant dispatch.
Locally measured results (GCC 14.3 Release, 5M positions microbench,
500K positions end-to-end):
* ForEachPositionDelete microbench: 2.2x-10.6x over the per-position
baseline across contiguous/run/sparse/alternating scenarios; no
regressions on tiny inputs.
* End-to-end LoadPositionDeletes (Parquet decode + Arrow iteration +
bitmap inserts): 2.1x-2.5x across all scenarios.
Equivalent of the Java change in apache/iceberg#16052.
Baunsgaard
added a commit
to Baunsgaard/iceberg-cpp
that referenced
this pull request
May 7, 2026
Add ForEachPositionDelete, the C++ equivalent of Java's
PositionDeleteRangeConsumer. When DeleteLoader buffers matching
positions for a data file, it now hands them to ForEachPositionDelete
instead of calling PositionDeleteIndex::Delete(pos) per position.
The implementation sniffs the first 1024 positions to estimate boundary
density (fraction of adjacent pairs where pos[i] != pos[i-1] + 1) and
dispatches to one of two optimized paths:
* Coalesce: walk the input emitting closed-interval runs to
PositionDeleteIndex::Delete(start, end), letting CRoaring's
addRange collapse contiguous runs.
* Bulk addMany: group by high-32-bit key and flush each group via a
private RoaringPositionBitmap::AddManyForKey hook over CRoaring's
bulk addMany. Used when boundary density exceeds 10%.
Below 64 positions the sniff is skipped entirely; coalescing wins for
small inputs because the bulk path's fixed overhead (thread-local
buffer, per-key grouping) exceeds the work to be done.
Rework DeleteLoader::LoadPositionDelete to read Arrow batches via
nanoarrow ArrowArrayView directly instead of the row-oriented
ArrowArrayStructLike wrapper. The loader reuses the ArrowArrayView
across batches and exposes the int64 pos column as a raw buffer. When
the delete file's referenced_data_file matches the target data file
(Iceberg V2 writer hint, the common case), positions are passed to
ForEachPositionDelete as a zero-copy span directly from the Arrow
buffer. Otherwise a per-batch staging vector is used with
ArrowArrayViewGetStringUnsafe + memcmp to filter by path, still much
faster than the Scalar variant dispatch.
Locally measured results (GCC 14.3 Release, 5M positions microbench,
500K positions end-to-end):
* ForEachPositionDelete microbench: 2.2x-10.6x over the per-position
baseline across contiguous/run/sparse/alternating scenarios; no
regressions on tiny inputs.
* End-to-end LoadPositionDeletes (Parquet decode + Arrow iteration +
bitmap inserts): 2.1x-2.5x across all scenarios.
Equivalent of the Java change in apache/iceberg#16052.
Baunsgaard
added a commit
to Baunsgaard/iceberg-cpp
that referenced
this pull request
May 8, 2026
Add ForEachPositionDelete (the C++ equivalent of Java's
PositionDeleteRangeConsumer) and route DeleteLoader through it instead
of calling PositionDeleteIndex::Delete(pos) per position. The function
sniffs a 1024-position prefix to estimate boundary density and
dispatches to one of two paths:
* Coalesce: emit closed-interval runs to
PositionDeleteIndex::Delete(start, end), letting CRoaring's
addRange collapse contiguous runs.
* Bulk addMany: group by high-32-bit key and flush each group via a
private RoaringPositionBitmap::AddManyForKey hook over CRoaring's
addMany. Used when boundary density exceeds 10%. Inputs below 64
positions skip the sniff and use the coalesce path directly.
Also rework DeleteLoader::LoadPositionDelete to read Arrow batches via
nanoarrow's ArrowArrayView directly instead of the row-oriented
ArrowArrayStructLike wrapper. The view is reused across batches and
exposes the int64 pos column as a raw buffer. When the delete file's
referenced_data_file matches the target (Iceberg V2 writer hint, the
common case), positions are passed to ForEachPositionDelete as a
zero-copy span; otherwise a per-batch staging vector filters by path
using nanoarrow's unsafe direct-buffer accessors.
Local microbenchmarks (GCC 14.3 Release): 2.2x-10.6x speedup for
ForEachPositionDelete vs the per-position baseline, and 2.1x-2.5x for
the end-to-end loader (Parquet decode + Arrow iteration + bitmap
inserts).
Equivalent of apache/iceberg#16052.
Baunsgaard
added a commit
to Baunsgaard/iceberg-cpp
that referenced
this pull request
May 8, 2026
Add ForEachPositionDelete (the C++ equivalent of Java's PositionDeleteRangeConsumer) and route DeleteLoader through it, replacing the per-position PositionDeleteIndex::Delete(pos) call. The function sniffs a 1024-position prefix and dispatches to either run coalescing (CRoaring addRange) or bulk addMany grouped by high-32-bit key. Also rework DeleteLoader::LoadPositionDelete to read Arrow batches via nanoarrow's ArrowArrayView directly. When the delete file's referenced_data_file matches the target (V2 writer hint), positions are passed as a zero-copy span; otherwise a per-batch staging vector filters by path. Local microbenchmarks: 2.2x-10.6x for ForEachPositionDelete and 2.1x-2.5x end-to-end through LoadPositionDeletes. Equivalent of apache/iceberg#16052.
Baunsgaard
added a commit
to Baunsgaard/iceberg-cpp
that referenced
this pull request
May 11, 2026
Add ForEachPositionDelete (the C++ equivalent of Java's PositionDeleteRangeConsumer) and route DeleteLoader through it, replacing the per-position PositionDeleteIndex::Delete(pos) call. The function sniffs a 1024-position prefix and dispatches to either run coalescing (CRoaring addRange) or bulk addMany grouped by high-32-bit key. Also rework DeleteLoader::LoadPositionDelete to read Arrow batches via nanoarrow's ArrowArrayView directly. When the delete file's referenced_data_file matches the target (V2 writer hint), positions are passed as a zero-copy span; otherwise a per-batch staging vector filters by path. Local microbenchmarks: 2.2x-10.6x for ForEachPositionDelete and 2.1x-2.5x end-to-end through LoadPositionDeletes. Equivalent of apache/iceberg#16052.
Baunsgaard
added a commit
to Baunsgaard/iceberg-cpp
that referenced
this pull request
May 11, 2026
Add ForEachPositionDelete (the C++ equivalent of Java's PositionDeleteRangeConsumer) and route DeleteLoader through it, replacing the per-position PositionDeleteIndex::Delete(pos) call. The function sniffs a 1024-position prefix and dispatches to either run coalescing (CRoaring addRange) or bulk addMany grouped by high-32-bit key. Also rework DeleteLoader::LoadPositionDelete to read Arrow batches via nanoarrow's ArrowArrayView directly. When the delete file's referenced_data_file matches the target (V2 writer hint), positions are passed as a zero-copy span; otherwise a per-batch staging vector filters by path. Local microbenchmarks: 2.2x-10.6x for ForEachPositionDelete and 2.1x-2.5x end-to-end through LoadPositionDeletes. Equivalent of apache/iceberg#16052.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
PositionDeleteRangeConsumer, a stateless utility that processs consecutive positions into a singlePositionDeleteIndex.delete(start, end)call instead of onedelete(pos)per position.This primarily benefits Iceberg V2 tables; V3 deletion vectors are loaded as a serialised RoaringBitmap and bypass this path.
To avoid overhead on inputs with no runs, the consumer sniffs the first
SNIFF_SIZE = 256positions and counts boundaries (pairs wherepos[i] - pos[i-1] != 1). AboveBOUNDARY_THRESHOLD_PERCENT = 30, the remaining stream use a per-position loop equivalent to the original behavior.