Add NullBuffer::try_from_unsliced helper and refactor call sites#9411
Add NullBuffer::try_from_unsliced helper and refactor call sites#9411Eyad3skr wants to merge 2 commits intoapache:mainfrom
NullBuffer::try_from_unsliced helper and refactor call sites#9411Conversation
alamb
left a comment
There was a problem hiding this comment.
Thank you @Eyad3skr -- this looks nice to me
cc @liamzwbao in case you have time to help reivew
arrow-buffer/src/buffer/null.rs
Outdated
There was a problem hiding this comment.
Could you also update this comment to reflect the offset is in bits (not bytes)? Mixing the units is a common mistake so making sure the documentation is as clear as possible would help
arrow-buffer/src/buffer/null.rs
Outdated
There was a problem hiding this comment.
Maybe we should call it from_buffer? try_* probably should be reserved for functions that return a result, and its probably better to be direct that we're constructing directly from a buffer?
|
@alamb I guess there is nothing else to take care after anymore? maybe if someone can just trigger/approve the CI pipeline workflows left that would be awesome. |
Done! |
|
Seems like the CI is broken |
|
true, my schedule this week is a bit busy because of university. I will be working on fixing the CI between today and Friday. On it. |
liamzwbao
left a comment
There was a problem hiding this comment.
Overall LGTM! thanks for the change.
It would be better to refactor the following call sites as well
arrow-rs/arrow-array/src/array/boolean_array.rs
Lines 537 to 542 in c129c7c
arrow-rs/arrow-string/src/regexp.rs
Lines 203 to 207 in c129c7c
arrow-rs/arrow-string/src/substring.rs
Lines 216 to 220 in c129c7c
arrow-rs/arrow-string/src/substring.rs
Lines 318 to 322 in c129c7c
arrow-rs/arrow-string/src/substring.rs
Lines 356 to 360 in c129c7c
| #[test] | ||
| fn test_from_unsliced_buffer_with_nulls() { | ||
| // Buffer with some nulls: 0b10110010 = valid, null, valid, valid, null, null, valid, null | ||
| let buf = Buffer::from([0b10110010u8]); |
There was a problem hiding this comment.
Arrow uses LSB numbering and that's why this test failed. Could refer to the doc to fix the test
Implements a helper to replace the pattern of creating a
BooleanBufferfrom an unsliced validity bitmap and filtering by null count. Previously this was done withBooleanBuffer::new(...)plusSome(NullBuffer::new(...)).filter(|n| n.null_count() > 0);now it is a single call toNullBuffer::try_from_unsliced(buffer, len), which returnsSome(NullBuffer)when there are nulls andNonewhen all values are valid.try_from_unslicedinarrow-buffer/src/buffer/null.rswith tests for nulls, all valid, all null, emptyFixedSizeBinaryArray::try_from_iter_with_sizeandtry_from_sparse_iter_with_sizeto use ittake_nullsinarrow-selectto use itCloses #9385