refactor AccGenericColumn for better data type inference and memory usage statistics.

[richox]

No description provided.

[Copilot]

Pull Request Overview

Refactors the generic accumulator column into specialized column types to improve data type inference and uninitialized memory allocation, and updates aggregation implementations accordingly.

• Introduce UninitializedInit for zero-cost buffer allocation in file readers and batch serde.
• Replace the monolithic AccGenericColumn with AccPrimColumn, AccBooleanColumn, AccBytesColumn, and AccScalarValueColumn, plus helper functions create_acc_generic_column and acc_generic_column_to_array.
• Update all Agg implementations (sum, maxmin, first, count, collect, bloom_filter, avg, spark_udaf_wrapper) to use the new column structs and remove unwrap() on downcasts.

Reviewed Changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
native-engine/datafusion-ext-plans/src/scan/internal_file_reader.rs	Use uninitialized-init buffer allocation in read_fully.
native-engine/datafusion-ext-plans/src/agg/sum.rs	Migrate sum to AccPrimColumn and helper APIs.
native-engine/datafusion-ext-plans/src/agg/spark_udaf_wrapper.rs	Use Vec::uninitialized_init for JVM byte arrays and ? on downcasts.
native-engine/datafusion-ext-plans/src/agg/maxmin.rs	Refactor max/min to specialized Acc*Column types.
native-engine/datafusion-ext-plans/src/agg/first_ignores_null.rs	Swap out AccGenericColumn for concrete column types.
native-engine/datafusion-ext-plans/src/agg/first.rs	Introduce separate flags column and new accumulator helpers.
native-engine/datafusion-ext-plans/src/agg/count.rs	Replace unwrap() on downcasts with error propagation ?.
native-engine/datafusion-ext-plans/src/agg/collect.rs	Add guard for small-to-huge conversion and refactor collection.
native-engine/datafusion-ext-plans/src/agg/bloom_filter.rs	Switch to specialized columns and remove panicking unwraps.
native-engine/datafusion-ext-plans/src/agg/avg.rs	Update AVG to use new column creation and merge patterns.
native-engine/datafusion-ext-plans/src/agg/acc.rs	Overhaul AccGenericColumn into concrete column structs and APIs.
native-engine/datafusion-ext-commons/src/lib.rs	Add UninitializedInit trait for Vec and SmallVec.
native-engine/datafusion-ext-commons/src/io/batch_serde.rs	Apply uninitialized-init to raw batch serialization buffers.
native-engine/datafusion-ext-commons/Cargo.toml	Add smallvec dependency for UninitializedInit support.

Comments suppressed due to low confidence (2)

native-engine/datafusion-ext-plans/src/agg/first.rs:157

• For a Binary data type you must downcast to BinaryArray, not StringArray. Please use downcast_any!(partial_arg, BinaryArray)?.

            DataType::Utf8 => handle_bytes!(downcast_any!(partial_arg, StringArray)?),

native-engine/datafusion-ext-plans/src/agg/collect.rs:514

• The if let with && is invalid syntax. To add a guard, use if let Self::Small(s) = self if s.len() >= 4 { ... } or nest an if.

            && s.len() >= 4

[Copilot]

Binary arrays should be downcast to BinaryArray, not StringArray. Update to downcast_any!(partial_arg, BinaryArray)?.

Suggested change

	DataType::Binary => handle_bytes!(downcast_any!(partial_arg, StringArray)?),
	DataType::Binary => handle_bytes!(downcast_any!(partial_arg, BinaryArray)?),

[Copilot]

[nitpick] The threshold 4 is a magic number. Consider extracting it into a named constant to clarify its purpose.

Suggested change

	&& s.len() >= 4
	&& s.len() >= SMALL_TO_HUGE_THRESHOLD