support the counting filtered rows when filter is pushed down to scan node

[alpass163]

support the counting filtered rows when filter is pushed down to scan node

[Copilot]

Pull request overview

This PR extends IoTDB’s query statistics to count filtered rows when predicates are pushed down into scan nodes, and threads a new verbose flag through planning/execution so row-level filtered counts can be collected and displayed during EXPLAIN ANALYZE VERBOSE.

Changes:

• Add new filtered-row counters (time series / chunk / page / row-scan) to QueryStatistics and expose them via Thrift (TQueryStatistics).
• Propagate verbose from SQL (EXPLAIN ANALYZE VERBOSE) through MPPQueryContext → FragmentInstance → FragmentInstanceContext/QueryContext, and instrument scan/reader paths to record filtered counts.
• Add an integration test validating filtered-row metrics in table model.

Reviewed changes

Copilot reviewed 57 out of 57 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
iotdb-protocol/thrift-datanode/src/main/thrift/datanode.thrift	Add filtered-row fields to TQueryStatistics.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/wal/recover/file/TsFilePlanRedoerTest.java	Update QueryContext construction signature in tests.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/MemChunkLoaderTest.java	Pass QueryContext into mem iterators/loader tests.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/MemAlignedChunkLoaderTest.java	Update aligned mem iterator creation (currently passes null context).
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/PrimitiveMemTableTest.java	Update QueryContext constructor usage in tests.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/MemChunkDeserializeTest.java	Update QueryContext constructor usage in tests.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/relational/planner/CteSubqueryTest.java	Adjust mock signature to include isVerbose.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/relational/planner/CteMaterializerTest.java	Adjust mock signature to include isVerbose.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/planner/FragmentInstanceSerdeTest.java	Update FragmentInstance ctor calls for new boolean.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/TVList.java	Thread QueryContext through iterators; record row-level filtered rows.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/OrderedMultiTVListIterator.java	Pass QueryContext into iterator base.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/OrderedMultiAlignedTVListIterator.java	Pass QueryContext into iterator base.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MultiTVListIterator.java	Record pushdown-filtered rows; propagate QueryContext.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MultiAlignedTVListIterator.java	Record filtered rows via consumer; propagate QueryContext.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortMultiTVListIterator.java	Pass QueryContext into iterator base.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortMultiAlignedTVListIterator.java	Pass QueryContext into iterator base.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MemPointIteratorFactory.java	Add QueryContext parameter to iterator factory methods.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MemPointIterator.java	Store/propagate QueryContext in iterator base.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java	Track filtered rows and route filter recording via verbose flag.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/TsFileResource.java	Clarify inability to count filtered rows at tsfile level.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/metadata/MemChunkMetadataLoader.java	Use shared skip+record helper when filtering chunk metadata.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/metadata/MemAlignedChunkMetadataLoader.java	Use shared skip+record helper when filtering aligned chunk metadata.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/metadata/DiskChunkMetadataLoader.java	Use shared skip+record helper when filtering disk chunk metadata.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/metadata/DiskAlignedChunkMetadataLoader.java	Use shared skip+record helper when filtering aligned disk chunk metadata.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/metadata/ChunkMetadataLoaderUtils.java	New helper to skip chunk metadata and record filtered-row stats.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/MemPageReader.java	Add filter-rows recorder support to getAllSatisfiedData.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/chunk/MemAlignedPageReader.java	Add filter-rows recorder support to getAllSatisfiedData.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/WritableMemChunk.java	Create mem iterators with new factory signature (currently passes null context).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/ReadOnlyMemChunk.java	Pass QueryContext into iterator factory.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java	Create aligned mem iterators with new factory signature (currently passes null context).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedReadOnlyMemChunk.java	Pass QueryContext into iterator factory.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/statistics/FragmentInstanceStatisticsDrawer.java	Render filtered-row metrics; gate row-scan metrics behind verbose.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/load/LoadTsFileScheduler.java	Pass isVerbose into FragmentInstance.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/ir/PredicateWithUncorrelatedScalarSubqueryReconstructor.java	Pass isVerbose into coordinator execution.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableModelQueryFragmentPlanner.java	Pass isVerbose into FragmentInstance.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/CteMaterializer.java	Pass isVerbose into coordinator execution.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/FragmentInstance.java	Add/serialize verbose flag in fragment instances.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/distribution/WriteFragmentParallelPlanner.java	Update FragmentInstance ctor usage.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/distribution/SimpleFragmentParallelPlanner.java	Pass isVerbose into FragmentInstance.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/TableOperatorGenerator.java	Record downstream plan node ids in operator specified info.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/OperatorTreeGenerator.java	Record downstream plan node ids for sink operators.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java	Populate verbose flag from EXPLAIN ANALYZE VERBOSE.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java	Extend execute signature with isVerbose; propagate into query context.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/LastQueryAggTableScanOperator.java	Whitespace-only change.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/AbstractTableScanOperator.java	Whitespace-only change.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/SeriesScanUtil.java	Record filtered rows at time series/chunk/page and row levels (verbose).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/FileLoaderUtils.java	Record filtered rows when time series metadata is skipped.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/ExchangeOperator.java	Record received TsBlock bytes in operator specified info.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/sink/IdentitySinkOperator.java	Add constants for specified-info keys.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/OperatorContext.java	Make specified-info map persist and be concurrent.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/QueryStatistics.java	Add filtered-row counters and serialize via Thrift.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/QueryContext.java	Add verbose flag and new ctor signature.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java	Pass verbose into fragment instance context creation.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceContext.java	Carry verbose through fragment instance context constructors.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FakedFragmentInstanceContext.java	Update ctor signature for verbose.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java	Rename/store verbose flag consistently.
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/IoTDBTableFilteredRowsIT.java	New IT validating filtered-row metrics for pushed-down filters.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.