DMStoreForSegmentReadTaskTest.FetchPagesNoTinyNoInMem has data race

[xzhangxian1008]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Build with tsan mode

2. What did you expect to see? (Required)

No error

3. What did you see instead (Required)

WARNING: ThreadSanitizer: data race (pid=4095575)
  Write of size 8 at 0x7b1400081830 by thread T3 (mutexes: write M0):
    #0 operator delete(void*) /DATA/disk3/xzx/llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 (gtests_dbms+0x36d8b4f)
    #1 void std::__1::__libcpp_operator_delete[abi:ue170006]<void*>(void*) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/new:278:3 (gtests_dbms+0x1180a439)
    #2 void std::__1::__do_deallocate_handle_size[abi:ue170006]<>(void*, unsigned long) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/new:302:10 (gtests_dbms+0x1180a439)
    #3 std::__1::__libcpp_deallocate[abi:ue170006](void*, unsigned long, unsigned long) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/new:318:14 (gtests_dbms+0x1180a439)
    #4 std::__1::allocator<std::__1::__shared_ptr_emplace<DB::PS::V3::PageDirectorySnapshot, std::__1::allocator<DB::PS::V3::PageDirectorySnapshot>>>::deallocate[abi:ue170006](std::__1::__shared_ptr_emplace<DB::PS::V3::PageDirectorySnapshot, std::__1::allocator<DB::PS::V3::PageDirectorySnapshot>>*, unsigned long) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/allocator.h:130:13 (gtests_dbms+0x1180a439)
    #5 std::__1::allocator_traits<std::__1::allocator<std::__1::__shared_ptr_emplace<DB::PS::V3::PageDirectorySnapshot, std::__1::allocator<DB::PS::V3::PageDirectorySnapshot>>>>::deallocate[abi:ue170006](std::__1::allocator<std::__1::__shared_ptr_emplace<DB::PS::V3::PageDirectorySnapshot, std::__1::allocator<DB::PS::V3::PageDirectorySnapshot>>>&, std::__1::__shared_ptr_emplace<DB::PS::V3::PageDirectorySnapshot, std::__1::allocator<DB::PS::V3::PageDirectorySnapshot>>*, unsigned long) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/allocator_traits.h:288:13 (gtests_dbms+0x1180a439)
    #6 std::__1::__shared_ptr_emplace<DB::PS::V3::PageDirectorySnapshot, std::__1::allocator<DB::PS::V3::PageDirectorySnapshot>>::__on_zero_shared_weak() /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:332:9 (gtests_dbms+0x1180a439)
    #7 std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>::~weak_ptr() /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:1777:19 (gtests_dbms+0x117f4d9e)
    #8 void std::__1::__destroy_at[abi:ue170006]<std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>, 0>(std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>*) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/construct_at.h:69:13 (gtests_dbms+0x117f4d9e)
    #9 void std::__1::destroy_at[abi:ue170006]<std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>, 0>(std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>*) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/construct_at.h:104:5 (gtests_dbms+0x117f4d9e)
    #10 void std::__1::allocator_traits<std::__1::allocator<std::__1::__list_node<std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>, void*>>>::destroy[abi:ue170006]<std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>, void, void>(std::__1::allocator<std::__1::__list_node<std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>, void*>>&, std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>*) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/allocator_traits.h:323:9 (gtests_dbms+0x117f4d9e)
    #11 std::__1::list<std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>, std::__1::allocator<std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>>>::erase(std::__1::__list_const_iterator<std::__1::weak_ptr<DB::PS::V3::PageDirectorySnapshot>, void*>) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/list:1626:5 (gtests_dbms+0x117f4d9e)
    #12 DB::PS::V3::PageDirectory<DB::PS::V3::universal::PageDirectoryTrait>::gcInMemSnapshots() const /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/PageDirectory.cpp:2155:30 (gtests_dbms+0x117f4d9e)
    #13 DB::PS::V3::PageDirectory<DB::PS::V3::universal::PageDirectoryTrait>::gcInMemEntries(DB::PS::V3::PageDirectory<DB::PS::V3::universal::PageDirectoryTrait>::InMemGCOption const&) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/PageDirectory.cpp:2216:28 (gtests_dbms+0x117fe996)
    #14 DB::PS::V3::ExternalPageCallbacksManager<DB::PS::V3::universal::ExternalPageCallbacksManagerTrait>::doGC(DB::PS::V3::BlobStore<DB::PS::V3::universal::BlobStoreTrait>&, DB::PS::V3::PageDirectory<DB::PS::V3::universal::PageDirectoryTrait>&, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, unsigned long>>>*) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/GCDefines.cpp:255:47 (gtests_dbms+0x118388fe)
    #15 DB::PS::V3::ExternalPageCallbacksManager<DB::PS::V3::universal::ExternalPageCallbacksManagerTrait>::gc(DB::PS::V3::BlobStore<DB::PS::V3::universal::BlobStoreTrait>&, DB::PS::V3::PageDirectory<DB::PS::V3::universal::PageDirectoryTrait>&, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, unsigned long>>>*, std::__1::shared_ptr<DB::Logger>) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/GCDefines.cpp:147:41 (gtests_dbms+0x118385c6)
    #16 DB::UniversalPageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/Universal/UniversalPageStorage.cpp:448:34 (gtests_dbms+0x118aeffb)
    #17 DB::UniversalPageStorageService::gc() const /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:313:36 (gtests_dbms+0x118beebb)
    #18 DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1::operator()() const /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:87:29 (gtests_dbms+0x118bf88a)
    #19 decltype(std::declval<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1&>()()) std::__1::__invoke[abi:ue170006]<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1&>(DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__type_traits/invoke.h:340:25 (gtests_dbms+0x118bf88a)
    #20 bool std::__1::__invoke_void_return_wrapper<bool, false>::__call[abi:ue170006]<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1&>(DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__type_traits/invoke.h:407:12 (gtests_dbms+0x118bf88a)
    #21 std::__1::__function::__alloc_func<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1, std::__1::allocator<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1>, bool ()>::operator()[abi:ue170006]() /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__functional/function.h:192:16 (gtests_dbms+0x118bf88a)
    #22 std::__1::__function::__func<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1, std::__1::allocator<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_1>, bool ()>::operator()() /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__functional/function.h:363:12 (gtests_dbms+0x118bf88a)
    #23 std::__1::__function::__value_func<bool ()>::operator()[abi:ue170006]() const /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__functional/function.h:517:16 (gtests_dbms+0x10c3d146)
    #24 std::__1::function<bool ()>::operator()() const /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__functional/function.h:1168:12 (gtests_dbms+0x10c3d146)
    #25 DB::BackgroundProcessingPool::threadFunction(unsigned long) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/BackgroundProcessingPool.cpp:267:25 (gtests_dbms+0x10c3d146)
    #26 DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1::operator()() const /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/BackgroundProcessingPool.cpp:103:46 (gtests_dbms+0x10c3df98)
    #27 decltype(std::declval<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1>()()) std::__1::__invoke[abi:ue170006]<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1&&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__type_traits/invoke.h:340:25 (gtests_dbms+0x10c3df98)
    #28 void std::__1::__thread_execute[abi:ue170006]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1>&, std::__1::__tuple_indices<>) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__thread/thread.h:221:5 (gtests_dbms+0x10c3df98)
    #29 void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1>>(void*) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__thread/thread.h:232:5 (gtests_dbms+0x10c3df98)

  Previous read of size 8 at 0x7b1400081830 by main thread:
    #0 std::__1::__shared_count::__release_shared[abi:ue170006]() /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:172:9 (gtests_dbms+0xe23d2de)
    #1 std::__1::__shared_weak_count::__release_shared[abi:ue170006]() /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:213:27 (gtests_dbms+0xe23d2de)
    #2 std::__1::shared_ptr<DB::PageStorageSnapshot>::~shared_ptr[abi:ue170006]() /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:772:23 (gtests_dbms+0xe23d2de)
    #3 DB::DM::Remote::RNLocalPageCache::RNLocalPageCache(DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/DeltaMerge/Remote/RNLocalPageCache.cpp:47:9 (gtests_dbms+0xe23d2de)
    #4 DB::DM::Remote::RNLocalPageCache* std::__1::construct_at[abi:ue170006]<DB::DM::Remote::RNLocalPageCache, DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&, DB::DM::Remote::RNLocalPageCache*>(DB::DM::Remote::RNLocalPageCache*, DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/construct_at.h:41:46 (gtests_dbms+0x10980d98)
    #5 void std::__1::allocator_traits<std::__1::allocator<DB::DM::Remote::RNLocalPageCache>>::construct[abi:ue170006]<DB::DM::Remote::RNLocalPageCache, DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&, void, void>(std::__1::allocator<DB::DM::Remote::RNLocalPageCache>&, DB::DM::Remote::RNLocalPageCache*, DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/allocator_traits.h:304:9 (gtests_dbms+0x10980d98)
    #6 std::__1::__shared_ptr_emplace<DB::DM::Remote::RNLocalPageCache, std::__1::allocator<DB::DM::Remote::RNLocalPageCache>>::__shared_ptr_emplace[abi:ue170006]<DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&>(std::__1::allocator<DB::DM::Remote::RNLocalPageCache>, DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:299:13 (gtests_dbms+0x10980d98)
    #7 std::__1::shared_ptr<DB::DM::Remote::RNLocalPageCache> std::__1::allocate_shared[abi:ue170006]<DB::DM::Remote::RNLocalPageCache, std::__1::allocator<DB::DM::Remote::RNLocalPageCache>, DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&, void>(std::__1::allocator<DB::DM::Remote::RNLocalPageCache> const&, DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:1022:55 (gtests_dbms+0x10980d98)
    #8 std::__1::shared_ptr<DB::DM::Remote::RNLocalPageCache> std::__1::make_shared[abi:ue170006]<DB::DM::Remote::RNLocalPageCache, DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&, void>(DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:1031:12 (gtests_dbms+0x1097f6e5)
    #9 DB::DM::Remote::RNLocalPageCache::create(DB::DM::Remote::RNLocalPageCache::RNLocalPageCacheOptions const&) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/DeltaMerge/Remote/RNLocalPageCache.h:158:16 (gtests_dbms+0x1097f6e5)
    #10 DB::SharedContextDisagg::initReadNodePageCache(DB::PathPool const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long) /DATA/disk3/xzx/tf-tmp/dbms/src/Interpreters/SharedContexts/Disagg.cpp:80:25 (gtests_dbms+0x1097f6e5)
    #11 DB::DM::tests::DMStoreForSegmentReadTaskTest::initReadNodePageCacheIfUninitialized() /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/DeltaMerge/tests/gtest_segment_read_task.cpp:220:51 (gtests_dbms+0x6ec4c6a)
    #12 DB::DM::tests::DMStoreForSegmentReadTaskTest::SetUp() /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/DeltaMerge/tests/gtest_segment_read_task.cpp:154:9 (gtests_dbms+0x6eb9571)
    #13 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xe5645e9)
    #14 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xe5645e9)
    #15 testing::Test::Run() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2513:3 (gtests_dbms+0xe539c7e)
    #16 testing::TestInfo::Run() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xe53b59e)
    #17 testing::TestCase::Run() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xe53c583)
    #18 testing::internal::UnitTestImpl::RunAllTests() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xe54e99a)
    #19 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xe565afd)
    #20 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xe565afd)
    #21 testing::UnitTest::Run() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xe54e264)
    #22 RUN_ALL_TESTS() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x77c32c7)
    #23 main /DATA/disk3/xzx/tf-tmp/dbms/src/TestUtils/gtests_dbms_main.cpp:120:16 (gtests_dbms+0x77c32c7)

  Mutex M0 (0x7b4c00006f50) created at:
    #0 pthread_mutex_lock /DATA/disk3/xzx/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1339:3 (gtests_dbms+0x362a87b)
    #1 std::__1::mutex::lock() <null> (libc++.so.1+0x52e35)
    #2 DB::PS::V3::PageDirectory<DB::PS::V3::universal::PageDirectoryTrait>::gcInMemEntries(DB::PS::V3::PageDirectory<DB::PS::V3::universal::PageDirectoryTrait>::InMemGCOption const&) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/PageDirectory.cpp:2216:28 (gtests_dbms+0x117fe996)
    #3 DB::PS::V3::PageDirectoryFactory<DB::PS::V3::universal::FactoryTrait>::createFromReader(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PS::V3::WALStoreReader>, std::__1::unique_ptr<DB::PS::V3::WALStore, std::__1::default_delete<DB::PS::V3::WALStore>>) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp:63:10 (gtests_dbms+0x1181d3f1)
    #4 DB::PS::V3::PageDirectoryFactory<DB::PS::V3::universal::FactoryTrait>::create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::FileProvider>&, std::__1::shared_ptr<DB::PSDiskDelegator>&, DB::PS::V3::WALConfig const&) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp:45:12 (gtests_dbms+0x1181d1e7)
    #5 DB::UniversalPageStorage::restore() /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/Universal/UniversalPageStorage.cpp:91:27 (gtests_dbms+0x118a9cd2)
    #6 DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:57:32 (gtests_dbms+0x118bb959)
    #7 DB::SharedContextDisagg::initReadNodePageCache(DB::PathPool const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long) /DATA/disk3/xzx/tf-tmp/dbms/src/Interpreters/SharedContexts/Disagg.cpp:75:33 (gtests_dbms+0x1097f57c)
    #8 DB::DM::tests::DMStoreForSegmentReadTaskTest::initReadNodePageCacheIfUninitialized() /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/DeltaMerge/tests/gtest_segment_read_task.cpp:220:51 (gtests_dbms+0x6ec4c6a)
    #9 DB::DM::tests::DMStoreForSegmentReadTaskTest::SetUp() /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/DeltaMerge/tests/gtest_segment_read_task.cpp:154:9 (gtests_dbms+0x6eb9571)
    #10 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xe5645e9)
    #11 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xe5645e9)
    #12 testing::Test::Run() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2513:3 (gtests_dbms+0xe539c7e)
    #13 testing::TestInfo::Run() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xe53b59e)
    #14 testing::TestCase::Run() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xe53c583)
    #15 testing::internal::UnitTestImpl::RunAllTests() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xe54e99a)
    #16 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xe565afd)
    #17 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xe565afd)
    #18 testing::UnitTest::Run() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xe54e264)
    #19 RUN_ALL_TESTS() /DATA/disk3/xzx/tf-tmp/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x77c32c7)
    #20 main /DATA/disk3/xzx/tf-tmp/dbms/src/TestUtils/gtests_dbms_main.cpp:120:16 (gtests_dbms+0x77c32c7)

  Thread T3 'bg-0' (tid=4095579, running) created by main thread at:
    #0 pthread_create /DATA/disk3/xzx/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1020:3 (gtests_dbms+0x3628b7b)
    #1 std::__1::__libcpp_thread_create[abi:ue170006](unsigned long*, void* (*)(void*), void*) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__threading_support:371:10 (gtests_dbms+0x10c3c0bb)
    #2 std::__1::thread::thread<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1, void>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1&&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__thread/thread.h:248:16 (gtests_dbms+0x10c3c0bb)
    #3 DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>) /DATA/disk3/xzx/tf-tmp/dbms/src/Storages/BackgroundProcessingPool.cpp:103:22 (gtests_dbms+0x10c3c0bb)
    #4 DB::BackgroundProcessingPool* std::__1::construct_at[abi:ue170006]<DB::BackgroundProcessingPool, unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>, DB::BackgroundProcessingPool*>(DB::BackgroundProcessingPool*, unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>&&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/construct_at.h:41:46 (gtests_dbms+0x100d70f2)
    #5 void std::__1::allocator_traits<std::__1::allocator<DB::BackgroundProcessingPool>>::construct[abi:ue170006]<DB::BackgroundProcessingPool, unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>, void, void>(std::__1::allocator<DB::BackgroundProcessingPool>&, DB::BackgroundProcessingPool*, unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>&&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/allocator_traits.h:304:9 (gtests_dbms+0x100b92a9)
    #6 std::__1::__shared_ptr_emplace<DB::BackgroundProcessingPool, std::__1::allocator<DB::BackgroundProcessingPool>>::__shared_ptr_emplace[abi:ue170006]<unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>>(std::__1::allocator<DB::BackgroundProcessingPool>, unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>&&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:299:13 (gtests_dbms+0x100b92a9)
    #7 std::__1::shared_ptr<DB::BackgroundProcessingPool> std::__1::allocate_shared[abi:ue170006]<DB::BackgroundProcessingPool, std::__1::allocator<DB::BackgroundProcessingPool>, unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>, void>(std::__1::allocator<DB::BackgroundProcessingPool> const&, unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>&&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:1022:55 (gtests_dbms+0x100b92a9)
    #8 std::__1::shared_ptr<DB::BackgroundProcessingPool> std::__1::make_shared[abi:ue170006]<DB::BackgroundProcessingPool, unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>, void>(unsigned short&, char const (&) [4], std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>&&) /DATA/disk3/xzx/llvm-17.0.6/bin/../include/c++/v1/__memory/shared_ptr.h:1031:12 (gtests_dbms+0x100b92a9)
    #9 DB::Context::initializeBackgroundPool(unsigned short) /DATA/disk3/xzx/tf-tmp/dbms/src/Interpreters/Context.cpp:1336:15 (gtests_dbms+0x100b92a9)
    #10 DB::tests::TiFlashTestEnv::addGlobalContext(DB::Settings const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>, DB::PageStorageRunMode, unsigned long) /DATA/disk3/xzx/tf-tmp/dbms/src/Debug/TiFlashTestEnv.cpp:126:21 (gtests_dbms+0xfad7321)
    #11 DB::tests::TiFlashTestEnv::initializeGlobalContext(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>, DB::PageStorageRunMode, unsigned long) /DATA/disk3/xzx/tf-tmp/dbms/src/Debug/TiFlashTestEnv.cpp:103:5 (gtests_dbms+0xfad6eb4)
    #12 main /DATA/disk3/xzx/tf-tmp/dbms/src/TestUtils/gtests_dbms_main.cpp:73:5 (gtests_dbms+0x77c22af)

SUMMARY: ThreadSanitizer: data race /DATA/disk3/xzx/llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 in operator delete(void*)

4. What is your TiFlash version? (Required)

[JaySon-Huang]

Only affects the testing code, mark as moderate

[JaySon-Huang]

@coderabbitai can you try to fix this issue?

[coderabbitai]

Warning

A plan generation is already in progress. Please wait for it to complete before requesting a new plan.

[coderabbitai]

Coding Plan

Approach

Fix a ThreadSanitizer-detected data race between the background GC task's gcInMemSnapshots() and snapshot destruction in RNLocalPageCache constructor. The race occurs because the background GC task can execute immediately after registration in UniversalPageStorageService::create(), while the main thread is still creating snapshots during RNLocalPageCache initialization.

Observations

The codebase uses a PageDirectory with weak_ptr-based snapshot tracking protected by snapshots_mutex. UniversalPageStorageService creates a background GC task during create() that runs gcInMemSnapshots() to clean up expired snapshot references. RNLocalPageCache constructor calls traverseEntries() which creates temporary snapshots. The race occurs when the GC task's weak_ptr destruction (holding snapshots_mutex) overlaps with the main thread's shared_ptr destruction (not holding the mutex), both operating on the same shared_ptr control block.

Assumptions

Assumption 1: Should the fix be in the service layer or the test layer?

Options Considered:

1. Fix in UniversalPageStorageService by deferring GC enablement until after full initialization
2. Fix only in test code by using createForTest() or adding explicit synchronization
3. Fix in gcInMemSnapshots() by changing the snapshot cleanup pattern

Chosen Option: 1

Rationale: Fixing in UniversalPageStorageService is more robust as it prevents the race at the source and protects against similar issues in other callers.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Add GC Initialization Control to UniversalPageStorageService

This phase adds a mechanism to defer background GC execution until after the service is fully initialized and returned to the caller, preventing the race between GC cleanup and concurrent snapshot operations.

Task 1: Add Atomic GC Enable Flag

Add an atomic boolean flag to UniversalPageStorageService that controls whether the GC task should execute its work or return early.

• Add std::atomic<bool> gc_enabled{false} member to dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.h
• Initialize the flag to false in the constructor
• The flag indicates whether the service has completed initialization and is ready for GC operations

Task 2: Check Flag in GC Method

Modify the GC method to check the enable flag before performing GC work.

• Update gc() method in dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp to early-return false if gc_enabled is false
• Place the check before calling uni_page_storage->gc()
• Return false to indicate no work was done, allowing the background pool to reschedule

Task 3: Enable GC After Service Creation

Set the enable flag to true after the service is fully constructed and ready to be returned.

• In UniversalPageStorageService::create() in dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp, set service->gc_enabled = true after all initialization is complete but before the return statement
• This ensures the background GC task will skip execution until after create() returns the fully-initialized service

🤖 Prompt for AI agents

Add a mechanism to defer background GC execution until after
UniversalPageStorageService is fully initialized, preventing the race between GC
cleanup and concurrent snapshot operations.

**Add Atomic GC Enable Flag:**
- In `dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.h`, add a
member variable `std::atomic<bool> gc_enabled{false}` to the
UniversalPageStorageService class
- Initialize this flag to `false` in the constructor to indicate the service is
not yet ready for GC operations
- This flag will control whether the GC task should execute its cleanup work or
return early

**Check Flag in GC Method:**
- In `dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp`,
modify the `gc()` method to check the `gc_enabled` flag at the beginning
- If `gc_enabled` is `false`, return `false` immediately before calling
`uni_page_storage->gc()`
- Returning `false` indicates no work was done and allows the background pool to
reschedule the task

**Enable GC After Service Creation:**
- In the `UniversalPageStorageService::create()` method in
`dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp`, set
`service->gc_enabled = true` after all initialization is complete but before the
return statement
- This ensures the background GC task cannot execute until after `create()`
returns the fully-initialized service to the caller

Phase 2: Update Initialization Call Site

This phase ensures the GC flag is properly enabled at the appropriate point in the initialization sequence.

Task 1: Enable GC After RNLocalPageCache Construction

In the disaggregated storage initialization, ensure GC is only enabled after both the service and cache are fully initialized.

• Review dbms/src/Interpreters/SharedContexts/Disagg.cpp initReadNodePageCache() method
• The current sequence creates service (line 75), then cache (line 80)
• With the flag approach, GC is naturally deferred since gc_enabled is set at end of create() before return
• Verify the RNLocalPageCache constructor completes before any GC can run (the flag set in create() happens before initReadNodePageCache continues to cache creation)

Task 2: Verify Test Behavior

Ensure the fix resolves the TSan-detected race in the test.

• The test DMStoreForSegmentReadTaskTest::SetUp() in dbms/src/Storages/DeltaMerge/tests/gtest_segment_read_task.cpp calls initReadNodePageCacheIfUninitialized()
• With the fix, the GC task will skip execution until after create() returns
• The race window between GC and RNLocalPageCache constructor is eliminated because GC cannot run during the constructor

🤖 Prompt for AI agents

Verify that the GC flag is properly enabled at the appropriate point in the
initialization sequence and that the fix resolves the ThreadSanitizer-detected
race.

**Enable GC After RNLocalPageCache Construction:**
- Review the `initReadNodePageCache()` method in
`dbms/src/Interpreters/SharedContexts/Disagg.cpp`
- Verify the initialization sequence: service creation (line 75), then cache
creation (line 80)
- With the atomic flag approach from Phase 1, GC is automatically deferred since
`gc_enabled` is set at the end of `create()` before returning
- Confirm that the RNLocalPageCache constructor completes before any GC can run,
as the flag is set in `create()` before `initReadNodePageCache` continues to
cache creation
- No code changes should be needed here; this is a verification step

**Verify Test Behavior:**
- Review the test `DMStoreForSegmentReadTaskTest::SetUp()` in
`dbms/src/Storages/DeltaMerge/tests/gtest_segment_read_task.cpp` which calls
`initReadNodePageCacheIfUninitialized()`
- Verify that with the fix, the GC task will skip execution until after
`create()` returns
- Confirm that the race window between GC and RNLocalPageCache constructor is
eliminated because GC cannot run during the constructor
- Run the test under ThreadSanitizer to verify the data race is resolved

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Add a mechanism to defer background GC execution until after
UniversalPageStorageService is fully initialized, preventing the race between GC
cleanup and concurrent snapshot operations.

**Add Atomic GC Enable Flag:**
- In `dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.h`, add a
member variable `std::atomic<bool> gc_enabled{false}` to the
UniversalPageStorageService class
- Initialize this flag to `false` in the constructor to indicate the service is
not yet ready for GC operations
- This flag will control whether the GC task should execute its cleanup work or
return early

**Check Flag in GC Method:**
- In `dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp`,
modify the `gc()` method to check the `gc_enabled` flag at the beginning
- If `gc_enabled` is `false`, return `false` immediately before calling
`uni_page_storage->gc()`
- Returning `false` indicates no work was done and allows the background pool to
reschedule the task

**Enable GC After Service Creation:**
- In the `UniversalPageStorageService::create()` method in
`dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp`, set
`service->gc_enabled = true` after all initialization is complete but before the
return statement
- This ensures the background GC task cannot execute until after `create()`
returns the fully-initialized service to the caller
===============================================================================

Task: 2

Verify that the GC flag is properly enabled at the appropriate point in the
initialization sequence and that the fix resolves the ThreadSanitizer-detected
race.

**Enable GC After RNLocalPageCache Construction:**
- Review the `initReadNodePageCache()` method in
`dbms/src/Interpreters/SharedContexts/Disagg.cpp`
- Verify the initialization sequence: service creation (line 75), then cache
creation (line 80)
- With the atomic flag approach from Phase 1, GC is automatically deferred since
`gc_enabled` is set at the end of `create()` before returning
- Confirm that the RNLocalPageCache constructor completes before any GC can run,
as the flag is set in `create()` before `initReadNodePageCache` continues to
cache creation
- No code changes should be needed here; this is a verification step

**Verify Test Behavior:**
- Review the test `DMStoreForSegmentReadTaskTest::SetUp()` in
`dbms/src/Storages/DeltaMerge/tests/gtest_segment_read_task.cpp` which calls
`initReadNodePageCacheIfUninitialized()`
- Verify that with the fix, the GC task will skip execution until after
`create()` returns
- Confirm that the race window between GC and RNLocalPageCache constructor is
eliminated because GC cannot run during the constructor
- Run the test under ThreadSanitizer to verify the data race is resolved

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!