Lockup in library when client shuts down

[andysCaplin]

C++ compiler version:
Hazelcast Cpp client version: 4.0.1
Hazelcast server version: N/A
Number of the clients:
Cluster size, i.e. the number of Hazelcast cluster members:
OS version (Windows/Linux/OSX): Linux

Please attach relevant logs and files for client and server side.

Expected behaviour

When a SIGTERM is handled the binary should shut down

Actual behaviour

When the SIGTERM is received the Hazelcast library gets stuck sometimes which prevents the binary shutdown
We intercept SIGTERM and write to a pipe to call our specific signal handling code - i.e. none of our signal handling code, including the call to Hazelcast shutdown is called in the signal handler itself.

You can see that in thread 41 of the attached file. I don't know why frame 3 isn't shown but it's a call to a routine in our code that calls Hazelcast shutdown. When that returns we exit.
I can see from the logs that we've called the Hazelcast shutdown

2021/07/09-22:49:19.542328 +0000: (rttpd): NOTIFY: Discovery: Shutting down client
2021/07/09-22:49:19.542349 +0000: (rttpd): NOTIFY: Discovery: Peer - Removing entry <ccbbfb16-e5be-4c1a-af5b-840492060004>
2021/07/09-22:49:19.542465 +0000: (4276): INFO: Discovery: Library - <hz.client_1><soaktest> LifecycleService::LifecycleEvent SHUTTING_DOWN
2021/07/09-22:49:19.542631 +0000: (4276): INFO: Discovery: Library - <hz.client_1><soaktest> Removed connection to endpoint: Address[10.42.3.224:5701], connection: ClientConnection{alive=0, connectionId=3, remoteEndpoint=Address[10.42.3.224:5701], lastReadTime=2021-07-09 22:49:19.0-5, closedTime=2021-07-09 22:49:19.000, connected server version=4.0.3}
2021/07/09-22:49:19.542676 +0000: (4276): INFO: Discovery: Library - <hz.client_1><soaktest> Removed connection to endpoint: Address[10.42.3.226:5701], connection: ClientConnection{alive=0, connectionId=1, remoteEndpoint=Address[10.42.3.226:5701], lastReadTime=2021-07-09 22:49:19.0-5, closedTime=2021-07-09 22:49:19.000, connected server version=4.0.3}
2021/07/09-22:49:19.542755 +0000: (4276): INFO: Discovery: Library - <hz.client_1><soaktest> Removed connection to endpoint: Address[10.42.3.230:5701], connection: ClientConnection{alive=0, connectionId=2, remoteEndpoint=Address[10.42.3.230:5701], lastReadTime=2021-07-09 22:49:19.-35, closedTime=2021-07-09 22:49:19.000, connected server version=4.0.3}
2021/07/09-22:49:19.542763 +0000: (4276): INFO: Discovery: Library - <hz.client_1><soaktest> LifecycleService::LifecycleEvent CLIENT_DISCONNECTED

In the attached trace file

• There is a Hazelcast thread - 40 - that seems to doing shut down stuff
• Thread 22 is handling a queue item
• All the other threads don't appear to be locked up

The reason we shut down is because we timed out on receiving an expected response on a queue - which may be related to thread 22, i.e.

• Thread 22 got stuck ( don't know why )
• We time out and try to shutdown
• We call hazelcast->shutdown() and it can't shut down because thread 22 still has some resource ?

The queue that has timed out has previously received several responses.
We have had issues with queues randomly not receiving messages sent by the server but in those cases when I looked at the client threads none of them were stuck as shown in the trace shown below.

Steps to reproduce the behaviour

This doesn't happen every time

gdb.txt

[andysCaplin]

Happened again. Listed the 2 Hazelcast threads doing shutdown stuff.
Full stack trace attached at end of comment.

Thread 36 (Thread 0x7f16013e8700 (LWP 5566)):
#0  0x00007f163af9e2fd in pthread_join () from /lib64/libpthread.so.0
#1  0x0000000000ad4efa in boost::asio::detail::thread_group::join() [clone .isra.108] ()
#2  0x0000000000ad692a in hazelcast::util::hz_thread_pool::close() ()
#3  0x0000000000a0b328 in hazelcast::client::connection::ClientConnectionManagerImpl::shutdown() ()
#4  0x0000000000aae7d9 in hazelcast::client::spi::lifecycle_service::shutdown() ()
#5  0x000000000094ca27 in boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >::run(boost::shared_ptr<boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}&&) ()
#6  0x00000000009484ea in boost::detail::thread_data<void (*)(boost::shared_ptr<boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> > >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}&&), boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}>::run() ()
#7  0x0000000000ac87c7 in thread_proxy ()
#8  0x00007f163af9daa1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f163a54ac4d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f163b7ce760 (LWP 5228)):
#0  0x00007f163afa168c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000008f959c in boost::posix::pthread_cond_wait (m=<optimized out>, c=<optimized out>) at /opt/boost/include/boost/thread/pthread/pthread_helpers.hpp:112
#2  boost::condition_variable::wait (m=..., this=0x418fb38) at /opt/boost/include/boost/thread/pthread/condition_variable.hpp:79
#3  boost::condition_variable::wait<boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, boost::detail::shared_state_base>, boost::_bi::list1<boost::reference_wrapper<boost::detail::shared_state_base> > > > (pred=..., m=..., this=0x418fb38) at /opt/boost/include/boost/thread/pthread/condition_variable_fwd.hpp:93
#4  boost::detail::shared_state_base::wait_internal (this=0x418fae0, lk=..., rethrow=rethrow@entry=true) at /opt/boost/include/boost/thread/future.hpp:396
#5  0x00000000008fb184 in boost::detail::shared_state<void>::get (lock=..., this=<optimized out>) at /opt/boost/include/boost/thread/future.hpp:868
#6  boost::future<void>::get (this=this@entry=0x7ffd0e705560) at /opt/boost/include/boost/thread/future.hpp:1805
#7  0x00000000008d6ec8 in _ds_cpp_shutdown () at ../src/discovery.cpp:1214
#8  0x0000000000874fa4 in rttpd_exit (exit_code=0, sig=15) at ../src/main.c:161
#9  0x00000000008a0480 in handle_event (ev=0x41ab100, ptr=0x41ab100 "\350\003", thr=<optimized out>, thr=<optimized out>) at ../src/threads.c:1653
#10 0x00000000008a179d in threads_flush_write_queue_direct (thr=thr@entry=0x1241b00 <main_thread>) at ../src/threads.c:2256
#11 0x00000000008a1848 in threads_read_event (fd=<optimized out>, val=<optimized out>, data=0x1241b00 <main_thread>) at ../src/threads.c:1898
#12 0x000000000091ea52 in loop_events (_ds_event_globals=_ds_event_globals@entry=0x3f2b1b0, head=0x3f2b2b0, memroot=0x3fd3e70, eventmask_check=eventmask_check@entry=57, eventmask_set=eventmask_set@entry=1) at ../src/event_poll.c:438
#13 0x000000000091ec4a in event_poll_loop () at ../src/event_poll.c:547
#14 0x0000000000875b02 in main_function (argc=<optimized out>, argv=<optimized out>) at ../src/main.c:621
#15 0x00000000007de189 in main (argc=<optimized out>, argv=<optimized out>) at ../src/main.c:637

gstack.txt

[ihsandemir]

Hello @andysCaplin , thanks for reporting this problem. I am examining the stack trace gstack.txt. According to the stack trace i see that the client started shutdown (thread 36) and the connection manager is waiting for the cluster connect thread to terminate gracefully. Thread 36 is waiting for thread 29. Thread 29 is at this line. I want to see the logs since there is a log here that prints how long it will sleep. Do you by any chance kept that log around?

So, I need to understand what happened, is that sleep time too long, or are we missing something else. The logs will help a lot.

[ihsandemir]

My analysis on the gdb.txt file:

According to the stack traces the client shutdown is waiting at listener_service shutdown. The shutdown is waiting for the event executor to be finished at this line, this is thread 22. It calls your event handler method and it seems like it is blocked on that method (wait_internal(boost::unique_lock<boost::mutex>&, bool)). Can you share what your queue item_listener event handler does? Is it something possible of blocking?

Also, if there are client logs, that would also be great. Thanks.

[andysCaplin]

Sorry I didn't grab the logs. I should have done that. We have the log handler set up so that log would be in our event log. ​

Whenever our queue listener gets a response it handles it asynchronously on a dedicated thread we have for handling Hazelcast events so it shouldn't block the Discovery thread.

Also if it was blocked in our code then I'd see that in the stack frames

[andysCaplin]

If/when the problem recurs I'll grab the stack and the logs.

[andysCaplin]

Hi @ihsandemir. I haven't had that shutdown issue but I saw another one when testing under valgrind.

It's not an issue with shutting down the client itself but with the Hazelcast library shutdown routine.

I got an invalid read in a Hazelcast thread for a queue that shutdown in another Hazelcast thread had deleted. This was in an overnight test and I suspect that the shutdown was due to servers disconnecting.

Unfortunately since it happened overnight and our logs cycle I don't have any related logs about lifecycle events.

This hasn't caused an issue for us in normal operation yet. I just saw it when I was looking for issues in our code and thought I should pass on the information.

==2125== Thread 29:
==2125== Invalid read of size 8
==2125==    at 0x551DF78: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) (in /usr/lib64/libstdc++.so.6.0.19)
==2125==    by 0xA68F6C: hazelcast::client::entry_event::entry_event(std::string const&, hazelcast::client::member&&, hazelcast::client::entry_event::type, hazelcast::client::typed_data&&, hazelcast::client::typed_data&&, hazelcast::client::typed_data&&, hazelcast::client::typed_data&&) (in /opt/workspace/bin/rttpd)
==2125==    by 0x913698: hazelcast::client::impl::EntryEventHandler<hazelcast::client::protocol::codec::map_addentrylistener_handler>::fire_entry_event(boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, int, boost::uuids::uuid, int) (EntryEventHandler.h:104)
==2125==    by 0x913F5A: hazelcast::client::impl::EntryEventHandler<hazelcast::client::protocol::codec::map_addentrylistener_handler>::handle_entry(boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, int, boost::uuids::uuid, int) (EntryEventHandler.h:57)
==2125==    by 0x98CB5D: hazelcast::client::protocol::codec::map_addentrylistener_handler::handle(hazelcast::client::protocol::ClientMessage&) (in /opt/workspace/bin/rttpd)
==2125==    by 0xAB1C5A: hazelcast::client::spi::impl::listener::listener_service_impl::process_event_message(std::shared_ptr<hazelcast::client::spi::impl::ClientInvocation>, std::shared_ptr<hazelcast::client::protocol::ClientMessage>) (in /opt/workspace/bin/rttpd)
==2125==    by 0xAB20B3: boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<hazelcast::client::spi::impl::listener::listener_service_impl::handle_client_message(std::shared_ptr<hazelcast::client::spi::impl::ClientInvocation>, std::shared_ptr<hazelcast::client::protocol::ClientMessage>)::{lambda()#2}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, std::allocator<void>*, boost::system::error_code const&, unsigned long) (in /opt/workspace/bin/rttpd)
==2125==    by 0xAC3161: boost::asio::detail::strand_executor_service::invoker<boost::asio::thread_pool::executor_type const>::operator()() (in /opt/workspace/bin/rttpd)
==2125==    by 0xAC3295: boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::thread_pool::executor_type const>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (in /opt/workspace/bin/rttpd)
==2125==    by 0x95D4C1: boost::asio::detail::scheduler::run(boost::system::error_code&) (in /opt/workspace/bin/rttpd)
==2125==    by 0xADBB79: boost::asio::detail::posix_thread::func<boost::asio::thread_pool::thread_function>::run() (in /opt/workspace/bin/rttpd)
==2125==    by 0x95049E: boost_asio_detail_posix_thread_function (in /opt/workspace/bin/rttpd)
==2125==    by 0x524AEA4: start_thread (in /usr/lib64/libpthread-2.17.so)
==2125==    by 0x5D7D96C: clone (in /usr/lib64/libc-2.17.so)

==2125==  Address 0x26634898 is 8 bytes inside a block of size 104 free'd
==2125==    at 0x4C2B51D: operator delete(void*) (vg_replace_malloc.c:586)
==2125==    by 0x826766: _M_release (shared_ptr_base.h:155)
==2125==    by 0x826766: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:148)
==2125==    by 0x8F718B: ~__shared_count (shared_ptr_base.h:728)
==2125==    by 0x8F718B: ~__shared_ptr (shared_ptr_base.h:1167)
==2125==    by 0x8F718B: ~shared_ptr (shared_ptr.h:103)
==2125==    by 0x8F718B: destroy_impl (optional.hpp:771)
==2125==    by 0x8F718B: destroy (optional.hpp:757)
==2125==    by 0x8F718B: ~optional_base (optional.hpp:255)
==2125==    by 0x8F718B: ~optional (optional.hpp:845)
==2125==    by 0x8F718B: boost::detail::shared_state<std::shared_ptr<hazelcast::client::spi::ClientProxy> >::~shared_state() (future.hpp:557)
==2125==    by 0x963807: boost::detail::future_sync_continuation_shared_state<boost::future<void>, std::shared_ptr<hazelcast::client::spi::ClientProxy>, boost::shared_future<std::shared_ptr<hazelcast::client::imap> > hazelcast::client::spi::ProxyManager::get_or_create_proxy<hazelcast::client::imap>(std::string const&, std::string const&)::{lambda(boost::future<void>)#1}>::~future_sync_continuation_shared_state() (in /opt/workspace/bin/rttpd)
==2125==    by 0x8E96C9: release (sp_counted_base_std_atomic.hpp:112)
==2125==    by 0x8E96C9: boost::detail::sp_counted_base::release() (sp_counted_base_std_atomic.hpp:108)
==2125==    by 0xAAE037: hazelcast::client::spi::ProxyManager::destroy() (in /opt/workspace/bin/rttpd)
==2125==    by 0xAAEC27: hazelcast::client::spi::lifecycle_service::shutdown() (in /opt/workspace/bin/rttpd)
==2125==    by 0x94CE86: boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >::run(boost::shared_ptr<boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}&&) (in /opt/workspace/bin/rttpd)
==2125==    by 0x948949: boost::detail::thread_data<void (*)(boost::shared_ptr<boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> > >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}&&), boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}>::run() (in /opt/workspace/bin/rttpd)
==2125==    by 0xAC8C26: thread_proxy (in /opt/workspace/bin/rttpd)
==2125==    by 0x524AEA4: start_thread (in /usr/lib64/libpthread-2.17.so)
==2125==    by 0x5D7D96C: clone (in /usr/lib64/libc-2.17.so)

[yemreinci]

Hi @ihsandemir. I haven't had that shutdown issue but I saw another one when testing under valgrind.

It's not an issue with shutting down the client itself but with the Hazelcast library shutdown routine.

I got an invalid read in a Hazelcast thread for a queue that shutdown in another Hazelcast thread had deleted. This was in an overnight test and I suspect that the shutdown was due to servers disconnecting.

Unfortunately since it happened overnight and our logs cycle I don't have any related logs about lifecycle events.

This hasn't caused an issue for us in normal operation yet. I just saw it when I was looking for issues in our code and thought I should pass on the information.

==2125== Thread 29:
==2125== Invalid read of size 8
==2125==    at 0x551DF78: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) (in /usr/lib64/libstdc++.so.6.0.19)
==2125==    by 0xA68F6C: hazelcast::client::entry_event::entry_event(std::string const&, hazelcast::client::member&&, hazelcast::client::entry_event::type, hazelcast::client::typed_data&&, hazelcast::client::typed_data&&, hazelcast::client::typed_data&&, hazelcast::client::typed_data&&) (in /opt/workspace/bin/rttpd)
==2125==    by 0x913698: hazelcast::client::impl::EntryEventHandler<hazelcast::client::protocol::codec::map_addentrylistener_handler>::fire_entry_event(boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, int, boost::uuids::uuid, int) (EntryEventHandler.h:104)
==2125==    by 0x913F5A: hazelcast::client::impl::EntryEventHandler<hazelcast::client::protocol::codec::map_addentrylistener_handler>::handle_entry(boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, boost::optional<hazelcast::client::serialization::pimpl::data> const&, int, boost::uuids::uuid, int) (EntryEventHandler.h:57)
==2125==    by 0x98CB5D: hazelcast::client::protocol::codec::map_addentrylistener_handler::handle(hazelcast::client::protocol::ClientMessage&) (in /opt/workspace/bin/rttpd)
==2125==    by 0xAB1C5A: hazelcast::client::spi::impl::listener::listener_service_impl::process_event_message(std::shared_ptr<hazelcast::client::spi::impl::ClientInvocation>, std::shared_ptr<hazelcast::client::protocol::ClientMessage>) (in /opt/workspace/bin/rttpd)
==2125==    by 0xAB20B3: boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<hazelcast::client::spi::impl::listener::listener_service_impl::handle_client_message(std::shared_ptr<hazelcast::client::spi::impl::ClientInvocation>, std::shared_ptr<hazelcast::client::protocol::ClientMessage>)::{lambda()#2}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, std::allocator<void>*, boost::system::error_code const&, unsigned long) (in /opt/workspace/bin/rttpd)
==2125==    by 0xAC3161: boost::asio::detail::strand_executor_service::invoker<boost::asio::thread_pool::executor_type const>::operator()() (in /opt/workspace/bin/rttpd)
==2125==    by 0xAC3295: boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::thread_pool::executor_type const>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (in /opt/workspace/bin/rttpd)
==2125==    by 0x95D4C1: boost::asio::detail::scheduler::run(boost::system::error_code&) (in /opt/workspace/bin/rttpd)
==2125==    by 0xADBB79: boost::asio::detail::posix_thread::func<boost::asio::thread_pool::thread_function>::run() (in /opt/workspace/bin/rttpd)
==2125==    by 0x95049E: boost_asio_detail_posix_thread_function (in /opt/workspace/bin/rttpd)
==2125==    by 0x524AEA4: start_thread (in /usr/lib64/libpthread-2.17.so)
==2125==    by 0x5D7D96C: clone (in /usr/lib64/libc-2.17.so)

==2125==  Address 0x26634898 is 8 bytes inside a block of size 104 free'd
==2125==    at 0x4C2B51D: operator delete(void*) (vg_replace_malloc.c:586)
==2125==    by 0x826766: _M_release (shared_ptr_base.h:155)
==2125==    by 0x826766: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:148)
==2125==    by 0x8F718B: ~__shared_count (shared_ptr_base.h:728)
==2125==    by 0x8F718B: ~__shared_ptr (shared_ptr_base.h:1167)
==2125==    by 0x8F718B: ~shared_ptr (shared_ptr.h:103)
==2125==    by 0x8F718B: destroy_impl (optional.hpp:771)
==2125==    by 0x8F718B: destroy (optional.hpp:757)
==2125==    by 0x8F718B: ~optional_base (optional.hpp:255)
==2125==    by 0x8F718B: ~optional (optional.hpp:845)
==2125==    by 0x8F718B: boost::detail::shared_state<std::shared_ptr<hazelcast::client::spi::ClientProxy> >::~shared_state() (future.hpp:557)
==2125==    by 0x963807: boost::detail::future_sync_continuation_shared_state<boost::future<void>, std::shared_ptr<hazelcast::client::spi::ClientProxy>, boost::shared_future<std::shared_ptr<hazelcast::client::imap> > hazelcast::client::spi::ProxyManager::get_or_create_proxy<hazelcast::client::imap>(std::string const&, std::string const&)::{lambda(boost::future<void>)#1}>::~future_sync_continuation_shared_state() (in /opt/workspace/bin/rttpd)
==2125==    by 0x8E96C9: release (sp_counted_base_std_atomic.hpp:112)
==2125==    by 0x8E96C9: boost::detail::sp_counted_base::release() (sp_counted_base_std_atomic.hpp:108)
==2125==    by 0xAAE037: hazelcast::client::spi::ProxyManager::destroy() (in /opt/workspace/bin/rttpd)
==2125==    by 0xAAEC27: hazelcast::client::spi::lifecycle_service::shutdown() (in /opt/workspace/bin/rttpd)
==2125==    by 0x94CE86: boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >::run(boost::shared_ptr<boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}&&) (in /opt/workspace/bin/rttpd)
==2125==    by 0x948949: boost::detail::thread_data<void (*)(boost::shared_ptr<boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> > >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}&&), boost::detail::future_async_shared_state<void, boost::detail::invoker<hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}> >, hazelcast::client::hazelcast_client::shutdown()::{lambda()#1}>::run() (in /opt/workspace/bin/rttpd)
==2125==    by 0xAC8C26: thread_proxy (in /opt/workspace/bin/rttpd)
==2125==    by 0x524AEA4: start_thread (in /usr/lib64/libpthread-2.17.so)
==2125==    by 0x5D7D96C: clone (in /usr/lib64/libc-2.17.so)

Hello @andysCaplin,

Thanks for sharing this error report. The stack traces were enough to debug the problem and I've sent a pull-request to fix it.

However, there probably isn't a relation between this bug and the original shutdown issue that you reported. So the issue can stay open.