`<Future as std::..::Future>::poll()` always allocates, growing until we're woken

[TheBlueMatt]

<Future as std::..::Future>::poll(), if state.complete is not set, always pushes the provided waker onto the callbacks vec. This means if we get poll'd regularly our memory will grow and grow until we get woken, which may be quite a while. Thus, this may end up using a very nontrivial amount of memory, at least on busy nodes.

[nate-lightspark]

To add more color to the issue, carrying notes from the Discord thread:

Looking at the memory profiler run with dhat leads to the result in the screenshot. The hot piece looks to be chain_monitor.get_update_future() :

Profiler Result 1

Profiler After

It is an adapted from this example code

The code block is like this for the severe memory leak: Gist 1

Change is like this, seems to be resolving the issue (or makes it less severe): Gist 2