Update request pool to allow individual request operations

[zhao-shihan]

Addressing #47. Changes:

• Update request pool to allow request operations individually
• Add check in request_pool::cancel

[zhao-shihan]

Actually I'm not quite sure about this. I added this check since I notice that it is not impossible for user to push an already moved request into the pool. So when I noticed that base_request::cancel (line 100) has this check, I added also here. But if there's an assumption that user will never push a null request into the pool, this check can be removed for consistency.

[rabauke]

I need to double-check this. But I think the check against MPI_REQUEST_NULL is actually needed here. Thanks for spotting this.

[rabauke]

I am not sure if this is actually an improvement. I can hardly imagine a situation where one wants to put a request into a request_pool and wait for its completion individually (rather than using waitall or waitany). If one needs to wait for a specific request, one should not move it into a pool.

Furthermore, the new functions are inconsistent with the existing class method. The new methods return a status object, whereas the existing one operate on status objects that are maintained by the pool but not accessible by the user.

[zhao-shihan]

There is a common situation where we need both individual MPI_Start/Wait and MPI_Waitsome: a master-slave scheduler. The master process/thread maintain a task queue and communicates with worker processes with persistent point-to-point send/recv. The master process uses MPI_Waitsome to wait for one or more workers finishing their current jobs and send next batch of jobs to workers by individual MPI_Start and wait for their following response by MPI_Wait.

The issue actually raise from the flexibility. If we use native MPI interface, it is natural to perform both individual operations like MPI_Start/Wait and multiple operations like MPI_Wait/Testall/some/any since we are maintaining an array of requests. But in MPL, MPI_Wait/Testall/some/any are wrapped into a request pool, and cannot be call with an request array (e.g. std::vector<prequest>), and we have to completely fallback to native MPI since there's not interoperability between MPL request and MPI request if we need both individual and multiple operations.

If we wish to keep the current pattern of request_pool, I suggest

1. Adding functions like mpl::waitsome(<range of request>) to allow user to perform those actions. (<range of request> can be pair of random access iterators of a contiguous request array, or std::span<T> if C++20 is enabled)
2. Improve the interoperability between MPL request and MPI request, but I'm not really sure whether it is possible.

[zhao-shihan]

I think try_get_status is a more semantic naming than get_status_of.