Investigate device memory usage outside memory pool

[gaohao95]

By default, the memory pool size used is the total GPU memory - 500MB. During some OOM runs, we observed using smaller memory pool solves the OOM issue. This indicates that the program uses a lot of device memory outside of memory pool. Tracking what memory is used outside of memory pool and make sure they are allocated within the memory pool should fix such issues.

[gaohao95]

One potential cause of this OOM is that thrust does not use RMM memory pool during table generation:

distributed-join/generate_dataset/generate_dataset.cuh

Line 231 in bc7563a

thrust::sort(thrust::device, build_tbl_sorted.begin(), build_tbl_sorted.end());