nccl/rccl integration

[seagater]

Use dlopen to load nccl/rccl apis from shared library to replace Fallback code for Allgather, Allreduce, Broadcast, ReduceScatter.

Add two related environment variables
-x MSCCLPP_ENABLE_SHARED_LIB=TRUE -x MSCCLPP_NCCL_LIB_PATH=path_to_libnccl.so/librccl.so

[SreevatsaAnantharamu]

Additional symbols that need to be loaded from libnccl.so:

• ncclGetUniqueId
• ncclCommInitRank

nccl_ops_t->ncclGetUniqueId(

Here, before returning, you have to call nccl_ops_t->ncclCommInitRank and create a new real NCCL's communicator. Inside MSCCL++'s ncclComm_t, you can have a void * or ncclComm_t nccl_comm.

nccl_ops_t->ncclCommInitRank(&commPtr->nccl_comm, ... )

[seagater]

Add two related environment variables:
-x MSCCLPP_ENABLE_SHARED_LIB=TRUE -x MSCCLPP_NCCL_LIB_PATH=path_to_libnccl.so/librccl.so

Support dlopen for following nccl apis:
ncclCommInitRank
ncclGetUniqueId
ncclCommDestroy
ncclCommUserRank
ncclAllGather
ncclAllReduce
ncclBroadcast

Pass following tests
nccl-test:
Allgather, Allreduce, Broadcast, ReduceScatter

rccl-test:
Allgather, Allreduce, Broadcast, ReduceScatter

[Binyang2014]

Thanks for this PR! A few comments

[SreevatsaAnantharamu]

I have left comments. Please take a look.

The most important is that we have to create both NCCL's communicator and our MSCCLPP communicator.

[seagater]

Update the environment variable names:
-x MSCCLPP_ENABLE_NCCL_FALLBACK=TRUE
-x MSCCLPP_NCCL_LIB_PATH=$NCCL_LIB
-x MSCCLPP_FORCE_NCCL_FALLBACK_OPERATION="allreduce,allgather,broadcast,reducescatter" or "all"

[Binyang2014]

Overall looks good to me, leaves some comments. BTW maybe we can add CI for this

[Binyang2014]

LGTM

[Binyang2014]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).