Install/run issues of cuPyNumeric

[ShaohuiLiu]

I tried the default installation on doc. But I was not able to run tests. cupynumeric has been installed to current env but I still received "ModuleNotFoundError: No module named 'cupynumeric'".
Hardware: 2x AMD EPYC 9454 48-Core Processor ("Genoa"), NVIDIA H100 PCIE 80GB
OS:
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: Rocky
Description: Rocky Linux release 8.10 (Green Obsidian)
Release: 8.10
Codename: GreenObsidian
Loaded Modules:

1. cmake/3.24.2 2) pmix/3.2.3 3) TACC 4) cuda/12.0 (g) 5) gcc/11.2.0 6) python3/3.9.7 7) impi/19.0.9

[manopapad]

Could we see the output of legate-issue? Also the output of conda info legate and conda info cupynumeric?

[ShaohuiLiu]

Thx for checking this issue. Below are the outputs:

legate-issue
Python      :  3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
Platform    :  Linux-4.18.0-372.26.1.el8_6.x86_64-x86_64-with-glibc2.28
Legion      :  24.11.0 (commit: ac6aae07cb18fa9de978b073766dd9e3def29dbb)
Legate      :  24.11.1.dev3+g53e9db6f
cuPynumeric :  (failed to detect)
Numpy       :  1.26.4
Scipy       :  1.14.1
Numba       :  (failed to detect)
CTK package :  cuda-version-12.6-h7480c83_3 (conda-forge)
GPU driver  :  535.104.12
GPU devices :
  GPU 0: NVIDIA A100-PCIE-40GB

Now conda info legate/cupynumeric return nothing. legate seems to work.

legate
Python 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

However, if I tried to install again conda install -c conda-forge -c legate legate-core, I would receive the following error

Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - legate-core

Current channels:

  - https://conda.anaconda.org/conda-forge/linux-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://conda.anaconda.org/legate/linux-64
  - https://conda.anaconda.org/legate/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

[manopapad]

legate-core is an old package name. The latest name is legate. Installing latest cupynumeric should pull in the latest version of that package automatically.

Based on the output of legate-issue and conda list cupynumeric, It looks like you don't have cupynumeric installed. Could you please try creating a new conda environment, with the following command?

conda create -n myenv -c conda-forge -c legate cupynumeric

This should install versions 24.11 of both package cupynumeric and legate.

[ShaohuiLiu]

Thanks! It's working on now on a single node. I'll run some multi-node tests later. Actually the command is the first I tried earlier but I received some weird error (e.g. < 80 not enough cores) so I resorted to the legate installation later.

[ShaohuiLiu]

Now I see what happened before. The package ran well on 1 node but has issues when running one 2 nodes. Could it be the issue with MPI compatibility?

If I specify launcher as 'mpirun',

WARNING: There was an error initializing an OpenFabrics device.

  Local host:   c301-001
  Local device: mlx5_0

[c301-001:1567937:1:1567961] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xf8)
[c301-001:1567937] *** Process received signal ***
[c301-001:1567937] Signal: Segmentation fault (11)
[c301-001:1567937] Signal code: Address not mapped (1)
[c301-001:1567937] Failing at address: 0x8

An MPI communication peer process has unexpectedly disconnected.  This
usually indicates a failure in the peer process (e.g., a crash or
otherwise exiting without calling MPI_FINALIZE first).

Although this local MPI process will likely now behave unpredictably
(it may even hang or crash), the root cause of this problem is the
failure of the peer -- that is what you need to investigate.  For
example, there may be a core file that you can examine.  More
generally: such peer hangups are frequently caused by application bugs
or other external events.

  Local host: c301-002
  Local PID:  766341
  Peer host:  c301-001

Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

[c301-001.ls6.tacc.utexas.edu:1567923] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init
[c301-001.ls6.tacc.utexas.edu:1567923] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

mpirun noticed that process rank 0 with PID 0 on node c301-001 exited on signal 11 (Segmentation fault).

If I don't specify the launcher,

WARNING: There was an error initializing an OpenFabrics device.

  Local host:   c301-001
  Local device: mlx5_0

[c301-001:1568649] *** Process received signal ***
[c301-001:1568649] Signal: Segmentation fault (11)
[c301-001:1568649] Signal code: Address not mapped (1)
[c301-001:1568649] Failing at address: (nil)

[manopapad]

MPI incompatibility would be my first hunch. You may want to conda uninstall -f openmpi to remove the OpenMPI conda package (that the Legate packages were built against), and try using the Legate MPI wrapper to combine Legate with your system-level MPI.

Because this is inherently a very cluster-specific workflow, it would help to know more about the cluster you're trying to use, the version of MPI that you have available on your system etc. And also the exact legate command you're using, and the --verbose output from that, that will have the actual mpirun command executed.