Advanced indexing output contains unexpected 0s

[manopapad]

See nv-legate/discussion#6 for full context. Copying here one of the bugs reported by @IlgarBaghishov:

Another issue we have now is with values of the masked a_train matrix. If I print the last rows of the a_train matrix instead of first rows after masking operation I get zeros like this:

[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]]

Seems like this advanced indexing operation did not finish fully and thus left some of the rows as zeros.

We also tried cupynumeric.compress but it just crashes silently.

@IlgarBaghishov could you please share the full script (updated with printing the last rows rather than the first rows), and how you're running it?

[IlgarBaghishov]

You can use this example code:

import cupynumeric as cnp

a = cnp.random.rand(3_000_000, 1500)
train_mask = cnp.random.rand(a.shape[0]) < 0.5
a_train = a[train_mask]

print(a_train[:3,:10])   # This line print random values
print(a_train[-3:,:10])  # This line results in zeros

I am running this on NVIDIA's Grace Hopper nodes on Vista supercomputer at TACC. Here is the nvidia-smi output:

Mon Jul  7 22:37:25 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GH200 120GB             On  |   00000009:01:00.0 Off |                    0 |
| N/A   25C    P0             86W /  900W |       1MiB /  97871MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

This is the command I use to run the python script:
legate --fbmem 93000 test_zeros.py

Please let me know if you need any other information and thank you so much for your help!

[ravankhidirov]

Hi team @manopapad @ipdemes !
Thanks for documenting this issue. As a temporary workaround, we have been using small batches for masking which seems to prevent zero rows. But we would love to see if you have any progress on this issue. Thanks in advance for your support.

[kharvey2]

Hi @ravankhidirov! I tried reproducing the issue on the latest packages, and print(a_train[-3:,:10]) in the example you shared, is not printing all zeros for me. Can you share the output you get when running legate-issue? Also, could you try running the code using the latest version of the Conda package to see if the issue persists? You can find instructions here under the "Installing Conda Packages" section for how to install the nightly top-of-tree builds. This will ensure you're using the most updated version of our packages!

[lightsighter]

Do we know if this is using indirect copy tasks or using Legion copy operations to perform the advanced indexing?

[ravankhidirov]

Hi team! @manopapad @ipdemes @lightsighter Thanks for responses

We tested the same script using the latest nightly top-of-tree Legate packages as suggested. The good news is that the issue with zeroed rows during indexing is no longer present. However, we're now encountering an out-of-memory error during execution.

Here are the details:

The code we are trying to run is as follows:

Python Script (t4.py)

from legate.io.hdf5 import from_file
import cupynumeric as np
import time

a = np.asarray(from_file("/work/10745/ravankhidirov/vista/data/be_data/X.hdf5", dataset_name="X"))
train_mask = np.random.rand(a.shape[0]) < 0.5
test_mask = ~train_mask

print("a shape:", a.shape, flush=True)
memory_bytes = a.size * a.itemsize
memory_mb = memory_bytes / (1024 ** 2)
print(f"a memory usage: {memory_mb:.2f} MB", flush=True)


a_train = a[train_mask]
print("done train", flush = True)
print(a_train[:2,:10], flush = True)
a_test = a[test_mask]
print("done test", flush = True)
print(a_test[-2:,:10], flush = True)
print(a_test[:2,:10], flush = True)

del a
print("deleted", flush = True)

q, r = np.linalg.qr(a_train)
print("did qr", flush = True)
print(q[:2,:10], flush = True)

And we are submitting the code using this sh file:

Job Script (run.sh)

#!/bin/sh
#SBATCH -N 1
#SBATCH -n 72
#SBATCH -o ck1
#SBATCH -p gh-dev
#SBATCH -t 00:30:00
#SBATCH -A CHE25011


nvidia-smi

legate --zcmem 60000 --fbmem 93000 t4.py

The error output is:

Thu Jul 24 15:23:29 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GH200 120GB             On  |   00000009:01:00.0 Off |                    0 |
| N/A   25C    P0             78W /  900W |       1MiB /  97871MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
a shape: (3002511, 1595)
a memory usage: 36537.21 MB
done train
[[745.85894958   5.39497333  -2.75847204  25.90323441  -5.22455458
   -4.65501324  -6.09911914  36.55466128  -5.76915657  -5.32533951]
 [-29.95083978  -1.50196355  -4.01499491  89.15857215   1.13277267
   -4.20540298  -3.93647822 -45.31518812   6.3745965   -1.53690685]]
done test
[0 - 4031afecf840]   14.552519 {5}{legate.mapper}: Failed to allocate 19174592360 bytes on memory 1e00000000000002 (of kind SOCKET_MEM) for region requirement(s) {0} of Inline mapping [/home1/10745/ravankhidirov/rdlv/t4.py:20] (UID 47)
[0 - 4031afecf840]   14.554607 {5}{legate.mapper}:   corresponding to a LogicalStore allocated at ["/home1/10745/ravankhidirov/rdlv/t4.py:18", {"file": "/home1/10745/ravankhidirov/rdlv/t4.py", "line": 18}]
[0 - 4031afecf840]   14.554628 {5}{legate.mapper}: There is not enough space because Legate is reserving 57492637742 of the available 101146689536 bytes for the following LogicalStores:
[0 - 4031afecf840]   14.554630 {5}{legate.mapper}: LogicalStore allocated at ["/home1/10745/ravankhidirov/rdlv/t4.py:18", {"file": "/home1/10745/ravankhidirov/rdlv/t4.py", "line": 18}]:
[0 - 4031afecf840]   14.554642 {5}{legate.mapper}:   Instance 4000000000800010 of size 9591526120 covering elements <751024,0>..<1502710,1594>
[0 - 4031afecf840]   14.554644 {5}{legate.mapper}:   Instance 400000000080000f of size 9583066240 covering elements <0,0>..<751023,1594>
[0 - 4031afecf840]   14.554646 {5}{legate.mapper}: LogicalStore allocated at ["/home1/10745/ravankhidirov/rdlv/t4.py:7", {"file": "/home1/10745/ravankhidirov/rdlv/t4.py", "line": 7}]:
[0 - 4031afecf840]   14.554649 {5}{legate.mapper}:   Instance 400000000080000a of size 1501256 covering elements <0>..<1501255>
[0 - 4031afecf840]   14.554650 {5}{legate.mapper}:     created for an operation launched at /home1/10745/ravankhidirov/rdlv/t4.py:7
[0 - 4031afecf840]   14.554651 {5}{legate.mapper}:   Instance 400000000080000c of size 1501255 covering elements <1501256>..<3002510>
[0 - 4031afecf840]   14.554652 {5}{legate.mapper}:     created for an operation launched at /home1/10745/ravankhidirov/rdlv/t4.py:7
[0 - 4031afecf840]   14.554653 {5}{legate.mapper}: LogicalStore allocated at ["/home1/10745/ravankhidirov/rdlv/t4.py:6", {"file": "/home1/10745/ravankhidirov/rdlv/t4.py", "line": 6}]:
[0 - 4031afecf840]   14.554654 {5}{legate.mapper}:   Instance 4000000000800004 of size 1501256 covering elements <0>..<1501255>
[0 - 4031afecf840]   14.554655 {5}{legate.mapper}:     created for an operation launched at /home1/10745/ravankhidirov/rdlv/t4.py:6
[0 - 4031afecf840]   14.554656 {5}{legate.mapper}:   Instance 4000000000800006 of size 1501255 covering elements <1501256>..<3002510>
[0 - 4031afecf840]   14.554657 {5}{legate.mapper}:     created for an operation launched at /home1/10745/ravankhidirov/rdlv/t4.py:6
[0 - 4031afecf840]   14.554658 {5}{legate.mapper}: LogicalStore allocated at (unknown provenance):
[0 - 4031afecf840]   14.554659 {5}{legate.mapper}:   Instance 4000000000800001 of size 19156026560 covering elements <0,0>..<1501255,1594>
[0 - 4031afecf840]   14.554661 {5}{legate.mapper}:   Instance 4000000000800000 of size 19156013800 covering elements <1501256,0>..<3002510,1594>
LEGATE ERROR: ================================================================================
LEGATE ERROR: System: Linux, 5.14.0-362.24.1.el9_3.aarch64+64k, c608-062.vista.tacc.utexas.edu, #1 SMP PREEMPT_DYNAMIC Wed Mar 13 14:27:26 EDT 2024, aarch64
LEGATE ERROR: Legate version: 25.8.0 (e5dde035a7553f596f934c0c28375e0d78c747f5)
LEGATE ERROR: Legion version: 25.8.0 (16441d05a66bdf0f21e271b943a60315f113a55e)
LEGATE ERROR: Configure options: --LEGATE_ARCH=arch-conda --with-python --with-cc=/tmp/conda-croot/legate/_build_env/bin/aarch64-conda-linux-gnu-cc --with-cxx=/tmp/conda-croot/legate/_build_env/bin/aarch64-conda-linux-gnu-c++ --build-march=armv8-a --cmake-generator=Ninja --with-openmp --with-cuda=0 --build-type=release --with-ucx
LEGATE ERROR: LEGATE_CONFIG: --fbmem 93000 --zcmem 60000 --logdir /home1/10745/ravankhidirov/rdlv
LEGATE ERROR: --------------------------------------------------------------------------------
LEGATE ERROR: Exception stack contains 1 exception(s) (bottom-most exception first):
LEGATE ERROR:
LEGATE ERROR: #0 Legate called abort at /tmp/conda-croot/legate/work/src/cpp/legate/mapping/detail/base_mapper.cc:1298 in report_failed_mapping_()
LEGATE ERROR: #0 Out of memory
LEGATE ERROR:
LEGATE ERROR: Stack trace (most recent call first):
LEGATE ERROR: #0  0x0000400037639f5f at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.08.00
LEGATE ERROR: #1  0x00004000376339ff at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.08.00
LEGATE ERROR: #2  0x0000400037634c4b at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.08.00
LEGATE ERROR: #3  0x0000400037634f33 at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.08.00
LEGATE ERROR: #4  0x0000400037635643 at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.08.00
LEGATE ERROR: #5  0x00004000376358df at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.08.00
LEGATE ERROR: #6  0x00004000384dd523 at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
LEGATE ERROR: #7  0x00004000388cfd97 at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
LEGATE ERROR: #8  0x0000400038701607 at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
LEGATE ERROR: #9  0x000040003870d25b at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
LEGATE ERROR: #10 0x00004000389decaf at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
LEGATE ERROR: #11 0x000040003a28a5e3 at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
LEGATE ERROR: #12 0x000040003a28a777 at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
LEGATE ERROR: #13 0x000040003a288cb7 at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
LEGATE ERROR: #14 0x000040003a292d0b at /home1/10745/ravankhidirov/miniconda3/envs/legate-nightly/lib/python3.13/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
LEGATE ERROR: #15 0x000040002f8075ff at /usr/lib64/libc.so.6
LEGATE ERROR: ================================================================================

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

As we see, the matrix a is ~36.5 GB. When applying the train mask, it runs fine — but during the test mask, it runs out of memory. It seems the train masking step requires almost double memory, and that memory isn’t released before the test masking starts. We tried these using 25.07.00 and 25.08.00 version and got the same results.

Thanks!!!

[ipdemes]

@ravankhidirov can you try increasing -sysmem ? The error you are seeing is saying there is no enough CPU memory to perform print(a_train[:2,:10], flush = True). When you call print, the data is being copied from GPUs to CPUs.
What happen is:
Also, your mrmory allocation is larger on GPU: you allocate ~36.5 GB for a, then some amount of memory is allocated for train_mask and a_train as well

[ravankhidirov]

Hi @ipdemes . Thanks for the response. We checked your suggestion (by setting sysmem to 128000). So the error we got seemed to be solved. but now we have another issue. The code is like that:

from legate.io.hdf5 import from_file
import cupynumeric as np
import time

a = np.asarray(from_file("/work/10745/ravankhidirov/vista/data/be_data/X.hdf5", dataset_name="X"))
train_mask = np.random.rand(a.shape[0]) < 0.5
test_mask = ~train_mask

print("a shape:", a.shape, flush=True)
memory_bytes = a.size * a.itemsize
memory_mb = memory_bytes / (1024 ** 2)
print(f"a memory usage: {memory_mb:.2f} MB", flush=True)


a_train = a[train_mask]
print("done train", flush = True)
print(a_train[:2,:10], flush = True)
a_test = a[test_mask]
print("done test", flush = True)
print(a_test[-2:,:10], flush = True)
print(a_test[:2,:10], flush = True)

del a
print("deleted", flush = True)

q, r = np.linalg.qr(a_train)
print("did qr", flush = True)
print(q[:2,:10], flush = True)

The output:

 Fri Jul 25 05:53:37 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GH200 120GB             On  |   00000009:01:00.0 Off |                    0 |
| N/A   28C    P0             87W /  900W |       1MiB /  97871MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
a shape: (3002511, 1595)
a memory usage: 36537.21 MB
done train
[[745.85894958   5.39497333  -2.75847204  25.90323441  -5.22455458
   -4.65501324  -6.09911914  36.55466128  -5.76915657  -5.32533951]
 [-50.75991209 -12.83782766  -2.68057037  19.69751638   3.11615877
    3.919075   -11.00095446 -47.21921808   0.84424413  -7.46781616]]
done test
[[334.31807188  79.42721303  -4.53969169  32.63012877   0.34772092
   -4.65172078  -3.38637454 -47.22612334   0.46546726   3.37824525]
 [ 58.75702239  -1.42646588  -2.14960546  54.41389479   1.24914396
   -9.21892349   0.69568364  -0.40110467   7.41358594  -0.88882055]]
[[ 1.77765029e+02  2.26405832e+01  1.72682659e-01 -5.08910161e+01
   1.35788812e+00 -3.16507033e+00  2.41284913e+00 -9.01915493e+01
  -3.24156757e+00  9.27116587e+00]
 [ 2.22259083e+01  1.60355332e+00  6.64064275e-02  3.65134463e+00
  -7.89238808e-01 -6.01287543e+00 -2.89352748e+00 -1.70180637e+01
   2.55902176e+00  5.87963264e+00]]
deleted

So it hangs after deleting the big matrix. We are allocating around 93 GB GPU, and a matrix has size of 36.5 GB, (mask array has very negligible size ~23 megabayt), While masking we separate this array into train and test (~18GB each) after deleting a matrix there must be enough space for QR operation. However it crashes silently. Seems like a matrix is not deleted properly and it holds some space.

Thanks!

[kharvey2]

Hi @ravankhidirov, would you be able to send us an input file so we can properly reproduce this?

[IlgarBaghishov]

Hi @kharvey2, the input file is not necessary. I have tried with a random matrix too and I still get the same silent crash, so you can use this instead of line 5:

a = cnp.random.rand(3_000_000, 1500)

I think the important part is to have a matrix of size more than 35-40% of the available GPU memory. If you calculate the memory usage, theoretically this code should run successfully with such a matrix but it silently crashed after deleting that original matrix and it does not continue to the QR step. Thank you for your help!

[ravankhidirov]

Hi team! @kharvey2

Just checking in - is there any progress on this issue? Could you reproduce the problem or do you need us to provide more details? Also, is there any method to force to free gpu memory (something similar to cp._default_memory_pool.free_all_blocks() in cupy)?

Thanks!

[ipdemes]

@ravankhidirov @kharvey2 wasn't able to reproduce your issue on our local cluster. She is going to try on a machine with GH200.

Also, is there any method to force to free gpu memory

I don't think so. You can delete arrays by del Array_name