nvdisasm error when launching modified kernel

[elainewu1222]

I'm working on a demo to encrypt kernel code in CPU side and then decrypt it in GPU side.
To locate GPU side kernel code position, I use NVBit's API nvbit_get_func_addr.
However, after modifying CPU side kernel code by intercepting cuLibraryLoadData, nvbit will throw error when launching kernel:

nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000000
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000010
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000020
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000030
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000040
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000050
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000060
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000070
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000080
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000090
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x000000a0
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x000000b0
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x000000c0
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x000000d0
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x000000e0
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x000000f0
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000100
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000110
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000120
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000130
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000140
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000150
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000160
nvdisasm error   : Unrecognized operation for functional unit 'uC' at address 0x00000170
Command "nvdisasm -b SM89 /tmp/nvbit_code_vudSNp > /tmp/nvbit_sass_gLpbNq;" exited with signal 0(256). Is the command in your PATH?

It seems that NVBit will call nvdisasm before launching the kernel, and it uses the modified kernel code. Actually I think NVbit should have no idea about the modified kernel code since I made a copy in the interception function and the modified code is invisible to NVbit. Unless NVbit read back the code through some CUDA driver API or it reads the code back from GPU side.

Can you post more details on NVbit's implementation, especially about how it handle kernel launch? I've read the NVBit's paper but it doesn't have much detail about it.

[Forsworns]

Could you plz share which version nvbit and cuda version do you use? I use the nvbit 1.7.6, cuda runtime 12.8 and cuda driver 570.133.20, and I cannot intercept cuLibraryLoadData at all. But it definitely called cuLibraryGetModule.

My snippet simply works like this

void nvbit_at_cuda_event(CUcontext ctx, int is_exit, nvbit_api_cuda_t cbid,
                         const char *name, void *params, CUresult *pStatus)
{
    if(strstr(name, "cuLibrary")!=NULL){
        // I can only get the `cuLibraryGetModule` output
        printf("[nvbit] ====== calling %s=========\n", name);
    }
    if (cbid == API_CUDA_cuLibraryLoadData) {
        cuLibraryLoadData_params *p = (cuLibraryLoadData_params *)params;
        // nothing printed
        printf("[nvbit] ====== %d cuLibraryLoadData library *%p == %p, code %p\n", is_exit, p->library, *(p->library), p->code);
    }
}