undefined symbol: cdequantize_blockwise_fp32

[jabberjabberjabber]

Install problem?

(unsloth_env) user@t7610:$ python train.py
/home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
/home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/__init__.py:71: UserWarning: Unsloth: Running `ldconfig /usr/lib64-nvidia` to link CUDA.
  warnings.warn(
/sbin/ldconfig.real: Can't create temporary cache file /etc/ld.so.cache~: Permission denied
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/__init__.py", line 68, in <module>
    cdequantize_blockwise_fp32 = bnb.functional.lib.cdequantize_blockwise_fp32
  File "/home/user/miniconda3/envs/unsloth_env/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/home/user/miniconda3/envs/unsloth_env/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cdequantize_blockwise_fp32

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/__init__.py", line 99, in <module>
    cdequantize_blockwise_fp32 = bnb.functional.lib.cdequantize_blockwise_fp32
  File "/home/user/miniconda3/envs/unsloth_env/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/home/user/miniconda3/envs/unsloth_env/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cdequantize_blockwise_fp32

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/train.py", line 1, in <module>
    from unsloth import FastLanguageModel
  File "/home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/__init__.py", line 102, in <module>
    raise ImportError("Unsloth: CUDA is not linked properly.\n"\
ImportError: Unsloth: CUDA is not linked properly.
We tried running `ldconfig /usr/lib64-nvidia` ourselves, but it didn't work.
You need to run in your terminal `sudo ldconfig /usr/lib64-nvidia` yourself, then import Unsloth.
Also try `sudo ldconfig /usr/local/cuda-xx.x` - find the latest cuda version.

Wed Feb 28 04:57:41 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla P40                      Off | 00000000:03:00.0 Off |                  Off |
| N/A   42C    P0              51W / 250W |  10166MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Quadro P600                    Off | 00000000:04:00.0  On |                  N/A |
| 34%   36C    P8              N/A /  N/A |    487MiB /  2048MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  Tesla P40                      Off | 00000000:81:00.0 Off |                  Off |
| N/A   50C    P0              52W / 250W |   9612MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2916      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A      3090    C+G   ...libexec/gnome-remote-desktop-daemon      149MiB |
|    0   N/A  N/A    472727      C   python                                    10010MiB |
|    1   N/A  N/A      2916      G   /usr/lib/xorg/Xorg                          283MiB |
|    1   N/A  N/A      3137      G   /usr/bin/gnome-shell                         49MiB |
|    1   N/A  N/A      3646      G   /usr/libexec/xdg-desktop-portal-gnome        25MiB |
|    1   N/A  N/A    104244      G   /usr/bin/nautilus                            26MiB |
|    1   N/A  N/A    116633      G   ...,262144 --variations-seed-version=1       31MiB |
|    1   N/A  N/A    120705      G   /usr/bin/krusader                             1MiB |
|    1   N/A  N/A    471430      G   /usr/bin/gnome-text-editor                   17MiB |
|    1   N/A  N/A    472727      C   python                                       46MiB |
|    2   N/A  N/A      2916      G   /usr/lib/xorg/Xorg                            4MiB |
|    2   N/A  N/A    472727      C   python                                     9606MiB |
+---------------------------------------------------------------------------------------+

from unsloth import FastLanguageModel
import torch
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

local_dataset_path = "comments.jsonl"

dataset = load_dataset("json", data_files={"train": local_dataset_path}, split="train")
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/tinyllama-bnb-4bit",  # Change this to your model
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
    model=model,
)
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    dataset_text_field="Output",
    max_seq_length=2048,
    tokenizer=tokenizer,
    args=TrainingArguments(
       
    ),
)
trainer.train()

(unsloth_env) user@:/$ conda info -s -v

     active environment : unsloth_env
    active env location : /home/user/miniconda3/envs/unsloth_env
            shell level : 2
       user config file : /home/user/.condarc
 populated config files :
          conda version : 23.11.0
    conda-build version : not installed
         python version : 3.11.5.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=ivybridge
                          __conda=23.11.0=0
                          __cuda=12.2=0
                          __glibc=2.38=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /home/user/miniconda3  (writable)
      conda av data dir : /home/user/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/user/miniconda3/pkgs
                          /home/user/.conda/pkgs
       envs directories : /home/user/miniconda3/envs
                          /home/user/.conda/envs
               platform : linux-64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.11.5 Linux/6.5.0-17-generic ubuntu/23.10 glibc/2.38 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.3
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False


# conda environments:
#
base                     /home/user/miniconda3
LWM                      /home/user/miniconda3/envs/LWM
coolercontrol            /home/user/miniconda3/envs/coolercontrol
crewai                   /home/user/miniconda3/envs/crewai
forge                    /home/user/miniconda3/envs/forge
ogbu                     /home/user/miniconda3/envs/ogbu
ollama                   /home/user/miniconda3/envs/ollama
train                    /home/user/miniconda3/envs/train
unsloth_env           *  /home/user/miniconda3/envs/unsloth_env

sys.version: 3.11.5 (main, Sep 11 2023, 13:54:46) [GC...
sys.prefix: /home/user/miniconda3
sys.executable: /home/user/miniconda3/bin/python
conda location: /home/user/miniconda3/lib/python3.11/site-packages/conda
conda-build: None
conda-content-trust: /home/user/miniconda3/bin/conda-content-trust
conda-env: /home/user/miniconda3/bin/conda-env
user site dirs:

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: unsloth_env
CONDA_EXE: /home/user/miniconda3/bin/conda
CONDA_PREFIX: /home/user/miniconda3/envs/unsloth_env
CONDA_PREFIX_1: /home/user/miniconda3
CONDA_PROMPT_MODIFIER: (unsloth_env)
CONDA_PYTHON_EXE: /home/user/miniconda3/bin/python
CONDA_ROOT: /home/user/miniconda3
CONDA_SHLVL: 2
CURL_CA_BUNDLE: <not set>
LD_PRELOAD: <not set>
PATH: /home/user/miniconda3/envs/unsloth_env/bin:/home/user/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>

[danielhanchen]

@jabberjabberjabber Oh no more bitsandbytes issues :((

[sahilqure]

@danielhanchen I understand that we need to have same cudaversion for both pytorch and bitsandbytes, but it seems that bitsandbytes doesn't support cuda12 properly. Can you please check, this problem is very old and I have raised this issue earlier as well while working with LLamaFactory.

[danielhanchen]

@sahilqure Actually was gonna ask did you install bitsandbytes via Conda or pip? If you did pip - maybe first try uninstalling it, then install it with Conda / Mamba

[jabberjabberjabber]

@sahilqure Actually was gonna ask did you install bitsandbytes via Conda or pip? If you did pip - maybe first try uninstalling it, then install it with Conda / Mamba

I used this guide to install: #73

Do you know how to install bitsandbytes via conda? This is what happened when I tried:

(unsloth_env) user@t7610:~$ pip uninstall bitsandbytes
Found existing installation: bitsandbytes 0.41.1
Uninstalling bitsandbytes-0.41.1:
  Would remove:
    /home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/bitsandbytes-0.41.1.dist-info/*
    /home/user/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/bitsandbytes/*
Proceed (Y/n)? y
  Successfully uninstalled bitsandbytes-0.41.1
(unsloth_env) user@t7610:~$ conda install bitsandbytes
Retrieving notices: ...working... done
Channels:
 - defaults
 - nvidia
 - pytorch
 - xformers
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - bitsandbytes

Current channels:

  - defaults
  - https://conda.anaconda.org/nvidia/linux-64
  - https://conda.anaconda.org/pytorch/linux-64
  - https://conda.anaconda.org/pytorch/noarch
  - https://conda.anaconda.org/xformers/linux-64

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

[jabberjabberjabber]

OK, I got it working by reinstalling using CUDA 11.8 instead of 12.1

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

pip install bitsandbytes

pip install ipython

[danielhanchen]

Oh great!! Glad you managed to solve it! Sorry on the issue again!