GPU support may not be enabled

[svanschalkwyk]

"routes.go:634: Warning: GPU support may not enabled, check you have installed install GPU drivers: nvidia-smi command failed"
NVIDIA-SMI 535.86.10 Driver Version: 535.86.10 CUDA Version: 12.2
5.15.0-86-generic #96-Ubuntu
docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi:
NVIDIA-SMI 535.86.10 Driver Version: 535.86.10 CUDA Version: 12.2

[svanschalkwyk]

llama.go:333: skipping accelerated runner because num_gpu=0
genai-stack-llm-1 | 2023/10/21 03:52:30 llama.go:356: starting llama runner
genai-stack-llm-1 | 2023/10/21 03:52:30 llama.go:408: waiting for llama runner to start responding
genai-stack-llm-1 | [GIN] 2023/10/21 - 03:52:30 | 500 | 873.831145ms | 172.20.0.5 | POST "/api/generate"
genai-stack-llm-1 | 2023/10/21 03:52:30 llama.go:373: error starting llama runner: llama runner process has terminated
genai-stack-llm-1 | 2023/10/21 03:52:30 llama.go:438: llama runner stopped with error: signal: illegal instruction (core dumped)

[jexp]

Is that on linux? Can you share more of your OS and machine specs.
@mchiang0610

[matthieuml]

If you are using the Linux profile, you should add this to the llm service:

deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

And ensure that you have nvidia-container-toolkit installed

[svanschalkwyk]

Thank you. That works. Steph van Schalkwyk Principal, remcam llc, dba. Remcam Search Engines Elastic Partner +1.314.452.2896 ***@***.*** Skype: svanschalkwyk <http://linkedin.com/in/vanschalkwyk>

…

On Sat, Oct 21, 2023 at 12:36 PM Matthieu Mérigot-Lombard < ***@***.***> wrote: If you are using the Linux profile, I believe you should add this to the llm service: deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] And ensure that you have nvidia-container-toolkit installed — Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABTCTDNMV4LP7RYORMN5SLYAQB2PAVCNFSM6AAAAAA6J2LBACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZTHA3TENRYGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jexp]

Added it to the readme.