Introducing Unsloth Desktop

[danielhanchen]

---

This discussion was created from the release Introducing Unsloth Desktop.

[NoMansPC]

My only gripe with it is that it defaults to 4k context when it thinks it might exceed my VRAM. I'd rather the default behavior was it using RAM too instead of limiting the context. Other than that, it's excellent. It makes it way more convenient.

[shimmyshimmer]

Thank you, would it be possible to make a github isue? Thanks!

[NoMansPC]

Thank you, would it be possible to make a github isue? Thanks!

Done.

[jiriteach]

This is awesome. I have two severs with GPU's running Unsloth Studio. Those run well but having Unsloth Desktop on my Macbook as a client would be awesome. I've connected them as custom connections from Unsloth Desktop - this works but many things are greyed out like code or search etc.
Wanting to use Unsloth desktop interface to code directly to local folders using models coming from my servers running Unsloth Studio. Would this be possible?

Thanks

[shimmyshimmer]

Hello thanks for trying it out. Yes it should if you bypass permissions. Could you also make a feature request if possible thanks

[martin-braun]

@jiriteach You should be able to just download and manage models on your server using Studio, and then just run them using llama-server --models-dir "${XDG_CACHE_HOME-$HOME/.cache}/huggingface/hub" that gets all your models ready in the llama router, so they will be loaded on demand. Then on your Desktop, you can simply connect to it in Settings/Connections using the llama.cpp module. You can still do the same on your web UI on your server by connecting to http://127.0.0.1:/v1.

Basically you host the models yourself, not through Unsloth, you only use Unsloth to manage and access them.

[nickspath]

Might be a dumb question but, what is the difference between the desktop app and for example running on linux the client and server as per usual?