feat: add a local endpoint type for inference directly from chat-ui

[nsarrazin]

Part of #1774

• Run models locally from .gguf file
• Auto-download model if not stored locally
• Use GPU if available
• Get chat template from .gguf file
• Show every .gguf in models/ as a model if MODELS is undefined
• Handle batching & multiple model inference at once more gracefully

[nsarrazin]

Something is going wrong in the build step... Found this relevant issue, trying to fix

[nsarrazin]

Works well you can do something like

MODELS=`[{
  "name": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
  "parameters": {
    "stop_sequences": ["<|im_end|>", "<|endoftext|>"]
  },
  "endpoints": [{"type": "local", "modelPath": "hf:HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF:Q4_K_M"}]
}]`

it will automatically use your GPU if available and download models to the models folder if not available locally.

It's still super rough as it doesn't handle running out of memory gracefully so I'm still working on dealing with this better.

I also want to automatically expose any .gguf files in the models/ folder as a model in chat-ui without having to set the MODELS env var

[nsarrazin]

Merging for now, it works well in local testing! Will update the docs to explain this when I'm done with the quick setup.