You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Feature Request – Usage of Bonsai 2 27B as a model #19
How about using Bonsai 2 27B as a model? (See: https://www.youtube.com/watch?v=9sPRhdnVSd4 (video in German language) or elsewhere). Wouldn’t a 27B model produce translations of a better quality and still work quite well on a computer with 16 GB RAM?
Bonsai 2 27B can be used offline on a 16 GB laptop.
The model uses ternary quantization to compress a 27-billion-parameter architecture into a file size of approximately 5.95 GB to 8.60 GB, depending on the specific build (GGUF or MLX) and whether it includes the vision tower. This footprint fits comfortably within the 16 GB of unified memory found on most modern consumer laptops, such as Apple Silicon Macs or PCs with integrated graphics.
Key considerations for running it offline on 16 GB RAM include:
Runtime Requirements: The model requires PrismML’s specific fork of llama.cpp or the MLX framework; it is not compatible with standard runtimes like Ollama or stock LM Studio.
Context Window: While the model weights fit easily, the remaining RAM must accommodate the KV cache. A 16 GB laptop can typically handle moderate context lengths (e.g., 16K–32K tokens) comfortably, though larger contexts (up to the native 262K limit) will consume significant memory.
Performance: On an Apple M5 Max laptop, the model can generate text at approximately 47 tokens per second using the MLX build, demonstrating viable local inference speeds.
OK, that would be too complicated for me right now, too. Still, I'm looking for an even more powerful model that would still run on a laptop with 16 GB of RAM. How about, for example, the Hy-MT2-7B with a quantization of Q4_K_M? Would that be easy for you to implement without requiring any additional backend work?
Activity
[-]Feature Request[/-][+]Feature Request – Usage of Bonsai 2 27B as a model[/+]kdroidFilter commentedon Oct 5, 2026
27b seems very heavy for the majority of users, doesn't it?
kamaqnkatyqn commentedon Oct 5, 2026
Bonsai 2 27B can be used offline on a 16 GB laptop.
The model uses ternary quantization to compress a 27-billion-parameter architecture into a file size of approximately 5.95 GB to 8.60 GB, depending on the specific build (GGUF or MLX) and whether it includes the vision tower. This footprint fits comfortably within the 16 GB of unified memory found on most modern consumer laptops, such as Apple Silicon Macs or PCs with integrated graphics.
Key considerations for running it offline on 16 GB RAM include:
Runtime Requirements: The model requires PrismML’s specific fork of llama.cpp or the MLX framework; it is not compatible with standard runtimes like Ollama or stock LM Studio.
Context Window: While the model weights fit easily, the remaining RAM must accommodate the KV cache. A 16 GB laptop can typically handle moderate context lengths (e.g., 16K–32K tokens) comfortably, though larger contexts (up to the native 262K limit) will consume significant memory.
Performance: On an Apple M5 Max laptop, the model can generate text at approximately 47 tokens per second using the MLX build, demonstrating viable local inference speeds.
So, why not give it a try?
kdroidFilter commentedon Oct 5, 2026
That would require adding a backend on top of LiteRT; if you want to implement it, I might be interested, but for now, I prefer to keep things simple.
kamaqnkatyqn commentedon Oct 5, 2026
OK, that would be too complicated for me right now, too. Still, I'm looking for an even more powerful model that would still run on a laptop with 16 GB of RAM. How about, for example, the Hy-MT2-7B with a quantization of Q4_K_M? Would that be easy for you to implement without requiring any additional backend work?