Skip to content

Feature Request – Usage of Bonsai 2 27B as a model #19

Open

Description

@kamaqnkatyqn

How about using Bonsai 2 27B as a model? (See: https://www.youtube.com/watch?v=9sPRhdnVSd4 (video in German language) or elsewhere). Wouldn’t a 27B model produce translations of a better quality and still work quite well on a computer with 16 GB RAM?

Activity

  1. changed the title [-]Feature Request[/-] [+]Feature Request – Usage of Bonsai 2 27B as a model[/+] on Oct 1, 2026
  2. kdroidFilter commented on Oct 5, 2026

    @kdroidFilter
    Collaborator

    27b seems very heavy for the majority of users, doesn't it?

  3. kamaqnkatyqn commented on Oct 5, 2026

    @kamaqnkatyqn
    Author

    Bonsai 2 27B can be used offline on a 16 GB laptop.

    The model uses ternary quantization to compress a 27-billion-parameter architecture into a file size of approximately 5.95 GB to 8.60 GB, depending on the specific build (GGUF or MLX) and whether it includes the vision tower. This footprint fits comfortably within the 16 GB of unified memory found on most modern consumer laptops, such as Apple Silicon Macs or PCs with integrated graphics.

    Key considerations for running it offline on 16 GB RAM include:

    Runtime Requirements: The model requires PrismML’s specific fork of llama.cpp or the MLX framework; it is not compatible with standard runtimes like Ollama or stock LM Studio.
    Context Window: While the model weights fit easily, the remaining RAM must accommodate the KV cache. A 16 GB laptop can typically handle moderate context lengths (e.g., 16K–32K tokens) comfortably, though larger contexts (up to the native 262K limit) will consume significant memory.
    Performance: On an Apple M5 Max laptop, the model can generate text at approximately 47 tokens per second using the MLX build, demonstrating viable local inference speeds.

    So, why not give it a try?

  4. kdroidFilter commented on Oct 5, 2026

    @kdroidFilter
    Collaborator

    That would require adding a backend on top of LiteRT; if you want to implement it, I might be interested, but for now, I prefer to keep things simple.

  5. kamaqnkatyqn commented on Oct 5, 2026

    @kamaqnkatyqn
    Author

    OK, that would be too complicated for me right now, too. Still, I'm looking for an even more powerful model that would still run on a laptop with 16 GB of RAM. How about, for example, the Hy-MT2-7B with a quantization of Q4_K_M? Would that be easy for you to implement without requiring any additional backend work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions