model: support GLM4V vision encoder

[ngxson]

On first look, it seems to be an easy model to support as the HF implementation is pretty much the same as Qwen2.5VL

However, there are some very subtle differences that even some LLMs will miss (I tried both Grok and Gemini 3 and they both missed the 2 first points):

1. For text model, M-RoPE ordering is non-Neox. And because ggml's M-RoPE uses the Neox ordering by default, we need to convert the weight to neox ordering upon conversion. This is by far the most complicated change to support this model
2. Learned position embedding interpolation uses bicubic instead of bilinear
3. Added a norm layer right after patch bias
4. Use RMS norm

The embedding output was tested against HF transformers and confirmed to be matched

Important

RoPE ordering was corrected upon conversion - no more backend changes here in this PR

Testing

https://huggingface.co/zai-org/GLM-4.6V-Flash

I'm using the ./tools/mtmd/test-1.jpeg already included in this repo:

llama-mtmd-cli -m ..... -mm ..... --image ./tools/mtmd/test-1.jpeg -p "extract all texts from this image" --temp 0 -n 1024

Output:

"All the News That's Fit to Print"

Then the newspaper title: "The New York Times"

"LATE CITY EDITION"

"VOL. CXLVIII, No. 40,711"

"NEW YORK, MONDAY, JULY 21, 1969"

"10 CENTS"

Then the main headline: "MEN WALK ON MOON"

Next: "ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG"

Then a section: "Voice From Moon: 'Eagle Has Landed'"

Then the article by John Noble Wilford: "A Powdery Surface Is Closely Explored"

Now, let's transcribe each part carefully, including smaller text.

First, the top left box:

"All the News  
That's Fit to Print"

Then the newspaper header:

"The New York Times"

"LATE CITY EDITION"

"VOL. CXLVIII, No. 40,711"

"NEW YORK, MONDAY, JULY 21, 1969"

"10 CENTS"

[ngxson]

@tarruda just tested, it should work with the latest commit (feel free to give it a try)

[tarruda]

@tarruda just tested, it should work with the latest commit (feel free to give it a try)

Will do. Did you publish any GGUF weights?

[ngxson]

Because the 2 models (vision and non-vision) are mostly the same, except for the rope mode, so I was quite lazy not to duplicate it into a new arch (which adds involves quite a lot of copy-paste code)

I hope that we can somewhat allow de-duplicating some code via #18051

In the meantime, lmk if you're OK with keeping this hack, or a new arch is still preferable @ggerganov @CISC

[ngxson]

Will do. Did you publish any GGUF weights?

No because there is a chance we will change the arch name

[IIIIIllllIIIIIlllll]

@tarruda just tested, it should work with the latest commit (feel free to give it a try)

Will do. Did you publish any GGUF weights?

Sorry for the interruption.

I have quantized a Q4 model, but the current PR does not yet support the vision module.

Edited: 2025-12-16 10:37
Sorry, I have too many local branches and I got confused.

[CISC]

Sorry for the interruption.

I have quantized a Q4 model, but the current PR does not yet support the vision module.

Yes it does?

[IIIIIllllIIIIIlllll]

it work well (the 106B one) :)

[tarruda]

I tried https://huggingface.co/ggml-org/GLM-4.6V-GGUF with the annotate test page and it seems to work well.

While it is not as good as Qwen3-VL 32B, it seems to have good bounding box capabilities:

Note that I used Q8_0 mmproj, maybe with F16 or F32 it will improve?

[BetaDoggo]

The bf16 or fp16 mmproj should definitely be used whenever possible, mmproj degrades much faster than regular llm weights, and and 2GB is nothing when the regular model is already 70GB.