--- license: other license_name: mixed-mit-modified-mit base_model: baseten/GLM-5.2-Vision-NVFP4 pipeline_tag: image-text-to-text tags: - gguf - llama.cpp - glm - glm-5.2 - multimodal - vision - image-text-to-text - moonvit --- # GLM-5.2 Vision mmproj for llama.cpp ![Architecture](assets/01-architecture.png) This repository makes the existing [`unsloth/GLM-5.2-GGUF`](https://huggingface.co/unsloth/GLM-5.2-GGUF) `UD-Q3_K_XL` text model accept image input through llama.cpp. It does **not** modify, retrain, or include the 3-bit text weights. Vision is attached at runtime through a separate F16 GGUF containing the MoonViT vision tower and trained PatchMerger projector from [`baseten/GLM-5.2-Vision-NVFP4`](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4). This is an unofficial conversion and llama.cpp integration, not a newly trained checkpoint. Baseten, Z.ai, Moonshot AI, Unsloth, and the llama.cpp project were not involved in this conversion. ## Files | File | Description | |---|---| | `mmproj-GLM-5.2-Vision-f16.gguf` | 943 MB F16 MoonViT vision tower + PatchMerger projector | | `glm5v-mmproj-vision.patch` | Required llama.cpp support for `PROJECTOR_TYPE_GLM5V` | | `SHA256SUMS` | Checksums for the release files | | `eval/` | Programmatic cases, raw outputs, agent outputs, and diagnosis | | `LICENSES/` | Upstream license texts and notices | The text GGUF is intentionally not duplicated here. Download `UD-Q3_K_XL/GLM-5.2-UD-Q3_K_XL-00001-of-00009.gguf` and its split shards from Unsloth. ## Compatibility The patch was tested against llama.cpp commit: ```text ebd048fc5e4b43ec4e0b4abe0b9bf66e1724dad0 ``` Stock llama.cpp at that commit does not recognize `PROJECTOR_TYPE_GLM5V`. Apply the included patch before building: ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp git checkout ebd048fc5e4b43ec4e0b4abe0b9bf66e1724dad0 git apply /path/to/glm5v-mmproj-vision.patch cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release cmake --build build-cuda --target llama-server -j ``` The patch adds 44 lines across six files. It reuses llama.cpp's existing Kimi-K2.5 MoonViT graph, but assigns a separate projector type so mtmd emits GLM's image delimiters: ```text <|begin_of_image|> <|end_of_image|> ``` Kimi's `<|media_begin|>` and `<|media_end|>` tokens are not present in the GLM-5.2 vocabulary. ## Serving ```bash ./build-cuda/bin/llama-server \ --model /path/to/GLM-5.2-UD-Q3_K_XL-00001-of-00009.gguf \ --mmproj /path/to/mmproj-GLM-5.2-Vision-f16.gguf \ --alias glm-5.2 \ --host 0.0.0.0 \ --port 8006 \ --n-gpu-layers 999 \ --ctx-size 524288 \ --parallel 1 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --flash-attn on \ --jinja ``` The tested deployment used 8x RTX 6000 Ada GPUs, a 524,288-token server context, one slot, and the Unsloth `UD-Q3_K_XL` text GGUF. Example OpenAI-compatible request: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8006/v1", api_key="local") response = client.chat.completions.create( model="glm-5.2", messages=[{ "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://example.com/screenshot.png" }, }, {"type": "text", "text": "What is wrong with this screenshot?"}, ], }], temperature=0, max_tokens=1024, ) print(response.choices[0].message.content) ``` ## Evaluation ![Evaluation table](assets/02-eval-table.png) The included exploratory evaluation uses four programmatically generated image cases and programmatic ground truth; it does not use a judge model. Pi and Kilo were each repeated three times. | Case | Raw API | Pi | Kilo | |---|---:|---:|---:| | Code OCR + bug identification | 1.000 | 1.000 | 0.667 | | UI read | 1.000 | 1.000 | 1.000 | | Shape count + colors | 0.833 | 0.833 | 0.778 | | Unlabelled chart read | 0.143 | 0.619 | 0.476 | | **Overall** | **0.744** | **0.863** | **0.730** | One Kilo code run returned empty at a 200-second timeout and is counted as zero. The stored server evidence does not establish whether that event was a client, harness, model, or interaction failure, so no narrower attribution is made here. The labelled-chart diagnostic returned all five printed values exactly. The unlabelled-axis case remained weak, suggesting that magnitude estimation from bar height/gridlines is more difficult than OCR and label/value binding in this small test. This is an exploratory four-case suite, not a general vision benchmark. ![Code OCR example](assets/03-code-ocr.png) Raw artifacts are preserved under `eval/`. Note that the programmatic OCR score checks required identifiers and bug terms; the exact-transcription claim in the example above was separately verified from the stored response. ## Limitations - **Images only. Video is unsupported.** The source checkpoint does not ship the temporal position embedding needed for correct video inference. - This requires the included out-of-tree llama.cpp patch until equivalent support is available upstream. - The projector was not retrained for the Q3 text backbone. - The official Baseten reference uses NVFP4 text weights and Blackwell GPUs. No controlled BF16-vs-Q3 or NVFP4-vs-Q3 vision-quality comparison was performed. The included evaluation cannot establish quantization damage. - Fine-grained counting and axis-based magnitude estimation can be unreliable. - The provided context and GPU settings are the tested configuration, not minimum hardware requirements. ## Provenance The converted source components came from `baseten/GLM-5.2-Vision-NVFP4` revision: ```text f6eab6117386a0c69152fdf272dc65bfd0254f9f ``` That release states that the text backbone and MoonViT tower are frozen and that only the 49.5M-parameter PatchMerger projector was trained. This repository contains no GLM text weights. Related upstream sources: - GLM-5.2 text model: [`zai-org/GLM-5.2`](https://huggingface.co/zai-org/GLM-5.2) - Q3 text GGUF: [`unsloth/GLM-5.2-GGUF`](https://huggingface.co/unsloth/GLM-5.2-GGUF) - Vision release: [`baseten/GLM-5.2-Vision-NVFP4`](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4) - MoonViT source model: [`moonshotai/Kimi-K2.6`](https://huggingface.co/moonshotai/Kimi-K2.6) - Inference runtime: [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) ## License This bundle carries mixed upstream terms: - Baseten states that the trained projector is released under the MIT License. - The redistributed MoonViT tower comes from Kimi-K2.6 under the Modified MIT License. - The included llama.cpp patch is distributed under llama.cpp's MIT terms. Kimi-K2.6's Modified MIT License requires products or services with more than 100 million monthly active users, or more than USD 20 million in monthly revenue, to prominently display `Kimi K2.6` in their user interface. See `LICENSES/` and `THIRD_PARTY_NOTICES.md`. Redistributed upstream weights remain subject to their original terms. This summary is practical guidance, not legal advice.