unofficial-jona/MedGemma-27B-IT-GGUF

MedGemma 27B IT — GGUF Q4_K_M

This repository contains a 4-bit GGUF quantization of Google's MedGemma 27B IT, created for local inference with llama.cpp and compatible applications such as LM Studio.

It is a multimodal model. Download both files to use image inputs:

File	Purpose	Approximate disk size
MedGemma-27B-IT-Q4_K_M.gguf	Main language model, 4-bit Q4_K_M quantization	~16 GB
mmproj-MedGemma-27B-IT-F16.gguf	Medical vision encoder/projector (kept at F16)	~1 GB

Q4_K_M is a quality-oriented 4-bit quantization. It substantially reduces memory use relative to the original BF16 checkpoint while preserving the model's instruction-following capability. This is a lossy conversion; validate outputs for your intended task.

Requirements

Use a recent build of llama.cpp with Gemma 3 multimodal support.

Workload	Practical hardware guidance
Text-only inference	20 GB+ VRAM is a practical starting point for full GPU offload at modest context; 16 GB GPUs need partial CPU offload
Text + image inference	24 GB+ VRAM recommended; a 20 GB GPU can run modest contexts, while 16 GB GPUs need partial CPU offload
Long context	Extra memory is required for the KV cache; begin at 4K context on a 20 GB GPU and increase only after measuring memory use

Actual memory use depends on context size, batch size, GPU offload layers, and runtime build. The model weights alone are about 15.4 GiB and the F16 vision projector adds about 0.8 GiB. A 20 GB GPU is a good starting point for 4K context, but does not make the full 128K-token capability practical without substantial CPU/RAM offload.

Download

hf download unofficial-jona/MedGemma-27B-IT-GGUF \
  MedGemma-27B-IT-Q4_K_M.gguf mmproj-MedGemma-27B-IT-F16.gguf \
  --local-dir ./MedGemma-27B-IT-GGUF

Use with llama.cpp

Text-only

llama-cli -m ./MedGemma-27B-IT-Q4_K_M.gguf \
  -ngl 999 -c 8192 \
  -p "<start_of_turn>user\nExplain the difference between bacterial and viral pneumonia for a clinician.<end_of_turn>\n<start_of_turn>model\n"

Text and image

llama-mtmd-cli -m ./MedGemma-27B-IT-Q4_K_M.gguf \
  --mmproj ./mmproj-MedGemma-27B-IT-F16.gguf \
  --image ./image.png -ngl 999 -c 8192 \
  -p "<start_of_turn>user\nDescribe the image cautiously and list any uncertainty.<end_of_turn>\n<start_of_turn>model\n"

Use the chat template supplied by your runtime when it offers a Gemma 3 / MedGemma template. Command-line option names can differ slightly across llama.cpp releases; run the executable with --help to confirm the multimodal image flag.

Important medical-use notice

MedGemma is a research/developer model, not a medical device. It can make incorrect, incomplete, biased, or unsafe statements. Do not use it as the sole basis for diagnosis, triage, treatment, or other clinical decisions. Keep qualified clinical review and appropriate safeguards in any healthcare workflow.

Provenance and license

• Base model: google/medgemma-27b-it
• Quantization: llama.cpp Q4_K_M (main model); F16 vision projector
• Original author: Google
• Original license and access terms: Health AI Developer Foundations terms of use

This conversion does not change the original model's license, usage terms, or safety limitations. Access to the base model requires agreeing to its original terms on Hugging Face.

Citation

@article{sellergren2025medgemma,
  title={MedGemma Technical Report},
  author={Sellergren, Andrew and others},
  journal={arXiv preprint arXiv:2507.05201},
  year={2025}
}