Instructions to use CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct")
model = AutoModelForMultimodalLM.from_pretrained("CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct

SGLang

How to use CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct with Docker Model Runner:
```
docker model run hf.co/CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct
```

MANGO-Qwen3-Omni-30B-A3B-Instruct

MANGO (Multimodal Adaptation for thaNGuage Optimization — vision+text SFT variant) is a Thai-language fine-tune of Qwen/Qwen3-Omni-30B-A3B-Instruct, produced via LoRA supervised fine-tuning on a mixed vision-text + text-only corpus (~107.8K samples, 59% Thai / 41% English).

This checkpoint is Approach 1 (vision+text SFT) from the paper "Vision-Text Joint Fine-Tuning Improves Thai Language Proficiency in Omni Vision-Language Models" (CMKL University, 2026). It is the empirical answer to a practical question: when fine-tuning an omni-modal VLM for a low-resource language, does including vision-text pairs help or hurt text-only language performance? We find it helps — this model outperforms a text-only-SFT counterpart on Thai NLP benchmarks by +3.1% (normalized average) while remaining on par on vision-language benchmarks.

Model Details

Base model: Qwen/Qwen3-Omni-30B-A3B-Instruct — a mixture-of-experts omni-modal model, 30B total parameters (~3B activated per forward pass), with a ViT encoder, a projector aligner, and a sparse-MoE language backbone.
Fine-tuning method: LoRA (rank 8, alpha 16, target_modules=all-linear), applied only to the language backbone. The ViT encoder and the vision-language aligner are frozen, so all reported gains/losses are attributable to language-backbone adaptation, not visual feature drift.
Training data: ~107,800 samples (10% subsample of a ~1.08M-sample pool), spanning English text (Nemotron Science/Instruction-Following/Competitive-Programming/Math), Thai text (Thai Wikipedia multiturn), and vision+text pairs (Cambrian-1 EN/TH, COCO-IPU Thai captions, Thai OCR document data). See CMKL/mango-sft-datasets for the full composition.
Modality: text and image input, text output. Audio/video input capability of the base model is not fine-tuned or evaluated here.
Languages: Thai, English.
License: inherits the license of the base model. See Qwen/Qwen3-Omni-30B-A3B-Instruct for the exact terms before verifying and setting the license metadata field on this repo.

A companion checkpoint trained on the text-only subset of the same pool (~9,900 samples) is evaluated in the paper as Approach 2, but is not the model in this repository.

Intended Use

This model is intended for:

Thai-language chat, reasoning, exam-style QA, summarization, and translation.
Thai and English vision-language tasks (visual QA, captioning-adjacent understanding, chart/diagram/math reasoning) inherited from the strong Qwen3-Omni base and improved further by SFT.

It is a research artifact released for reproducibility, not a production-hardened assistant. It has not been safety-tuned beyond what the base model provides, and it inherits the base model's general limitations (hallucination, non-factual outputs, sensitivity to prompt phrasing).

How to Use

Load with the Transformers Qwen3Omni (or equivalent) pipeline, following the same interface as the base model:

from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {"role": "user", "content": [
        {"type": "text", "text": "อธิบายภาพนี้เป็นภาษาไทย"},
        {"type": "image", "image": "path/to/image.jpg"},
    ]}
]

inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(output[0], skip_special_tokens=True))

For text-only inference, omit the image content and use the same chat-template interface as Qwen/Qwen3-Omni-30B-A3B-Instruct.

Since this checkpoint was trained as a LoRA adapter merged onto the base weights, it can also be loaded as a standalone full checkpoint or, if released in adapter-only form, applied on top of Qwen/Qwen3-Omni-30B-A3B-Instruct via peft.

Training Details


Framework	ms-swift v3.12.5 + Megatron-LM backend
Hardware	2 nodes × 8× NVIDIA A100 40GB (16 GPUs, APEX supercomputer, CMKL University)
Parallelism	Tensor parallel = 2, Pipeline parallel = 2, Expert parallel = 8
LoRA config	rank 8, alpha 16, `target_modules=all-linear`, frozen ViT + aligner
Optimizer	LR 1e-4, 5% linear warmup, min LR 1e-5
Batch size	global 256, micro-batch 16/device, sequence packing enabled
Sequence length	max 2,048 tokens
Epochs	3
Precision / kernels	Flash Attention, MoE grouped GEMM, shared-expert overlap, MoE permutation fusion, full uniform gradient checkpointing

Full training command and configuration: Mango-Recipe (train/train.sh, multi-node launchers submit1.sh/submit2.sh, SLURM variant slurm_submit.sh).

Evaluation

Evaluated against the untuned base model and a text-only-SFT counterpart on:

9 Thai NLP benchmarks (ThaiLLM-Leaderboard protocol): M3Exam, Thai-Exam, FLORES (avg / en→th / th→en), iApp QA, XLSum, Belebele, Wisesight Sentiment, XCOPA, XNLI.
15 vision-language benchmarks in English and Thai: MMBench, MMT-Bench, MMStar, SeedBench, MathVerse-MINI, MathVista-MINI, ScienceQA, MMBench-95CI (TH), and TGAT (Thai General Aptitude Test, TH).

Thai NLP benchmarks (normalized average)

Model	Norm. Avg ↑
Qwen3-Omni-30B-A3B-Instruct (baseline)	0.379
Text-only SFT	0.435
MANGO (vision+text SFT, this model)	0.467

Largest gains vs. text-only SFT: XCOPA +10.0%, Thai-Exam +10.7%. NLG tasks (translation, summarization) are near-parity across both SFT variants.

Vision-language benchmarks (overall)

Model	EN	TH	EN+TH
Baseline	0.735	0.699	0.694
Text-only SFT	0.765	0.733	0.722
MANGO (vision+text SFT, this model)	0.762	0.732	0.720

Both SFT variants substantially exceed the baseline; the gap between them is within noise (≤0.3%). See the paper for a full per-benchmark breakdown, a failure-mode analysis on TGAT (Thai visual/spatial reasoning) and Wisesight (Thai sentiment), and task-conditioned recommendations for practitioners.

Limitations

Data volume confound: this checkpoint is trained on ~107.8K samples vs. ~9.9K for the text-only comparison model — an 11× difference — so part of the observed gap may reflect training volume rather than modality alone. See the paper's discussion for details.
TGAT regression: this model regresses on TGAT (a Thai visual/spatial reasoning benchmark drawn from real exam papers) relative to the untuned baseline, more so than the text-only variant. Neither SFT corpus contains abstract spatial-reasoning examples (3D rotation, box folding, symbol-matrix analogies), so this appears to be catastrophic forgetting of pretraining priors rather than a general capability loss.
Wisesight (sentiment) sensitivity: SFT can shift the model toward a formal/neutral register; this model partially mitigates that (thanks to informal-register Thai captions in its corpus) but still trails the untuned baseline on this benchmark.
Not intended for safety-critical deployment without additional evaluation.

Citation

If you use this model, please cite both this work and the base model it fine-tunes:

@article{cmkl2026mango,
  title   = {Vision-Text Joint Fine-Tuning Improves Thai Language Proficiency in Omni Vision-Language Models},
  author  = {{CMKL University}},
  year    = {2026},
  note    = {MANGO: Qwen3-Omni-30B-A3B-Instruct fine-tuned for Thai language proficiency}
}

@article{qwen3omni2025,
  title   = {Qwen3-Omni Technical Report},
  author  = {{Qwen Team}},
  year    = {2025},
  howpublished = {\url{https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct}}
}

Contributor

Atikan Payungthong (atikan@cmkl.ac.th)
Takdanai Kreangphet (tkreang@cmkl.ac.th)
Paksaran Kongkaew (pkongkae@cmkl.ac.th)
Jarukit Suchat (jarukit@cmkl.ac.th)
Dr.Akkarit Sangpetch (akkarit@cmkl.ac.th)

Related Artifacts

Reproducibility code: CMKL-MMFM/mango-recipe
Training dataset: CMKL/mango-sft-datasets
Base model: Qwen/Qwen3-Omni-30B-A3B-Instruct
Thai visual reasoning benchmark used in evaluation: CMKL/TGAT

Downloads last month: 19

Safetensors

Model size

35B params

Tensor type

BF16

Model tree for CMKL/MANGO-Qwen3-Omni-30B-A3B-Instruct

Base model

Qwen/Qwen3-Omni-30B-A3B-Instruct

Finetuned

(31)

this model

Adapters

2 models