Qwen3-VL-8B Distill32 Image-Math LoRA

Release role

This is the primary validated adapter in the accompanying two-adapter research release. It is a bounded teacher-student experiment, not a promoted general-purpose math specialist or a production tutoring model.

Qwen/Qwen3-VL-32B-Thinking generated Korean candidate solutions for a multimodal MMK12 subset. Only answer-verified traces were retained, and a LoRA adapter was trained on Qwen/Qwen3-VL-8B-Thinking with the vision tower frozen.

Model details

Base model: Qwen/Qwen3-VL-8B-Thinking
Base revision: 92f3c4b4feadd3a016ef468d103bb5f58b2a2c6b
Teacher model: Qwen/Qwen3-VL-32B-Thinking
Teacher revision: 7edd10ffd1196091948fb245ff63e406ccb2d4d1
Adapter: PEFT LoRA, rank 32, alpha 64, dropout 0.05
Target modules: q/k/v/o and gate/up/down projections
Vision tower: frozen
Adapter weight SHA-256: d3a64c058fe6a1449b9842a18e49cc4d0f19b59f323723c2d683ac57e82b8a18

Training data and filtering

The source pool was the math subset of FanqingM/MMK12. A deterministic 1,200-row sample was submitted to the teacher. Candidate traces were retained only when the extracted final answer matched the source answer and the trace stayed below the configured length cap.

Accepted rows: 755
Training rows: 675
Validation rows: 80
Images per retained row: exactly 1
Training JSONL SHA-256: e7a1aa80238a1fb8198b4e7c66c8601be5911bce0ce15ded33734f1644909e8d
Validation JSONL SHA-256: 0c35a42e259ac262baa2ba1e953b5138a28eb6f5b4bbf8adb1ef6ad0c6f024a0

Training rows and images are not redistributed. Obtain MMK12 from its source repository and comply with its license and attribution requirements.

Training procedure

Epochs: 2
Learning rate: 1e-4
Precision: BF16
Per-device batch size: 1
Gradient accumulation: 8
Maximum sequence length: 10,240
Maximum image long side: 896 pixels
Seed: 1234
Hardware: 8 GPUs

Evaluation

Base and adapter arms used identical inputs, prompts, greedy decoding, token budgets, scorer, and UID-aligned paired comparisons.

Evaluation set	N	Base	Adapter	Delta	Interpretation
Original AIHub15 head subset	500	65.4%	72.0%	+6.6pp	Leakage-clean bounded positive; family-wise significant
New pool-representative subset	500	54.6%	56.4%	+1.8pp	Directional, not significant; p=0.397
New pool subset	4,000	53.5%	55.475%	+1.975pp	Significant; p=0.002353
Pre-registered combined primary	4,500	53.622%	55.578%	+1.956pp	Significant; p=0.001451; 95% CI [+0.756, +3.156]pp

The pre-registered practical threshold was +2.0pp. The combined point estimate is narrowly below that threshold, while its confidence interval crosses it. The defensible conclusion is a small, statistically significant, pool-level effect whose magnitude is smaller than the original head-subset result.

The adapter used fewer output tokens and reached the token cap less often than Base in the combined evaluation, so the gain was not produced by spending a larger output budget.

Intended use

Reproduce or inspect a bounded teacher-trace distillation adapter.
Study controlled paired evaluation of multimodal LoRA adapters.
Use as one candidate in category- or answer-type-aware model selection research, with Base as a fallback.

Limitations

Do not claim broad Korean math or science improvement.
Do not claim a completed specialist, learned router, deployed ensemble, or production system.
Do not generalize the original +6.6pp head-subset magnitude to the full AIHub15 pool.
MMK12 transfer evaluation included a small overlap caveat and is not clean independent transfer evidence.
Training data were not Korean-native; the teacher was prompted to answer in Korean.
Safety, fairness, and high-stakes educational use were not evaluated.

Loading

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base_id = "Qwen/Qwen3-VL-8B-Thinking"
base_revision = "92f3c4b4feadd3a016ef468d103bb5f58b2a2c6b"
adapter_id = "TakTak/qwen3-vl-8b-distill32-image-math-lora"

processor = AutoProcessor.from_pretrained(base_id, revision=base_revision)
base = AutoModelForImageTextToText.from_pretrained(
    base_id,
    revision=base_revision,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)

License and upstream sources

The adapter is released under CC BY-NC 4.0. The pinned Qwen base and teacher revisions are Apache-2.0. MMK12's pinned source snapshot contains an Apache-2.0 license file. No raw source data, evaluation rows, predictions, or scored row-level outputs are included.