Gemma-3-4B — Urdu Education & Reasoning

Submission to the Adaption Labs AutoScientist Challenge (Urdu language track). This is a Gemma-3-4B model adapted for Urdu. The training data was produced by translating and localising English knowledge corpora into Urdu with Adaption AutoScientist and the Adaptive Data pipeline. The model is evaluated on UrduMMLU, a 26,431-question benchmark written natively in Urdu.

Live demo: https://huggingface.co/spaces/abdullah693/urdu-edu-reasoning

Summary of results

On UrduMMLU (Urdu prompt, zero-shot), the model scores 46.21%, compared with 44.96% for the base gemma-3-4b-it. Accuracy improves in every domain that carries transferable knowledge (STEM, professional, social sciences, general knowledge). The only domain that does not improve is Urdu literature, which is discussed under Limitations.

overall

What this entry validates

The challenge asks whether Adaption's adaptive-data tooling can build useful, locally-relevant models. We tested a specific claim: adapting English knowledge corpora into Urdu with AutoScientist produces measurable gains on a native Urdu benchmark for knowledge that is language-independent. The result supports the claim and also marks its boundary.

validated effect

  • Every transferable domain improved (STEM +5.9, professional +3.6, other +3.6, social sciences +2.7), and the model exceeded its base overall (+1.33).
  • The effect does not extend to language-intrinsic content. Urdu literature declined by 2.5 points, because that knowledge cannot be obtained by translating English sources and requires native Urdu data.

The practical conclusion for low-resource adaptation: AutoScientist-based translation is effective for the science, mathematics, reasoning, and social-knowledge portions of a benchmark, and should be paired with native-language collection for culturally specific content.

Per-domain results

Domain Base Gemma-3-4B This model Change
STEM 46.9 52.8 +5.9
Professional 49.1 52.7 +3.6
Other 42.9 46.5 +3.6
Social sciences 48.7 51.4 +2.7
Urdu literature / Humanities 41.3 38.8 −2.5
Overall 44.96 46.21 +1.33

per-domain

Overall accuracy was measured on the full 26,431-question test set with zero unparsed responses. Per-domain base figures are from a 1,499-question stratified sample.

Method

pipeline

  1. Source assembly. About 40,000 examples were drawn from open English datasets that cover the UrduMMLU subject areas (MMLU, GSM8K, MATH, ARC, AQuA, CommonsenseQA) together with native-Urdu instruction, grammar, and literature data. Training splits only.
  2. Adaptation (Adaptive Data + AutoScientist). English rows were translated and localised into Pakistani Urdu. The adaptation step also produced a reformulated prompt, an answer with explanation, and an English reasoning trace for each row.
  3. Fine-tuning. Gemma-3-4B was supervised-fine-tuned on the adapted set.
  4. Evaluation. Generation with answer parsing on UrduMMLU, Urdu prompt, zero-shot.

Training data: abdullah693/adaption-urdu-edu-cultural-reasoning (~39,913 examples).

data composition

Reproducibility

  • Harness validation. Our evaluation reproduces the base gemma-3-4b-it at 44.96%, within 0.1 point of the 44.88% reported in the UrduMMLU paper, which confirms the protocol matches.
  • Contamination. UrduMMLU is the held-out benchmark and was not used in training.
  • Protocol. Urdu prompt template from the UrduMMLU repository, generation plus answer parsing, zero-shot, full test set.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
mid = "abdullah693/gemma-3-4b-it-urdu-edu-reasoning"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "نظامِ شمسی میں کتنے سیارے ہیں؟"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0][ids.shape[1]:], skip_special_tokens=True))

Limitations

  • Urdu literature and other language-intrinsic content are weaker than the base model; native Urdu literary data is required to address this.
  • At 4 billion parameters the model has limited factual depth and can produce incorrect answers, particularly on long-tail facts.
  • It inherits the biases and knowledge cutoff of Gemma-3.
  • Intended for research and education. Not a reliable source for examinations, religious rulings, or legal and medical advice.

Citation

@misc{gemma3_4b_urdu_edu_2026,
  title  = {Gemma-3-4B Urdu Education and Reasoning},
  author = {abdullah693},
  year   = {2026},
  note   = {Adapted with Adaption AutoScientist; evaluated on UrduMMLU},
  url    = {https://huggingface.co/abdullah693/gemma-3-4b-it-urdu-edu-reasoning}
}
Downloads last month
73
Safetensors
Model size
4B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for abdullah693/gemma-3-4b-it-urdu-edu-reasoning

Finetuned
(730)
this model

Dataset used to train abdullah693/gemma-3-4b-it-urdu-edu-reasoning

Space using abdullah693/gemma-3-4b-it-urdu-edu-reasoning 1

Evaluation results

  • UrduMMLU accuracy (Urdu, zero-shot) on UrduMMLU
    self-reported
    46.210