sulabhkatiyar/english-to-indic-translate-gemma270m

English → Indian Language Translation (gemma-3-270m-it fine-tune)

Fine-tuned google/gemma-3-270m-it (268M params) for English → Indian language translation across 11 languages.

This is a lightweight translation model targeting short-to-medium text (up to ~512 tokens). It was trained on the ai4bharat/BPCC parallel corpus using sequence packing with document-level attention masking.

Supported Languages

Code	Language	Script
hi	Hindi	Devanagari
bn	Bengali	Bengali
ta	Tamil	Tamil
te	Telugu	Telugu
mr	Marathi	Devanagari
gu	Gujarati	Gujarati
kn	Kannada	Kannada
ml	Malayalam	Malayalam
pa	Punjabi	Gurmukhi
or	Odia	Odia
as	Assamese	Bengali

Prompt Format

Uses the Gemma IT chat template. The prompt format must match what was used during training:

<start_of_turn>user
Translate English to {Language}:
{source_text}<end_of_turn>
<start_of_turn>model

Where {Language} is the full language name (e.g. "Hindi", "Bengali", "Tamil") — not the ISO code.

Training Details

Data

• Dataset: ai4bharat/BPCC (Bharat Parallel Corpus Collection)
• Configs used: ilci, daily, massive, bpcc-seed-latest
• Cap: up to 200,000 examples per language
• Total packed sequences: ~2,559 per epoch (effective training size: 256,000 sequences over 2000 steps)

Architecture

• Model: Gemma3ForCausalLM (gemma3_text)
• Parameters: 268.1M
• Hidden size: 640
• Intermediate size: 2,048
• Layers: 18 (sliding + full attention pattern: 5 sliding → 1 full, repeated 3x)
• Attention heads: 4 (1 KV head, GQA)
• Head dim: 256
• Vocab size: 262,144 (SentencePiece)
• Sliding window: 512 tokens
• Max position embeddings: 32,768

Hyperparameters

Parameter	Value
Per-device batch size	8
Gradient accumulation steps	16
Effective batch size	128
Learning rate	1e-4
LR schedule	Linear with warmup
Warmup ratio	0.1
Weight decay	0.01
Precision	bfloat16
Max sequence length	512
Training steps	2,000 (~0.78 epochs)
Optimizer	AdamW
Seed	42

Sequence Packing

Multiple translation pairs are bin-packed into fixed-length 512-token sequences. Each packed sequence uses:

• Position ID resets at each document boundary (BOS token) to create block-diagonal causal attention masks
• Label masking on prompt tokens — only target (translation) tokens and EOS contribute to loss
• Layout per document: [BOS] [prompt_tokens] [target_tokens] [EOS]

Training Loss

Step	Train Loss	Eval Loss	Learning Rate
1	6.7994	—	0.00e+00
50	4.4387	8.5473	6.38e-06
100	3.4441	8.0905	1.29e-05
250	2.6934	8.5179	3.24e-05
500	2.2552	8.4022	6.50e-05
1000	1.7838	8.1475	9.67e-05
1500	1.5401	8.0835	8.94e-05
2000	1.2843	8.2637	8.22e-05

Training loss decreased steadily from 6.80 → 1.28 over 2000 steps. Eval loss is high due to the large vocabulary (262k tokens) and sequence packing evaluation — the model is learning meaningful translations despite the cross-entropy numbers.

Example Outputs

Translations from checkpoint-2000 (greedy decoding, repetition_penalty=1.3):

Language	Input	Output
Hindi	How are you?	आप कैसे हैं ?
Hindi	Please give me a glass of water.	मुझे एक ग्लास पानी दें।
Bengali	How are you?	তোমার কি আছে ?
Tamil	How are you?	நீங்கள் எப்படி இருக்கிறீர்கள் ?
Marathi	The weather is nice today.	असे तर, आज हवामान खूप चांगले आहे .
Gujarati	The weather is nice today.	આજે હવામાન સારું છે .
Malayalam	The weather is nice today.	കാലാവസ്ഥ ഇന്ന് നല്ലതാണ് .

Short, simple sentences translate well. Longer and more complex inputs may produce incomplete or less accurate results — this is expected from a 268M parameter model at a mid-training checkpoint.

Limitations

• Model size: At 268M parameters, this is a very small model. Translation quality varies across languages and degrades on longer or more complex inputs.
• Mid-training checkpoint: This is checkpoint-2000 out of a planned 4000 steps (~0.78 epochs). Further training may improve quality.
• English → Indic only: The model was trained for one-way translation (English to Indian languages). It is not designed for Indic → English or Indic → Indic translation.
• Max input length: Trained with max_seq_len=512; inputs longer than this may produce poor results.
• Script coverage: Relies on the base Gemma tokenizer's coverage of Indic scripts. Some scripts may be underrepresented in the base model's pretraining data.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "sulabhkatiyar/english-indic-gemma-3-270m-it"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Translate English to Hindi:\nThe weather is nice today."}]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
output = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))

Model Files

• model.safetensors (512 MB) — model weights
• config.json — architecture configuration
• generation_config.json — default generation parameters (top_k=64, top_p=0.95)
• Tokenizer files — from google/gemma-3-270m-it (262k vocab SentencePiece)

License

This model is derived from Google's Gemma and is subject to the Gemma Terms of Use.