---
language:
- en
- hi
license: mit
tags:
- translation
- transformer
- pytorch
- from-scratch
- english-hindi
- machine-translation
datasets:
- ai4bharat/samanantar
metrics:
- bleu
- chrf
pipeline_tag: translation
---

# SamanantarSetu-EN-HI 🌉

**"सेतु" (Setu) = Bridge** — an English → Hindi neural machine translation model, built as a **Transformer trained completely from scratch** (no pretrained weights, no fine-tuning shortcuts) on the [AI4Bharat Samanantar](https://huggingface.co/datasets/ai4bharat/samanantar) parallel corpus.

This is a from-first-principles implementation of the original *"Attention Is All You Need"* (Vaswani et al., 2017) transformer-base architecture — encoder, decoder, multi-head attention, positional encodings, label smoothing, Noam learning-rate schedule, beam search — all hand-written in PyTorch.

## Model Details

| | |
|---|---|
| Architecture | Transformer-base (encoder-decoder) |
| Parameters | 93.3M |
| Layers | 6 encoder + 6 decoder |
| d_model | 512 |
| Attention heads | 8 |
| Feed-forward dim | 2048 |
| Vocabulary | 32,000 (joint SentencePiece BPE, English + Hindi) |
| Max sequence length | 256 tokens |
| Training data | ~10M English-Hindi sentence pairs (AI4Bharat Samanantar, `hi` split) |
| Training steps | 200,000 (~12.8 epochs) |
| Optimizer | Adam (β1=0.9, β2=0.98, eps=1e-9) |
| LR schedule | Noam (warmup 4000 steps) |
| Label smoothing | 0.1 |
| Hardware | 4× NVIDIA RTX 6000 Ada (DataParallel) |

## Results

Evaluated on 150 held-out Samanantar test sentences with beam search (beam size 4):

| Metric | Score |
|---|---|
| BLEU (flores200 tokenizer) | **31.49** |
| chrF | **50.08** |

### Sample translations

```
EN : The number of coronavirus cases are growing at an alarming rate in the country.
REF: देश में कोरोना वायरस के मामले तेजी से बढ़ रहे हैं।
HYP: देश में कोरोना वायरस के मामले लगातार बढ़ते जा रहे हैं।

EN : The film will be releasing in Telugu, Hindi and Tamil.
REF: यह फिल्म हिंदी, तमिल और तेलुगू भाषा में एक साथ रिलीज की जाएगी.
HYP: यह फिल्म हिंदी, तमिल और तेलुगू में रिलीज होगी।

EN : They were taken to the civil hospital.
REF: जिन्हें सिविल अस्पताल ले जाया गया।
HYP: उन्हें सिविल अस्पताल लाया गया।
```

## Repository Structure

```
.
├── checkpoint/
│   └── model.pt              # final trained weights (step 200,000)
├── tokenizer/
│   ├── spm_joint.model       # SentencePiece BPE model (joint EN+HI, 32k vocab)
│   └── spm_joint.vocab
└── code/
    ├── config.py              # all hyperparameters
    ├── models/
    │   ├── embedding.py       # token embedding + sinusoidal positional encoding
    │   ├── attention.py       # scaled dot-product & multi-head attention, feed-forward
    │   ├── encoder.py
    │   ├── decoder.py
    │   └── transformer.py     # full encoder-decoder assembly
    ├── dataset/
    │   ├── dataset.py         # data loading + token-bucket batching
    │   └── masks.py           # padding & causal masks
    ├── train.py                # training loop (AMP, checkpointing, Noam LR, wandb)
    ├── train_tokenizer.py      # SentencePiece training script
    ├── translate.py            # beam-search inference
    ├── evaluate.py              # BLEU/chrF evaluation over a split
    ├── infer_and_compare.py     # inference + metrics + example printouts
    └── run_train_loop.sh        # auto-restart wrapper for long unattended training
```

## Usage

```python
import torch
import sentencepiece as spm
import sys
sys.path.insert(0, "code")

import config
from models.transformer import Transformer
from dataset.masks import make_src_mask, make_tgt_mask

# load tokenizer
sp = spm.SentencePieceProcessor()
sp.load("tokenizer/spm_joint.model")

# load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model = Transformer(
    src_vocab_size=sp.get_piece_size(),
    tgt_vocab_size=sp.get_piece_size(),
    d_model=config.D_MODEL,
    n_heads=config.N_HEADS,
    n_encoder_layers=config.N_ENCODER_LAYERS,
    n_decoder_layers=config.N_DECODER_LAYERS,
    d_ff=config.D_FF,
    dropout=config.DROPOUT,
    max_len=config.MAX_LEN,
).to(device)
ckpt = torch.load("checkpoint/model.pt", map_location=device)
model.load_state_dict(ckpt["model"])
model.eval()

# translate — see code/translate.py for the full beam_search() implementation
```

Or simply run the included script:

```bash
python3 code/translate.py --text "How are you today?" --checkpoint checkpoint/model.pt --beam_size 4
```

## Training Details

Trained on the full `hi` split of [AI4Bharat Samanantar](https://huggingface.co/datasets/ai4bharat/samanantar) (~10M sentence pairs after length/ratio filtering), using dynamic token-bucket batching (~20-32k tokens/batch), mixed-precision (AMP), and gradient clipping. Training ran for 200,000 steps (~12.8 epochs) across 4 GPUs with `nn.DataParallel`.

Validation loss / perplexity over training:

| Step | Val Loss | Val PPL |
|---|---|---|
| 20,000 | 3.75 | 42.4 |
| 60,000 | 3.50 | 33.2 |
| 110,000 | 3.47 | 32.3 |
| 150,000 | 2.96 | 19.3 |
| 200,000 | **2.84** | **17.1** |

The model plateaus mid-training (Noam schedule keeps LR relatively high through the middle steps) and improves sharply once the learning rate decays further in the final third of training — classic behavior for this LR schedule.

## Limitations

- Beam search inference in `translate.py` is not KV-cached / not batched across beams — it is correctness-oriented, not optimized for throughput.
- Trained on news/formal-register text (Samanantar is largely mined from news and web sources); informal or code-mixed Hindi may translate less reliably.
- Single-direction (EN→HI) only.

## Citation

If you use this model, please also cite the Samanantar dataset:

```bibtex
@article{ramesh2021samanantar,
  title={Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages},
  author={Ramesh, Gowtham and others},
  journal={Transactions of the Association for Computational Linguistics},
  year={2022}
}
```