SamanantarSetu-EN-HI 🌉

"सेतु" (Setu) = Bridge — an English → Hindi neural machine translation model, built as a Transformer trained completely from scratch (no pretrained weights, no fine-tuning shortcuts) on the AI4Bharat Samanantar parallel corpus.

This is a from-first-principles implementation of the original "Attention Is All You Need" (Vaswani et al., 2017) transformer-base architecture — encoder, decoder, multi-head attention, positional encodings, label smoothing, Noam learning-rate schedule, beam search — all hand-written in PyTorch.

Model Details

Architecture	Transformer-base (encoder-decoder)
Parameters	93.3M
Layers	6 encoder + 6 decoder
d_model	512
Attention heads	8
Feed-forward dim	2048
Vocabulary	32,000 (joint SentencePiece BPE, English + Hindi)
Max sequence length	256 tokens
Training data	~10M English-Hindi sentence pairs (AI4Bharat Samanantar, hi split)
Training steps	200,000 (~12.8 epochs)
Optimizer	Adam (β1=0.9, β2=0.98, eps=1e-9)
LR schedule	Noam (warmup 4000 steps)
Label smoothing	0.1
Hardware	4× NVIDIA RTX 6000 Ada (DataParallel)

Results

Evaluated on 150 held-out Samanantar test sentences with beam search (beam size 4):

Metric	Score
BLEU (flores200 tokenizer)	31.49
chrF	50.08

Sample translations

EN : The number of coronavirus cases are growing at an alarming rate in the country.
REF: देश में कोरोना वायरस के मामले तेजी से बढ़ रहे हैं।
HYP: देश में कोरोना वायरस के मामले लगातार बढ़ते जा रहे हैं।

EN : The film will be releasing in Telugu, Hindi and Tamil.
REF: यह फिल्म हिंदी, तमिल और तेलुगू भाषा में एक साथ रिलीज की जाएगी.
HYP: यह फिल्म हिंदी, तमिल और तेलुगू में रिलीज होगी।

EN : They were taken to the civil hospital.
REF: जिन्हें सिविल अस्पताल ले जाया गया।
HYP: उन्हें सिविल अस्पताल लाया गया।

Repository Structure

.
├── checkpoint/
│   └── model.pt              # final trained weights (step 200,000)
├── tokenizer/
│   ├── spm_joint.model       # SentencePiece BPE model (joint EN+HI, 32k vocab)
│   └── spm_joint.vocab
└── code/
    ├── config.py              # all hyperparameters
    ├── models/
    │   ├── embedding.py       # token embedding + sinusoidal positional encoding
    │   ├── attention.py       # scaled dot-product & multi-head attention, feed-forward
    │   ├── encoder.py
    │   ├── decoder.py
    │   └── transformer.py     # full encoder-decoder assembly
    ├── dataset/
    │   ├── dataset.py         # data loading + token-bucket batching
    │   └── masks.py           # padding & causal masks
    ├── train.py                # training loop (AMP, checkpointing, Noam LR, wandb)
    ├── train_tokenizer.py      # SentencePiece training script
    ├── translate.py            # beam-search inference
    ├── evaluate.py              # BLEU/chrF evaluation over a split
    ├── infer_and_compare.py     # inference + metrics + example printouts
    └── run_train_loop.sh        # auto-restart wrapper for long unattended training

Usage

import torch
import sentencepiece as spm
import sys
sys.path.insert(0, "code")

import config
from models.transformer import Transformer
from dataset.masks import make_src_mask, make_tgt_mask

# load tokenizer
sp = spm.SentencePieceProcessor()
sp.load("tokenizer/spm_joint.model")

# load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model = Transformer(
    src_vocab_size=sp.get_piece_size(),
    tgt_vocab_size=sp.get_piece_size(),
    d_model=config.D_MODEL,
    n_heads=config.N_HEADS,
    n_encoder_layers=config.N_ENCODER_LAYERS,
    n_decoder_layers=config.N_DECODER_LAYERS,
    d_ff=config.D_FF,
    dropout=config.DROPOUT,
    max_len=config.MAX_LEN,
).to(device)
ckpt = torch.load("checkpoint/model.pt", map_location=device)
model.load_state_dict(ckpt["model"])
model.eval()

# translate — see code/translate.py for the full beam_search() implementation

Or simply run the included script:

python3 code/translate.py --text "How are you today?" --checkpoint checkpoint/model.pt --beam_size 4

Training Details

Trained on the full hi split of AI4Bharat Samanantar (10M sentence pairs after length/ratio filtering), using dynamic token-bucket batching (20-32k tokens/batch), mixed-precision (AMP), and gradient clipping. Training ran for 200,000 steps (~12.8 epochs) across 4 GPUs with nn.DataParallel.

Validation loss / perplexity over training:

Step	Val Loss	Val PPL
20,000	3.75	42.4
60,000	3.50	33.2
110,000	3.47	32.3
150,000	2.96	19.3
200,000	2.84	17.1

The model plateaus mid-training (Noam schedule keeps LR relatively high through the middle steps) and improves sharply once the learning rate decays further in the final third of training — classic behavior for this LR schedule.

Limitations

• Beam search inference in translate.py is not KV-cached / not batched across beams — it is correctness-oriented, not optimized for throughput.
• Trained on news/formal-register text (Samanantar is largely mined from news and web sources); informal or code-mixed Hindi may translate less reliably.
• Single-direction (EN→HI) only.

Citation

If you use this model, please also cite the Samanantar dataset:

@article{ramesh2021samanantar,
  title={Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages},
  author={Ramesh, Gowtham and others},
  journal={Transactions of the Association for Computational Linguistics},
  year={2022}
}