---
license: mit
datasets:
- ai4bharat/sangraha
language:
- hi
- en
metrics:
- perplexity
base_model:
- FacebookAI/xlm-roberta-base
pipeline_tag: fill-mask
tags:
- hindi
- xlm-roberta
- knowledge-distillation
- masked-language-modeling
- model-compression
- edge-deployment
library_name: transformers
---
# Hindi XLM-RoBERTa 33M (Distilled)
## 📋 Model Description
> This is also an experimentation / learning for me. Maybe not suitable for production use.
This is the **first distilled version** of XLM-RoBERTa-base, specifically optimized for Hindi language understanding. The model achieves **~8x compression** (from 270M to 33M parameters) through knowledge distillation while retaining competitive performance on masked language modeling tasks.
We are currently working further to reduce the vocabulary and still hoping that performance is good.
> ⚠️ **Note**: This is an initial release. We are actively working on:
> - Further compression techniques
> - Extensive evaluation across downstream tasks
> - Benchmarking on edge devices
> - Optimizations for real-world deployment
This research aims to pave the way for effectively distilling monolingual models from multilingual ones, enabling efficient deployment on resource-constrained edge devices.
## 🎯 Model Details
- **Model Type**: Masked Language Model (MLM)
- **Architecture**: XLM-RoBERTa (Distilled)
- **Parameters**: 33M (~12% of original XLM-RoBERTa-base)
- **Teacher Model**: XLM-RoBERTa-base (270M parameters)
- **Training Method**: Knowledge Distillation (KL Divergence + Cross-Entropy) $\to$ Simple is best.
- **Primary Language**: Hindi (हिन्दी)
- **Secondary Language**: English (limited support)
- **Vocabulary Size**: 250,002 (inherited from XLM-RoBERTa) to be reduced.
- **Max Sequence Length**: 512 tokens
- **Embedding Dimension**: 384
- **Hidden Layers**: 6
- **Attention Heads**: 6
## 📊 Performance
| Metric | Value | Teacher (XLM-R-base) |
|--------|-------|----------------------|
| **Perplexity** | 4.87 | ~3.2 |
| **Model Size** | 33M | 270M |
| **Compression Ratio** | 8x | 1x |
| **Inference Speed** | ~8x faster | baseline |
| **Performance Retention** | ~90% | 100% |
> 📌 **Note**: Comprehensive downstream task evaluations (NER, classification, QA) are in progress and will be updated soon.
## 🚀 Usage
### Installation
```bash
pip install transformers torch
```
### Quick Start - Fill Mask
```python
from transformers import pipeline
# Load the fill-mask pipeline
fill_mask = pipeline("fill-mask", model="kkkamur07/hindi-xlm-roberta-33M")
# Hindi example
text = "भारत की राजधानी है।"
predictions = fill_mask(text)
for pred in predictions:
print(f"{pred['token_str']}: {pred['score']:.4f}")
```
### Feature Extraction
```python
# Use for feature extraction in downstream tasks
from transformers import AutoModel
model = AutoModel.from_pretrained("kkkamur07/hindi-xlm-roberta-33M")
inputs = tokenizer("यह एक वाक्य है।", return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
embeddings = outputs.last_hidden_state # Shape: [batch_size, seq_len, 384]
```
## 🎓 Training Details
### Training Data
- **Primary Dataset**: [AI4Bharat Sangraha](https://huggingface.co/datasets/ai4bharat/sangraha) (Hindi subset)
- **Data Size**: ~1M sentences around 100M sentences
- **Preprocessing**: Sentence tokenization, MLM masking (15% probability)
### Training Configuration
- **Distillation Loss**: α × KL_Divergence + (1-α) × Cross_Entropy
- **Alpha (α)**: 0.65
- **Temperature**: 2.5
- **Optimizer**: AdamW (lr=5e-5, weight_decay=0.01)
- **Batch Size**: 32
- **Training Steps**: ~50,000
- **Hardware**: 1x NVIDIA GPU 4090
- **Training Time**: ~12 hours
### Architectural Differences from Teacher
```
XLM-RoBERTa-base (270M) → Distilled (33M)
├── Layers: 12 → 6 layers (-50%)
├── Hidden Size: 768 → 384 (-50%)
├── Attention Heads: 12 → 6 (-50%)
├── Intermediate Size: 3072 → 1536 (-50%)
└── Vocab Size: 250,002 → 250,002 (same)
```
## 🔬 Research Motivation
This model is part of ongoing research to:
1. **Enable Multilingual-to-Monolingual Distillation**: Explore efficient methods to extract language-specific knowledge from large multilingual models
2. **Edge Deployment**: Create models suitable for deployment on mobile devices, IoT, and edge servers
3. **Resource Efficiency**: Reduce computational requirements while maintaining competitive performance
4. **Democratize NLP**: Make advanced Hindi NLP accessible to researchers and developers with limited resources
## 🔮 Roadmap & Future Work
We are actively working on:
- [ ] **Further Compression**: Quantization (INT8, INT4) and pruning techniques
- [ ] **Comprehensive Evaluation**: Benchmarking on XTREME, IndicGLUE, and custom Hindi tasks
- [ ] **Downstream Fine-tuning**: NER, sentiment analysis, text classification
- [ ] **Edge Optimization**: ONNX conversion, TensorFlow Lite, mobile deployment
- [ ] **Multilingual Variants**: Extend to other Indic languages (Tamil, Telugu, Bengali)
- [ ] **Ablation Studies**: Analysis of compression-performance tradeoffs
```bibtex
@misc{hindi_xlm_roberta_distilled_33m,
author = {Your Name},
title = {Hindi XLM-RoBERTa 33M: Knowledge Distillation for Efficient Hindi Language Modeling},
year = {2024},
publisher = {HuggingFace},
howpublished = {\url{https://huggingface.co/kkkamur07/hindi-xlm-roberta-33M}},
note = {First distilled version - Research in progress}
}
```
Also cite the original XLM-RoBERTa paper:
```bibtex
@article{conneau2019unsupervised,
title={Unsupervised Cross-lingual Representation Learning at Scale},
author={Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm{\'a}n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin},
journal={arXiv preprint arXiv:1911.02116},
year={2019}
}
```
## 🙏 Acknowledgments
- **XLM-RoBERTa Team** at Meta AI for the original model
- **AI4Bharat** for the Sangraha dataset
- HuggingFace 🤗 for the Transformers library and model hosting
---
🚧 Work in Progress 🚧
This model represents our first step towards efficient monolingual distillation from multilingual models.
Comprehensive evaluations and further optimizations are underway.
🤗 More Models •
💻 GitHub •
📧 Contact
