xlm-roberta-base-nepali-sentiment-v1

Fine-tuned XLM-RoBERTa Base for 3-class sentiment analysis on Nepali text across all three scripts: Devanagari (देवनागरी), Romanized Nepali, and code-mixed Nepali-English.

This is currently one of the largest open fine-tunes for Nepali sentiment analysis, trained on a merged corpus of ~125k samples from 7 public datasets. The model handles real-world Nepali social media text including emoji, slang, and script-switching within a single sentence.

---

Labels

ID	Label	Description
0	negative	Negative sentiment
1	neutral	Neutral / factual
2	positive	Positive sentiment

---

Quick Start

from transformers import pipeline

pipe = pipeline(
    "text-classification",
    model="YOUR_HF_USERNAME/xlm-roberta-base-nepali-sentiment-v1",
    tokenizer="YOUR_HF_USERNAME/xlm-roberta-base-nepali-sentiment-v1",
)

# Devanagari
pipe("यो चलचित्र साँच्चै राम्रो छ!")
# [{'label': 'positive', 'score': 0.92}]

# Romanized Nepali
pipe("yo movie ekdam ramro thiyo bro")
# [{'label': 'positive', 'score': 0.88}]

# Code-mixed
pipe("Movie chai ramro thiyo but ending weak lagyo 😕")
# [{'label': 'negative', 'score': 0.71}]

# Devanagari negative
pipe("सेवा एकदमै खराब थियो, फेरि कहिल्यै आउँदिन।")
# [{'label': 'negative', 'score': 0.89}]

With explicit probabilities

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch.nn.functional as F

model_id = "YOUR_HF_USERNAME/xlm-roberta-base-nepali-sentiment-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

id2label = {0: "negative", 1: "neutral", 2: "positive"}

def predict(text: str):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
    with torch.no_grad():
        logits = model(**inputs).logits
    probs = F.softmax(logits, dim=-1).squeeze()
    for i, p in enumerate(probs):
        print(f"  {id2label[i]}: {p:.3f}")

predict("सरकारले राम्रो काम गरेको छ।")

---

Training Data

The model was trained on a merged dataset built from 7 publicly available Nepali sentiment corpora. All datasets were fetched programmatically without manual downloads.

Sources

#	Dataset	Source	Script	Est. Size	Labels
1	Sagar32/NepaliDevanagariSentimentAnalysis	HuggingFace	Devanagari	~92k	pos / neg / neu
2	Titung/nepali-sentiment	HuggingFace	Devanagari	~4.8k	pos / neg / neu
3	Shushant/NepaliSentiment	HuggingFace	Devanagari	~3–5k	pos / neg
4	erabhash/romanized-nepali-sentiment-analysis-dataset	Kaggle	Romanized	~3k	pos / neg / neu
5	shikharghimire/nepali-language-sentiment-analysis-movie-reviews	Kaggle	Devanagari	~2.5k	star ratings → mapped
6	suprapandey/nepali-luxury-hotel-reviews-2024	Kaggle	Devanagari	~4k	star ratings → mapped
7	cognivex/nepali-english-cs-sentiment	Kaggle	Code-mixed	~2–3k	pos / neg

Total after merge and deduplication: ~125,624 samples

Label Unification

All datasets were mapped to a single 3-class schema: {0: negative, 1: neutral, 2: positive}.

Binary datasets (sources 3 and 7) contributed only to the negative (0) and positive (2) classes — no synthetic neutral labels were generated.

Star-rating datasets (sources 5 and 6) were mapped as:

Stars	→ Label
1–2	negative (0)
3	neutral (1)
4–5	positive (2)

Script Type Tagging

Each sample was automatically tagged with a script type using Unicode range detection:

• devanagari — >30% of characters in U+0900–U+097F
• romanized — ASCII-dominant with Nepali lexical markers (e.g., cha, thyo, haru, ramro)
• english — ASCII-dominant, no Nepali markers
• mixed — neither purely Devanagari nor ASCII-dominant

Final Split

Split	Samples
Train	100,499
Validation	12,562
Test	12,563
— test_devanagari	9,294
— test_romanized	1,251
— test_mixed	2,018

Splits were stratified by label class. All per-script test sets are held-out subsets of the same test split.

Class Distribution (Training Set)

Label	Count	%
negative (0)	37,762	37.6%
neutral (1)	9,878	9.8%
positive (2)	52,859	52.6%

Neutral is underrepresented (~5.3× fewer samples than positive). Inverse-frequency class weights were applied during training to partially compensate.

Preprocessing

Only minimal cleaning was applied:

• Unicode NFC normalization (critical for Devanagari combining characters)
• Lowercase
• URL removal
• Duplicate whitespace collapse
• Emojis retained — they carry strong sentiment signal in Nepali social media text

Aggressive cleaning (punctuation stripping, stopword removal) was intentionally avoided. Romanized normalization (xa→cha, xaina→chaina) was not applied to preserve the original signal distribution.

---

Training Procedure

Base Model

xlm-roberta-base — chosen for native Nepali, English, and code-mixed support with a strong multilingual pretraining foundation.

Architecture

XLM-RoBERTa Encoder (12 layers, 768 hidden, 125M params)
        ↓
Dropout (p=0.1, default classifier head)
        ↓
Linear (768 → 3)

Full fine-tune. No LoRA or adapters — dataset size (~100k) and GPU budget (2× NVIDIA T4) made full fine-tuning appropriate.

Hyperparameters

Parameter	Value
max_length	128
per_device_train_batch_size	32
gradient_accumulation_steps	2
Effective batch size	128 (32 × 2 devices × 2 accum)
learning_rate	2e-5
weight_decay	0.01
warmup_ratio	0.1
num_train_epochs	4
fp16	True
lr_scheduler	linear with warmup
Class weights	Inverse-frequency, normalized
Early stopping patience	2 epochs
Best model selection	macro F1 on validation set

Hardware

2× NVIDIA Tesla T4 (16 GB VRAM each), Kaggle notebook environment. Total training time: ~1 hr 40 min.

Training Curve

Epoch	Train Loss	Val Loss	Accuracy	Macro F1
1	1.2324	0.6287	0.7441	0.6823
2	1.0643	0.5346	0.7785	0.7183
3	0.9381	0.5406	0.8027	0.7406
4	0.8494	0.5503	0.8149	0.7525 ← saved

Validation loss bottomed at epoch 2; macro F1 continued improving through epoch 4. The model was still converging at epoch 4 (train loss descending), indicating further gains are achievable with continued training at a reduced learning rate.

---

Evaluation Results

All results are on the held-out test set (12,563 samples, never seen during training).

Overall Test Set

Metric	Score
Accuracy	0.8159
Macro F1	0.7533
Weighted F1	0.8251

Per-Class (Full Test Set)

Class	Precision	Recall	F1	Support
negative	0.85	0.81	0.83	4,721
neutral	0.45	0.73	0.56	1,234
positive	0.91	0.84	0.87	6,608

Per-Script Breakdown

Script	Samples	Accuracy	Macro F1	F1 neg	F1 neu	F1 pos
Devanagari	9,294	0.8072	0.7369	0.8448	0.5296	0.8363
Romanized	1,251	0.7346	0.7019	0.6241	0.6097	0.8719
Mixed/English	2,018	0.9063	0.7777	0.7684	0.5995	0.9651

---

Known Limitations

Neutral class precision is low (0.45). The model over-predicts neutral — more than half of samples it labels neutral are actually negative or positive. Root causes: neutral is the rarest class (~10% of training data), and neutral annotation criteria differ significantly across the 7 source datasets. For pipelines where neutral precision matters, apply a confidence threshold (≥0.55–0.60 on the neutral logit) before accepting a neutral label.

Romanized negative detection is weak (F1=0.624). The romanized training set is small (~3k samples) and likely skewed positive. Romanized negative expressions are underrepresented. Use with caution for negative sentiment triage in romanized text.

Binary use case is strongest. For positive/negative classification (ignoring neutral), the model performs well across all scripts. Treating low-confidence neutral predictions as "uncertain" rather than a hard label is recommended for downstream applications.

Dialect and register gaps. Training data skews toward social media, news, and review text. Performance on formal Nepali (legal, administrative, academic) or strongly dialectal text (Terai, Hill dialects) is untested.

Dataset label inconsistency. Labels were merged from 7 independent annotation efforts with no inter-annotator agreement study across sources. Noise from inconsistent labeling is present, especially at the neutral/negative boundary.

---

Intended Use

Suitable for:

• Bulk positive/negative triage of Nepali social media, reviews, and news comments
• Script-agnostic Nepali sentiment scoring (Devanagari, Romanized, code-mixed)
• Downstream tasks requiring a multilingual Nepali sentiment feature

Not suitable for:

• High-stakes per-sample decisions without human review
• Neutral class as a reliable hard label without threshold tuning
• Languages other than Nepali and English

---

Citation

If you use this model, please cite the base model and the primary training dataset:

@article{conneau2019unsupervised,
  title   = {Unsupervised Cross-lingual Representation Learning at Scale},
  author  = {Conneau, Alexis and others},
  journal = {arXiv preprint arXiv:1911.02116},
  year    = {2019}
}

---

Acknowledgements

Training data sourced from the open Nepali NLP community. Primary dataset backbone: Sagar32/NepaliDevanagariSentimentAnalysis. Full dataset list and dataset-building pipeline available in the linked training repository.