vaibhavmaurya/hindi-slm-v001

Hindi SLM v001

A 46M-parameter Hindi-only causal language model trained from scratch on the ai4bharat/sangraha Hindi corpus. Designed for edge deployment (Raspberry Pi 8 GB) while producing fluent Devanagari text.

Model Details

Property	Value
Architecture	LlamaForCausalLM (decoder-only transformer)
Parameters	46.54M (unique) / 62.93M (with untied lm_head)
Hidden size	512
Layers	10
Attention heads	8 (GQA with 2 KV heads)
Intermediate size	1,536 (SwiGLU)
Context length	512 tokens
Vocabulary	32,000 (Hindi Unigram SentencePiece)
Precision	bfloat16
Tokenizer	vaibhavmaurya/hindi-slm-tokenizer-v001

Training

Property	Value
Dataset	ai4bharat/sangraha (hin/verified), ~1 GB
Steps	50,000
Effective batch size	32 (batch=8 × grad_accum=4)
Learning rate	3e-4 with cosine decay, 1000-step warmup
Weight decay	0.1
Hardware	NVIDIA RTX 3000 Ada (8 GB VRAM), bfloat16
Final train loss	3.4323

Evaluation

Evaluated on held-out val and test splits (98/1/1 split of the training corpus):

Metric	Value
Val perplexity	38.91
Test perplexity	38.63
Avg Devanagari ratio	0.765
UNK tokens	0.0

A val ≈ test perplexity confirms no overfitting. PPL < 50 is rated EXCELLENT for a 46M-parameter model.

Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "vaibhavmaurya/hindi-slm-v001"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompts = [
    "आज का समाचार यह है कि",
    "भारत की राजधानी नई दिल्ली है।",
    "नमस्ते! आप कैसे हैं?",
    "एक बार की बात है,",
    "चाँद की रोशनी में,",
]

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=100,
            temperature=0.8,
            top_p=0.9,
            repetition_penalty=1.1,
            do_sample=True,
        )
    generated = tokenizer.decode(output_ids[0], skip_special_tokens=True)
    print(generated)
    print("---")

CPU-only inference (Raspberry Pi / laptop)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float32,   # bfloat16 not supported on CPU
    device_map="cpu",
)

Streaming generation

from transformers import TextStreamer

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

inputs = tokenizer("भारत की राजधानी", return_tensors="pt").to(model.device)
model.generate(
    **inputs,
    max_new_tokens=150,
    temperature=0.8,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True,
    streamer=streamer,
)

Intended Use

• Hindi text completion and generation
• Educational and research use for small Hindi LMs
• Edge deployment experiments (Raspberry Pi, microcontrollers with sufficient RAM)
• Foundation for Hindi instruction-tuning

Limitations

• Context window is 512 tokens — unsuitable for long-document tasks
• Trained on web/Wikipedia-style Hindi; may not handle dialects, transliterated text, or code-switching well
• No RLHF or instruction tuning — outputs are continuations, not instruction-following responses
• Not evaluated for factual accuracy; may hallucinate

License

Apache 2.0