README.md

metadata

language:
  - bn
license: cc-by-4.0
library_name: transformers
pipeline_tag: fill-mask
tags:
  - bengali
  - bangla
  - BnLM-F
  - bnlm
  - bangla-model
  - small-bangla-model
  - bengali-nlp
  - bnlp
  - low-resource
  - masked-language-modeling
  - pretrained
  - language-model
  - distilbert
datasets:
  - nahid-hub/B-CORE-bengali-corpus
model-index:
  - name: BnLM-F
    results: []

BnLM-F: Bangla Language Model (135M)

BnLM-F is a 135M-parameter Bangla-specific pretrained language model, one of three models in the BnLM suite (Bengali Language Model) introduced alongside the BLUGE benchmark and B-CORE pretraining corpus. BnLM models are pretrained from scratch on Bangla text using the Masked Language Modeling (MLM) objective, and are designed for efficient, low-resource NLP without relying on large multilingual models.

See the BLUGE collection for the full release — evaluation tasks, pretraining corpus, tokenizers, and all three BnLM variants (BnLM-F, BnLM-M, BnLM-C).

Model Details

Base architecture	DistilBERT-multilingual
Parameters	135M
Tokenizer	Custom Bangla WordPiece, 120K vocabulary
Objective	Masked Language Modeling (MLM)
Max sequence length	512 tokens
Pretraining data	B-CORE — 4.32B tokens, 52GB

Despite its compact size, BnLM-F achieves state-of-the-art results across BLUGE and four cross-lingual/Indic benchmarks, while requiring 44–91% fewer FLOPs and ~2x faster inference than competitive multilingual baselines. Full training configuration and hyperparameters are available in the paper.

Usage

Fine-tuning for sentiment classification

Example using BLUGE-TSC:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("nahid-hub/BnLM-F-135m")
model = AutoModelForSequenceClassification.from_pretrained(
    "nahid-hub/BnLM-F-135m",
    num_labels=3  # 0: neutral, 1: positive, 2: negative
)

text = "খাবারটা অসাধারণ ছিল, আমি খুব খুশি!"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_class = logits.argmax(dim=-1).item()
labels = {0: "neutral", 1: "positive", 2: "negative"}
print(labels[predicted_class])

Note: AutoModelForSequenceClassification attaches a randomly initialized classification head on top of the pretrained BnLM-F encoder — you'll need to fine-tune on labeled data (e.g. BLUGE-TSC) before this produces meaningful predictions. Skip straight to inference only if loading your own fine-tuned checkpoint.

Loading the base model (for masked-language-modeling or custom heads)

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("nahid-hub/BnLM-F-135m")
model = AutoModelForMaskedLM.from_pretrained("nahid-hub/BnLM-F-135m")

Training Data

Pretrained exclusively on B-CORE, a 16.5M-document, 4.32-billion-token Bangla corpus built via a reproducible multi-stage pipeline achieving a 22.4% reduction in corpus volume through quality filtering and cross-corpus deduplication.

Evaluation

Evaluated on BLUGE (7-task Bangla NLU benchmark) and 4 cross-lingual/Indic benchmarks. See the paper for full results tables.

Task	Metric	Score
[fill in from your paper's results tables]

Related Models

• BnLM-M — 135M parameters, larger-vocabulary variant tuned for different training regime
• BnLM-C — 66M parameters, compact variant with 30.5K-vocabulary tokenizer

Limitations

[e.g. domain coverage of B-CORE, max sequence length, any known biases — fill in from your paper's limitations section]

License

Released under CC BY 4.0.

Citation

If you use this model, please cite:

@ARTICLE{BnLM-BLUGE-B-CORE,
  author={Hossain, Nahid and Faisal Kabir, Md.},
  journal={IEEE Access}, 
  title={Efficient Monolingual Pretraining in Low-Resource Settings Through Morphology-Aware Tokenization, Principled Corpus Denoising, and Benchmark-Driven Evaluation}, 
  year={2026},
  volume={14},
  number={},
  pages={91979-92003},
  keywords={Modeling;Multilingual;Training;Cleaning;Vocabulary;Labeling;Tokenization;Computational linguistics;Pipelines;Conferences;B-CORE;BLUGE;BnLM;corpus;evaluation benchmark;pretrained models},
  doi={10.1109/ACCESS.2026.3701520}
}