| --- |
| language: |
| - en |
| license: apache-2.0 |
| base_model: HuggingFaceTB/SmolLM2-360M |
| tags: |
| - telecom |
| - 3gpp |
| - etsi |
| - standards |
| - domain-adaptation |
| - causal-lm |
| datasets: |
| - nareshmodina/TeleSpec-Data |
| metrics: |
| - perplexity |
| --- |
| |
| # SmolLM-TS-360M |
|
|
| A 360M parameter language model specialised in 3GPP and ETSI telecommunications standards, trained via full fine-tuning on [TeleSpec-Data](https://huggingface.co/datasets/nareshmodina/TeleSpec-Data). |
|
|
| Part of the **SmolLM-TS** series — small language models adapted exclusively to telecommunications standards documents, with zero arXiv or web content in the training corpus. |
|
|
| > **Looking for the instruction-tuned version?** See [nareshmodina/SmolLM-TS-360M-it](https://huggingface.co/nareshmodina/SmolLM-TS-360M-it) |
|
|
| --- |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | **Base model** | HuggingFaceTB/SmolLM2-360M | |
| | **Parameters** | 360M | |
| | **Training** | Full fine-tuning on TeleSpec-Data | |
| | **Pretraining data** | TeleSpec-Data (1.87B tokens) | |
| | **Context length** | 4096 tokens | |
| | **Hardware** | 3× NVIDIA L40S (48GB) | |
|
|
| --- |
|
|
| ## Training |
|
|
| Full fine-tuning of all model weights on 457,160 packed 4096-token blocks (1.87B tokens) from 38,302 standards documents — 15,054 3GPP (Rel-8 to Rel-19) and 23,248 ETSI documents spanning 15 working groups (2000–2024). Zero arXiv or web content — 100% standards text. |
|
|
| - Epochs: 2 |
| - Effective batch size: 128 — LR: 5e-5 (cosine with warmup) |
| - Context length: 4096 tokens |
|
|
| --- |
|
|
| ## Usage |
|
|
| This is a base model — it continues text rather than following instructions. For instruction following, use [SmolLM-TS-360M-it](https://huggingface.co/nareshmodina/SmolLM-TS-360M-it). |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| import torch |
| |
| model_id = "nareshmodina/SmolLM-TS-360M" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, dtype=torch.bfloat16, device_map="auto" |
| ) |
| |
| prompt = "The RRC Connection Establishment procedure in LTE is" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **Base model only** — does not follow instructions, use SmolLM-TS-360M-it for Q&A |
| - **Standards only** — strong 3GPP/ETSI knowledge, limited general telecom knowledge |
| - **Not for production** — intended for research purposes only |
|
|
| --- |
|
|
| ## Links |
|
|
| - 📦 Dataset: [nareshmodina/TeleSpec-Data](https://huggingface.co/datasets/nareshmodina/TeleSpec-Data) |
| - 🤖 Instruct version: [nareshmodina/SmolLM-TS-360M-it](https://huggingface.co/nareshmodina/SmolLM-TS-360M-it) |
| - 📊 Benchmark: [AliMaatouk/Tele-Eval](https://huggingface.co/datasets/AliMaatouk/Tele-Eval) |
| - 🗂️ Collection: [nareshmodina/SmolLM-TS](https://huggingface.co/collections/nareshmodina/smollm-ts) |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{modina2025smollmts, |
| author = {Naresh Modina}, |
| title = {SmolLM-TS: Small Language Models for Telecommunications Standards}, |
| year = {2025}, |
| publisher = {Hugging Face}, |
| url = {https://huggingface.co/nareshmodina/SmolLM-TS-360M} |
| } |
| ``` |
|
|
