Upload folder using huggingface_hub

README.md

@@ -0,0 +1,150 @@
+---
+language:
+- en
+license: apache-2.0
+base_model: HuggingFaceTB/SmolLM2-360M
+tags:
+- telecom
+- 3gpp
+- etsi
+- standards
+- domain-adaptation
+- causal-lm
+- instruction-tuned
+datasets:
+- nareshmodina/TeleSpec-Data
+- tatsu-lab/alpaca
+metrics:
+- perplexity
+---
+
+# SmolLM-TS-360M-it
+
+A 360M parameter instruction-tuned language model specialised in 3GPP and ETSI telecommunications standards. Trained via full fine-tuning on [TeleSpec-Data](https://huggingface.co/datasets/nareshmodina/TeleSpec-Data) followed by LoRA instruction fine-tuning on Alpaca.
+
+Part of the **SmolLM-TS** series — small language models adapted exclusively to telecommunications standards documents, with zero arXiv or web content in the training corpus.
+
+> **Looking for the base pretrained version?** See [nareshmodina/SmolLM-TS-360M](https://huggingface.co/nareshmodina/SmolLM-TS-360M)
+
+---
+
+## Model Details
+
+| | |
+|---|---|
+| **Base model** | HuggingFaceTB/SmolLM2-360M |
+| **Parameters** | 360M |
+| **Training** | Full FT pretrain → LoRA SFT (Alpaca) |
+| **Pretraining data** | TeleSpec-Data (1.87B tokens) |
+| **SFT data** | Alpaca 52k |
+| **Context length** | 4096 tokens |
+| **Hardware** | 3× NVIDIA L40S (48GB) |
+
+---
+
+## Training
+
+**Stage 1 — Full fine-tuning on TeleSpec-Data**
+
+All model weights updated on 457,160 packed 4096-token blocks (1.87B tokens) from 38,302 standards documents — 15,054 3GPP (Rel-8 to Rel-19) and 23,248 ETSI documents spanning 15 working groups (2000–2024). Zero arXiv or web content — 100% standards text.
+
+- Epochs: 2 — Effective batch size: 128 — LR: 5e-5 (cosine)
+
+**Stage 2 — LoRA instruction fine-tuning**
+
+LoRA (r=16, α=32) on Alpaca 52k. Base weights frozen to preserve domain knowledge.
+
+- Epochs: 1 — LR: 1e-5
+
+---
+
+## Evaluation
+
+Evaluated on [Tele-Eval](https://huggingface.co/datasets/AliMaatouk/Tele-Eval) using the metrics defined in Maatouk et al. (2024) — **standards-derived questions only** (`standard_*` IDs, 10,000 examples, seed 42).
+
+| Model | Ans-PPL ↓ | SemScore ↑ |
+|---|---|---|
+| SmolLM2-360M-alpaca (base + Alpaca SFT) | 10.86 | 0.6216 |
+| **SmolLM-TS-360M-it (ours)** | **8.62** | **0.6572** |
+
+**20.6% Ans-PPL reduction** vs base+SFT baseline. Comparison across model sizes:
+
+| Model | Ans-PPL ↓ | SemScore ↑ |
+|---|---|---|
+| SmolLM-TS-135M-it | 9.19 | 0.6504 |
+| **SmolLM-TS-360M-it** | **8.62** | **0.6572** |
+
+Clear improvement with model size on both metrics.
+
+---
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id  = "nareshmodina/SmolLM-TS-360M-it"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model     = AutoModelForCausalLM.from_pretrained(
+    model_id, dtype=torch.bfloat16, device_map="auto"
+)
+
+prompt = (
+    "The following is a question about telecommunications and networking.\n"
+    "Question: What is the purpose of the RRC Connection Establishment procedure in LTE?\n"
+    "Answer:"
+)
+
+inputs  = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=150,
+    do_sample=False,
+    repetition_penalty=1.3,
+)
+answer = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
+print(answer)
+```
+
+> **Note:** Use the Alpaca-style `Question: ... Answer:` prompt format for best results.
+
+---
+
+## Limitations
+
+- **Alpaca SFT** — trained for Q&A style responses, not multi-turn conversation
+- **Standards only** — strong 3GPP/ETSI knowledge, limited general telecom knowledge
+- **Not for production** — intended for research purposes only
+
+---
+
+## Links
+
+- 📦 Dataset: [nareshmodina/TeleSpec-Data](https://huggingface.co/datasets/nareshmodina/TeleSpec-Data)
+- 🤖 Base model: [nareshmodina/SmolLM-TS-360M](https://huggingface.co/nareshmodina/SmolLM-TS-360M)
+- 📊 Benchmark: [AliMaatouk/Tele-Eval](https://huggingface.co/datasets/AliMaatouk/Tele-Eval)
+- 🗂️ Collection: [nareshmodina/SmolLM-TS](https://huggingface.co/collections/nareshmodina/smollm-ts)
+
+---
+
+## Citation
+
+```bibtex
+@misc{modina2025smollmts,
+  author    = {Naresh Modina},
+  title     = {SmolLM-TS: Small Language Models for Telecommunications Standards},
+  year      = {2025},
+  publisher = {Hugging Face},
+  url       = {https://huggingface.co/nareshmodina/SmolLM-TS-360M-it}
+}
+
+@misc{maatouk2024telellms,
+  title         = {Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications},
+  author        = {Ali Maatouk and Kenny Chirino Ampudia and Rex Ying and Leandros Tassiulas},
+  year          = {2024},
+  eprint        = {2409.05314},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.IT}
+}
+```

chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant
+'}}{% endif %}

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 960,
+  "initializer_range": 0.02,
+  "intermediate_size": 2560,
+  "is_llama_config": true,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 15,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 5,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_interleaved": false,
+  "rope_parameters": {
+    "rope_theta": 100000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.3.0",
+  "use_cache": false,
+  "vocab_size": 49152
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": [
+    0
+  ],
+  "transformers_version": "5.3.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc58958cdb118a86ce19f8a19d3cc83444a3e1449fc64fcf6e59ce48813382c9
+size 723674912

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "is_local": true,
+  "model_max_length": 8192,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}