Squash history to single initial commit

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,114 @@
+---
+language:
+- bn
+license: apache-2.0
+tags:
+- bengali
+- bangla
+- causal-lm
+- llama
+- custom-tokenizer
+- parameter-efficient
+datasets:
+- HuggingFaceFW/fineweb-2
+- hishab/titulm-bangla-corpus
+metrics:
+- accuracy
+---
+
+# Alo-70M-Base
+
+## Model Summary
+**Alo-70M-Base** is a compact, 69-million parameter foundational Large Language Model (LLM) built exclusively for the Bengali (Bangla) language. Trained entirely from scratch, it aims to lower the compute barrier for Bengali NLP research and provide a viable path for deploying localized AI assistants on standard CPUs and resource-constrained edge devices. 
+
+Despite its ultra-lightweight footprint, Alo-70M-Base achieves competitive parity on localized reasoning tasks against significantly larger cross-lingual models (such as 270M and 1B parameter baselines). 
+
+* **Developer:** Fahad Hossain
+* **Language:** Bengali (Bangla)
+* **Model Type:** Causal Language Model (Autoregressive Transformer)
+* **Parameter Count:** 69 Million
+* **License:** Apache 2.0
+
+## Related Resources
+This model is the **base** foundational model. We also release the instruction-tuned version, the synthetic dataset used for alignment, and the standalone tokenizer:
+* **Instruction-Tuned Model:** [spitfire4794/Alo-70M](https://huggingface.co/spitfire4794/Alo-70M)
+* **Alignment Dataset:** [spitfire4794/Bangla-SFT-50k](https://huggingface.co/datasets/spitfire4794/Bangla-SFT-50k)
+* **Tokenizer:** [spitfire4794/beng_bpe](https://huggingface.co/spitfire4794/beng_bpe)
+
+## Usage 
+
+You can load and generate text with Alo-70M-Base using the `transformers` library. 
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "spitfire4794/Alo-70M-Base"
+tokenizer_id = "spitfire4794/beng_bpe"
+
+# Load the custom Bengali BPE tokenizer and model
+tokenizer = AutoTokenizer.from_pretrained(tokenizer_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
+
+prompt = "বাংলাদেশের রাজধানী ঢাকা"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+# Generate text
+outputs = model.generate(
+    **inputs, 
+    max_new_tokens=50, 
+    repetition_penalty=1.1,
+    do_sample=True,
+    temperature=0.7
+)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Model Details
+Alo-70M-Base is based on a scaled-down LLaMA transformer architecture optimized for hardware alignment and parameter efficiency.
+* **Layers:** 12
+* **Hidden Dimension ($d_{model}$):** 512
+* **Intermediate FFN Dimension:** 1408
+* **Attention Mechanism:** Grouped-Query Attention (GQA) with 8 query heads and 4 key-value heads.
+* **Positional Embeddings:** Rotary Position Embeddings (RoPE) with a base frequency ($\theta$) of 10,000.
+* **Activation Function:** SwiGLU (`silu`)
+* **Word Embeddings:** Untied (`tie_word_embeddings = False`). The static lookup tables account for 48.6% (33.5M) of the total parameter budget.
+* **Context Window:** 1024 tokens.
+* **Tokenizer:** Custom Bengali Byte-Pair Encoding (BPE) tokenizer ([spitfire4794/beng_bpe](https://huggingface.co/spitfire4794/beng_bpe)) with a highly optimized vocabulary size of 32,768 ($2^{15}$) designed to maximize TPU efficiency.
+
+## Pre-Training Data
+The model was pre-trained on a curated, strictly Bengali corpus containing approximately **19.25 billion tokens**. The dataset is a blend of:
+1. **FineWeb-2 (Bengali Split):** Providing diverse, web-scale filtered documents.
+2. **TituLM Bengali Corpus (CommonCrawl Split):** Introducing balanced, general-domain web data.
+
+## Training Procedure
+Alo-70M-Base was initialized with completely randomized weights (no transfer learning) and trained from scratch. 
+* **Hardware:** Kaggle Tensor Processing Unit (TPU) v5e-8
+* **Training Time:** 12.9 hours (wall time) across two sequential runs.
+* **Optimizer:** AdamW (`adam_torch_xla`) with $\beta_1 = 0.9, \beta_2 = 0.95, \epsilon = 10^{-8}$
+* **Weight Decay:** 0.1
+* **Learning Rate Schedule:** Cosine Decay with a gradual warmup phase, peaking at $6 \times 10^{-4}$.
+* **Precision & Optimization:** Mixed `bfloat16` (`bf16`) precision with gradient checkpointing enabled to maximize throughput.
+
+## Evaluation Results
+The model was evaluated in a zero-shot setting across various Bengali reasoning and knowledge benchmarks using a continuation-based evaluation methodology (calculating conditional log-probabilities directly over raw Bengali text to avoid "pointer-reasoning" bottlenecks).
+
+| Benchmark | Alo-70M-Base | Gemma-3-270M-IT | TigerLLM-1B-IT |
+| :--- | :---: | :---: | :---: |
+| **bangla_mmlu_bn** | 26.31% | 26.81% | 27.66% |
+| **bangla_commonsenseqa_bn** | **28.42%** | 22.77% | 25.14% |
+| **indicbench_arc_bn_challenge** | 22.70% | 25.34% | 27.13% |
+| **boolqa_bn** | 48.42% | 51.30% | 52.40% |
+| **openbookqa_bn** | 31.39% | 31.99% | 34.21% |
+| **piqa_bn** | **50.49%** | 49.51% | 49.51% |
+| **hellaswag_bn** | 27.27% | 27.85% | 31.01% |
+
+*Note: Alo-70M-Base matches or outperforms both the 270M and 1B baselines on CommonsenseQA and PIQA despite having 4x to 14x fewer parameters.*
+
+## Limitations and Biases
+* **Encyclopedic Knowledge:** Due to the severe parameter constraints (<100M parameters), the model physically lacks the parametric capacity to act as a deep encyclopedic knowledge base, which reflects in its performance on knowledge-intensive retrieval tasks like MMLU and ARC-Challenge.
+* **Tokenizer Evaluation Bias:** During continuation-based log-likelihood evaluations, Alo-70M's highly efficient native tokenizer may possess a mathematical advantage over multilingual baselines (which heavily fragment Bengali text), contributing partially to its high performance margins on specific tasks. 
+* **Instruction Following:** This is a base model. It will attempt to complete text rather than follow conversational instructions. For chat and instruction-following, please use **Alo-70M**.
+
+## Citation
+*Technical paper out soon.*

chat_template.jinja

@@ -0,0 +1,10 @@
+{%- if messages[0]['role'] == 'system' and messages[0]['content'] -%}{{ '<|im_start|>system
+' + messages[0]['content'] + '<|im_end|>
+' }}{%- set loop_messages = messages[1:] -%}{%- else -%}{{ '<|im_start|>system
+You are a helpful assistant.<|im_end|>
+' }}{%- set loop_messages = messages -%}{%- endif -%}{%- for message in loop_messages -%}{%- if message['role'] == 'tool' -%}{{ '<|im_start|>tool
+' + message['content'] + '<|im_end|>
+' }}{%- else -%}{{ '<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>
+' }}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{ '<|im_start|>assistant
+' }}{%- endif -%}

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1408,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.13.0",
+  "use_cache": false,
+  "vocab_size": 32768
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": [
+    2
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.13.0",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b07984f6f3be9a1299ea46f7781e009943da8594468beb37252b0a87076616f
+size 275839008

tokenizer_config.json

@@ -0,0 +1,23 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|tool_call|>",
+    "<|/tool_call|>",
+    "<|tool_response|>",
+    "<|begin_of_thought|>",
+    "<|end_of_thought|>",
+    "<|reserved_0|>",
+    "<|reserved_1|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}