README.md

---
language:
- bn
license: apache-2.0
tags:
- bengali
- bangla
- causal-lm
- llama
- custom-tokenizer
- parameter-efficient
- instruction-tuning
- sft
datasets:
- spitfire4794/Bangla-SFT-50k
metrics:
- accuracy
base_model:
- spitfire4794/Alo-70m-Base
pipeline_tag: text-generation
library_name: transformers
---

# Alo-70M (Instruct)

## Model Summary
**Alo-70M** is the instruction-tuned version of the ultra-lightweight 69-million parameter Bengali language model, [Alo-70M-Base](https://huggingface.co/spitfire4794/Alo-70M-Base). Built on a scaled-down LLaMA architecture, it is designed to act as a highly efficient, edge-deployable localized AI assistant. 

Fine-tuned on a curated dataset of instruction-response pairs using the **ChatML** format, Alo-70M is aligned for tasks such as summarization, entity extraction, text editing, and question answering in native Bengali. Despite its compact footprint, it offers a viable path for edge AI deployment on standard CPUs and mobile hardware.

* **Developer:** Fahad Hossain
* **Language:** Bengali (Bangla)
* **Model Type:** Causal Language Model (Instruction-Tuned Autoregressive Transformer)
* **Parameter Count:** 69 Million
* **License:** Apache 2.0

## Related Resources
* **Base Model:** [spitfire4794/Alo-70M-Base](https://huggingface.co/spitfire4794/Alo-70M-Base)
* **Alignment Dataset:** [spitfire4794/Bangla-SFT-50k](https://huggingface.co/datasets/spitfire4794/Bangla-SFT-50k)
* **Tokenizer:** [spitfire4794/beng_bpe](https://huggingface.co/spitfire4794/beng_bpe)

## Usage 

Alo-70M was trained using the ChatML template. The chat template is built directly into the Jinja template of the tokenizer (`spitfire4794/beng_bpe`). You can leverage it using Hugging Face's `apply_chat_template` interface:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "spitfire4794/Alo-70M"

# Load the custom Bengali BPE tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Define the instruction in ChatML format
messages = [
    {"role": "user", "content": "নিচের অনুচ্ছেদটি সংক্ষেপে সারসংক্ষেপ করুন: [এখানে আপনার টেক্সট লিখুন]"}
]

# Apply the pre-configured ChatML template
inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

# Generate text
outputs = model.generate(
    **inputs, 
    max_new_tokens=150, 
    repetition_penalty=1.1,
    do_sample=True,
    temperature=0.6,
    top_p=0.9
)

# Decode response (omitting user prompt)
response = outputs[0][inputs.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))
```

## Supervised Fine-Tuning (SFT) Details
Alo-70M was aligned using a curated subset of the **Bangla-SFT-50k** dataset formatted using **ChatML**.
* **Dataset Pruning:** Initial SFT experiments revealed that forcing a sub-100M parameter model to learn complex markdown syntax/tables caused severe representation crowding. Thus, the 12,517-sample *Structured Formatting* category was excluded. The final active training mixture consisted of **37,536** aligned pairs.
* **Engineering Properties:** The training data strictly forbade conversational prefaces (e.g., "নিশ্চয়ই, আমি এটি করে দিচ্ছি") so that responses begin immediately with the target output, optimizing inference speeds.
* **Hardware:** NVIDIA T4 and L4 GPUs.
* **Hyperparameters:** 
  * Optimizer: Fused AdamW (`adamw_torch_fused`) with $\beta_1 = 0.9, \beta_2 = 0.999, \epsilon = 10^{-8}$
  * Weight Decay: 0.0
  * Learning Rate Schedule: Cosine decay, peaking at $3 \times 10^{-4}$ with a 10% linear warmup.
  * Epochs: 3
  * Effective Batch Size: 32 (per-device 8 with gradient accumulation of 4).
  * Precision: Native Automatic Mixed Precision (AMP).

## Model Architecture details
Like its base model, Alo-70M utilizes a parameter-efficient architecture:
* **Layers:** 12
* **Hidden Dimension ($d_{model}$):** 512 | **Intermediate FFN:** 1408
* **Attention:** Grouped-Query Attention (GQA) with 8 query heads / 4 KV heads.
* **Positional Embeddings:** RoPE (Base freq: 10,000)
* **Word Embeddings:** Untied (`tie_word_embeddings = False`).
* **Context Window:** 1024 tokens.

## Evaluation Results
The model was evaluated zero-shot across Bengali reasoning and knowledge benchmarks (continuation-based log-probability evaluation):

| Benchmark | Alo-70M (SFT) | Alo-70M-Base | Gemma-3-270M-IT | TigerLLM-1B-IT |
| :--- | :---: | :---: | :---: | :---: |
| **bangla_mmlu_bn** | 26.29% | 26.31% | 26.81% | 27.66% |
| **bangla_commonsenseqa_bn** | **25.88%** | 28.42% | 22.77% | 25.14% |
| **indicbench_arc_bn_challenge** | 24.15% | 22.70% | 25.34% | 27.13% |
| **boolqa_bn** | 48.70% | 48.42% | 51.30% | 52.40% |
| **openbookqa_bn** | 30.58% | 31.39% | 31.99% | 34.21% |
| **piqa_bn** | **50.05%** | 50.49% | 49.51% | 49.51% |
| **hellaswag_bn** | 26.89% | 27.27% | 27.85% | 31.01% |

*Note: The 69M instruction-tuned model outperforms the larger Gemma-3-270M-IT baseline on tasks like CommonsenseQA and PIQA.*

## Limitations and Biases
* **Alignment Tax (Catastrophic Forgetting):** While SFT successfully aligned the model for text generation stability and instruction following, it introduced a measurable degradation in pure zero-shot reasoning compared to the Base model (e.g., dropping from 28.42% to 25.88% on CommonsenseQA). This happens because applying instructions to a highly capacity-constrained 69M model over-indexes weights toward output formatting at the expense of some pre-trained logical representations.
* **Knowledge Retrieval:** With under 100M parameters, the model physically lacks the capacity to serve as a comprehensive encyclopedic knowledge base. It is better suited for text processing tasks (editing, summarizing) than fact-retrieval.
* **Context Length:** The model is optimized for a 1024-token context window. Prompts exceeding this length will be truncated or result in degraded quality.

## Citation
*Technical paper out soon.*