Upload mengsay/Gemma-3-Tonsai-1B-v0.1

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,197 @@
+---
+language:
+- km
+- en
+license: gemma
+base_model: google/gemma-3-1b-pt
+tags:
+- khmer
+- continued-pretraining
+- gemma3
+- tonsai
+- preview
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# Gemma-3-Tonsai-1B-v0.1
+
+> **Preview Release**: This is an early preview (v0.1) for validation purposes.
+> Not intended for production use. Evaluation and model quality may improve in future versions.
+
+**Gemma-3-Tonsai-1B** is a Khmer-enhanced language model built through Continued Pre-Training (CPT)
+of Google's [Gemma 3 1B](https://huggingface.co/google/gemma-3-1b-pt) on a mixture of Khmer, English, and parallel data.
+
+"Tonsai" (ទន្សាយ) means "rabbit" in Khmer.
+
+> **Note**: This is a **base model** trained via Continued Pre-Training. It is designed as a
+> foundation for downstream task-specific fine-tuning (e.g., translation, summarization,
+> question answering). For best results, we recommend fine-tuning on your target task
+> using Supervised Fine-Tuning (SFT) before deployment.
+
+## Model Details
+
+| | |
+|---|---|
+| **Base Model** | [google/gemma-3-1b-pt](https://huggingface.co/google/gemma-3-1b-pt) |
+| **Training Method** | Continued Pre-Training (CPT), full parameter update |
+| **Languages** | Khmer (km), English (en) |
+| **Parameters** | ~1B |
+| **Context Length** | 4096 tokens |
+| **Precision** | bfloat16 |
+| **License** | [Gemma Terms of Use](https://ai.google.dev/gemma/terms) |
+
+### Model Lineage
+
+```
+google/gemma-3-1b-pt
+  └── mengsay/Gemma-3-Tonsai-1B-v0.1 (CPT on Khmer data)
+```
+
+## Training
+
+### Data Mix
+
+| Dataset | Type | Weight | Role |
+|---------|------|--------|------|
+| [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX) (km) | Monolingual | 55% | Khmer web text |
+| [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (km) | Monolingual | 5% | High-quality Khmer |
+| [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX) (en) | Monolingual | 10% | English retention |
+| [OPUS-100](https://huggingface.co/datasets/Helsinki-NLP/opus-100) (en-km) | Parallel | 15% | Cross-lingual alignment |
+| [OpenHermes 2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) | Instruction | 10% | Instruction following |
+| [Khmer Dictionary 44K](https://huggingface.co/datasets/seanghay/khmer-dictionary-44k) | Dictionary | 5% | Vocabulary knowledge |
+
+### Hyperparameters
+
+| Parameter | Value |
+|-----------|-------|
+| Effective batch size | 64 (32 per device x 2 grad accum) |
+| Max sequence length | 4096 |
+| Learning rate | 5e-5 (embedding: 1e-5) |
+| LR scheduler | Cosine with warmup |
+| Warmup steps | 200 |
+| Weight decay | 0.01 |
+| Optimizer | AdamW 8-bit |
+| Gradient checkpointing | Unsloth |
+| Hardware | NVIDIA RTX PRO 6000 Blackwell (95GB VRAM) |
+
+## Evaluation
+
+Evaluation on OPUS-100 (en-km) translation and Khmer perplexity tasks.
+
+### Perplexity (lower is better)
+
+| Dataset | Gemma-3-1B-PT (base) | Tonsai-1B v0.1 |
+|---------|----------------------|----------------|
+| Wikipedia (km) | 9.06 | **2.14** |
+| CulturaX (km) | 7.09 | 7.90 |
+
+Khmer Wikipedia perplexity drops dramatically (9.06 → 2.14), showing significant improvement in Khmer text prediction.
+CulturaX perplexity is comparable, as the model is still mid-training.
+
+### Translation (OPUS-100, 500 samples)
+
+| Task | Setting | Metric | Gemma-3-1B-PT (base) | Tonsai-1B v0.1 |
+|------|---------|--------|----------------------|----------------|
+| en→km | zero-shot | BLEU | 1.62 | **18.04** |
+| en→km | 5-shot | BLEU | 3.71 | **19.34** |
+| en→km | zero-shot | chrF | 4.45 | **36.25** |
+| en→km | 5-shot | chrF | 16.60 | **37.14** |
+| km→en | zero-shot | BLEU | 9.38 | **19.66** |
+| km→en | 5-shot | BLEU | 13.12 | **19.00** |
+| km→en | zero-shot | chrF | 31.21 | **44.57** |
+| km→en | 5-shot | chrF | 35.70 | **42.09** |
+
+Translation performance improves substantially in both directions, especially en→km zero-shot (BLEU 1.62 → 18.04).
+
+## Usage
+
+### Text Generation
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_name = "mengsay/Gemma-3-Tonsai-1B-v0.1"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+prompt = "ជីវិតរស់នៅក្នុងទីក្រុងសព្វថ្ងៃពិតជា"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+output = model.generate(**inputs, max_new_tokens=200)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+### Translation Example
+
+```python
+prompt = "English: Cambodia is a country in Southeast Asia.\nKhmer:"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+### With vLLM Serving
+
+```bash
+# Start vLLM server
+python -m vllm.entrypoints.openai.api_server \
+    --model mengsay/Gemma-3-Tonsai-1B-v0.1 --port 8000
+```
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
+response = client.completions.create(
+    model="mengsay/Gemma-3-Tonsai-1B-v0.1",
+    prompt="Cambodia is",
+    max_tokens=200,
+)
+print(response.choices[0].text)
+```
+
+## Intended Use
+
+This model is a **continual pre-trained base model** — it has been trained to improve Khmer language
+understanding and generation but has **not** been fine-tuned for any specific task or instruction following.
+
+**Recommended workflow:**
+1. Use this model as a starting point for **Supervised Fine-Tuning (SFT)** on your target task
+2. Example downstream tasks: translation (en↔km), summarization, question answering, text classification
+3. Fine-tuning with even a few thousand task-specific examples can significantly improve performance
+
+**Not recommended for:**
+- Direct use as a chatbot or instruction-following assistant (use an instruction-tuned variant instead)
+- Production deployment without task-specific fine-tuning and evaluation
+
+## Limitations
+
+- This is a **preview release (v0.1)** intended for validation and research
+- This is a CPT base model — **fine-tuning on a specific task is recommended** before use
+- Not optimized for instruction following or conversational use
+- May generate incorrect, biased, or harmful content
+- Khmer language quality is preliminary; comprehensive benchmarks will follow in future versions
+- Training data may contain biases present in web-crawled corpora
+
+## Citation
+
+```bibtex
+@misc{tonsai-lm-2026,
+  title   = {Tonsai LM: Continued Pre-Training for Khmer Language Models},
+  author  = {Mengsay Loem},
+  year    = {2026},
+  url     = {https://huggingface.co/mengsay/Gemma-3-Tonsai-1B-v0.1}
+}
+```
+
+## Acknowledgements
+
+- [Google Gemma](https://ai.google.dev/gemma) team for the base model
+- [Unsloth](https://github.com/unslothai/unsloth) for training optimization
+- HuggingFace dataset contributors for open Khmer language resources
+- [Tonsai LM project](https://github.com/loem-ms/tonsai-lm)

config.json

@@ -0,0 +1,78 @@
+{
+  "_sliding_window_pattern": 6,
+  "architectures": [
+    "Gemma3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": null,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "final_logit_softcapping": null,
+  "head_dim": 256,
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 1152,
+  "initializer_range": 0.02,
+  "intermediate_size": 6912,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "model_name": "unsloth/gemma-3-1b-pt",
+  "model_type": "gemma3_text",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 26,
+  "num_key_value_heads": 1,
+  "pad_token_id": 0,
+  "query_pre_attn_scalar": 256,
+  "rms_norm_eps": 1e-06,
+  "rope_local_base_freq": 10000.0,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 1000000,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000,
+      "rope_type": "default"
+    }
+  },
+  "rope_scaling": {
+    "rope_type": "default"
+  },
+  "rope_theta": 1000000.0,
+  "sliding_window": 512,
+  "sliding_window_pattern": 6,
+  "transformers_version": "4.57.6",
+  "unsloth_fixed": true,
+  "use_bidirectional_attention": false,
+  "use_cache": false,
+  "vocab_size": 262144
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "max_length": 32768,
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.57.6"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f2493b213c76018b83dfc5a14352066f212df339ff54483be77b660dcaafef7
+size 1999811208

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
+size 33384567