---
language:
- km
- en
license: gemma
base_model: google/gemma-3-1b-pt
tags:
- khmer
- continued-pretraining
- gemma3
- tonsai
- preview
library_name: transformers
pipeline_tag: text-generation
---

# Gemma-3-Tonsai-1B-v0.1

> **Preview Release**: This is an early preview (v0.1) for validation purposes.
> Not intended for production use. Evaluation and model quality may improve in future versions.

**Gemma-3-Tonsai-1B** is a Khmer-enhanced language model built through Continued Pre-Training (CPT)
of Google's [Gemma 3 1B](https://huggingface.co/google/gemma-3-1b-pt) on a mixture of Khmer, English, and parallel data.

"Tonsai" (ទន្សាយ) means "rabbit" in Khmer.

> **Note**: This is a **base model** trained via Continued Pre-Training. It is designed as a
> foundation for downstream task-specific fine-tuning (e.g., translation, summarization,
> question answering). For best results, we recommend fine-tuning on your target task
> using Supervised Fine-Tuning (SFT) before deployment.

## Model Details

| | |
|---|---|
| **Base Model** | [google/gemma-3-1b-pt](https://huggingface.co/google/gemma-3-1b-pt) |
| **Training Method** | Continued Pre-Training (CPT), full parameter update |
| **Languages** | Khmer (km), English (en) |
| **Parameters** | ~1B |
| **Context Length** | 4096 tokens |
| **Precision** | bfloat16 |
| **License** | [Gemma Terms of Use](https://ai.google.dev/gemma/terms) |

### Model Lineage

```
google/gemma-3-1b-pt
  └── mengsay/Gemma-3-Tonsai-1B-v0.1 (CPT on Khmer data)
```

## Training

### Data Mix

| Dataset | Type | Weight | Role |
|---------|------|--------|------|
| [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX) (km) | Monolingual | 55% | Khmer web text |
| [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (km) | Monolingual | 5% | High-quality Khmer |
| [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX) (en) | Monolingual | 10% | English retention |
| [OPUS-100](https://huggingface.co/datasets/Helsinki-NLP/opus-100) (en-km) | Parallel | 15% | Cross-lingual alignment |
| [OpenHermes 2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) | Instruction | 10% | Instruction following |
| [Khmer Dictionary 44K](https://huggingface.co/datasets/seanghay/khmer-dictionary-44k) | Dictionary | 5% | Vocabulary knowledge |

### Hyperparameters

| Parameter | Value |
|-----------|-------|
| Effective batch size | 64 (32 per device x 2 grad accum) |
| Max sequence length | 4096 |
| Learning rate | 5e-5 (embedding: 1e-5) |
| LR scheduler | Cosine with warmup |
| Warmup steps | 200 |
| Weight decay | 0.01 |
| Optimizer | AdamW 8-bit |
| Gradient checkpointing | Unsloth |
| Hardware | NVIDIA RTX PRO 6000 Blackwell (95GB VRAM) |

## Evaluation

Evaluation on OPUS-100 (en-km) translation and Khmer perplexity tasks.

### Perplexity (lower is better)

| Dataset | Gemma-3-1B-PT (base) | Tonsai-1B v0.1 |
|---------|----------------------|----------------|
| Wikipedia (km) | 9.06 | **2.14** |
| CulturaX (km) | 7.09 | 7.90 |

Khmer Wikipedia perplexity drops dramatically (9.06 → 2.14), showing significant improvement in Khmer text prediction.
CulturaX perplexity is comparable, as the model is still mid-training.

### Translation (OPUS-100, 500 samples)

| Task | Setting | Metric | Gemma-3-1B-PT (base) | Tonsai-1B v0.1 |
|------|---------|--------|----------------------|----------------|
| en→km | zero-shot | BLEU | 1.62 | **18.04** |
| en→km | 5-shot | BLEU | 3.71 | **19.34** |
| en→km | zero-shot | chrF | 4.45 | **36.25** |
| en→km | 5-shot | chrF | 16.60 | **37.14** |
| km→en | zero-shot | BLEU | 9.38 | **19.66** |
| km→en | 5-shot | BLEU | 13.12 | **19.00** |
| km→en | zero-shot | chrF | 31.21 | **44.57** |
| km→en | 5-shot | chrF | 35.70 | **42.09** |

Translation performance improves substantially in both directions, especially en→km zero-shot (BLEU 1.62 → 18.04).

## Usage

### Text Generation

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "mengsay/Gemma-3-Tonsai-1B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "ជីវិតរស់នៅក្នុងទីក្រុងសព្វថ្ងៃពិតជា"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

### Translation Example

```python
prompt = "English: Cambodia is a country in Southeast Asia.\nKhmer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

### With vLLM Serving

```bash
# Start vLLM server
python -m vllm.entrypoints.openai.api_server \
    --model mengsay/Gemma-3-Tonsai-1B-v0.1 --port 8000
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.completions.create(
    model="mengsay/Gemma-3-Tonsai-1B-v0.1",
    prompt="Cambodia is",
    max_tokens=200,
)
print(response.choices[0].text)
```

## Intended Use

This model is a **continual pre-trained base model** — it has been trained to improve Khmer language
understanding and generation but has **not** been fine-tuned for any specific task or instruction following.

**Recommended workflow:**
1. Use this model as a starting point for **Supervised Fine-Tuning (SFT)** on your target task
2. Example downstream tasks: translation (en↔km), summarization, question answering, text classification
3. Fine-tuning with even a few thousand task-specific examples can significantly improve performance

**Not recommended for:**
- Direct use as a chatbot or instruction-following assistant (use an instruction-tuned variant instead)
- Production deployment without task-specific fine-tuning and evaluation

## Limitations

- This is a **preview release (v0.1)** intended for validation and research
- This is a CPT base model — **fine-tuning on a specific task is recommended** before use
- Not optimized for instruction following or conversational use
- May generate incorrect, biased, or harmful content
- Khmer language quality is preliminary; comprehensive benchmarks will follow in future versions
- Training data may contain biases present in web-crawled corpora

## Citation

```bibtex
@misc{tonsai-lm-2026,
  title   = {Tonsai LM: Continued Pre-Training for Khmer Language Models},
  author  = {Mengsay Loem},
  year    = {2026},
  url     = {https://huggingface.co/mengsay/Gemma-3-Tonsai-1B-v0.1}
}
```

## Acknowledgements

- [Google Gemma](https://ai.google.dev/gemma) team for the base model
- [Unsloth](https://github.com/unslothai/unsloth) for training optimization
- HuggingFace dataset contributors for open Khmer language resources
- [Tonsai LM project](https://github.com/loem-ms/tonsai-lm)