Qwen2.5-7B-Indonesian-LoRA

Fine-tuning Qwen2.5-7B-Instruct with LoRA on Indonesian instruction dataset for better Bahasa Indonesia conversational fluency.

Overview

Component	Value
Base Model	Qwen/Qwen2.5-7B-Instruct (7.6B params)
Dataset	cahya/instructions_indonesian (~52K Indonesian instruction-response pairs)
Method	LoRA SFT (Supervised Fine-Tuning)
LoRA Rank	r=32, alpha=16
Trainable Params	~0.1% of full model
Learning Rate	2.0e-4 (10x base for LoRA)
Epochs	2
Max Seq Length	2048
Effective Batch Size	16 (batch=2 × grad_accum=8)

Training Recipe

The training follows the approach from SeaLLMs-v3 (arXiv:2407.19672) which demonstrated that Qwen2.5 tokenizer has superior Indonesian/SEA language coverage versus Llama/Mistral. Key hyperparameters from published results:

• lr=2.0e-4 for LoRA (vs 2.0e-5 for full FT)
• warmup_ratio=0.1 with cosine decay
• packing=True for training efficiency
• bf16 mixed precision
• gradient_checkpointing for memory efficiency

Dataset Format

The dataset is preprocessed from the original text/label columns into conversational messages format:

{
  "messages": [
    {"role": "user", "content": "Bagaimana cara meningkatkan koneksi pikiran-tubuh saya?"},
    {"role": "assistant", "content": "1. Mulailah dengan mengembangkan latihan kesadaran yang teratur..."}
  ]
}

How to Run

Option 1: HF Jobs (cloud GPU)

1. Top up credits at https://huggingface.co/settings/billing
2. Submit via hf_jobs or trl CLI:

python -m hf_jobs.run \
  --script train.py \
  --hardware a10g-large \
  --timeout 7h \
  --dependencies transformers,trl,torch,datasets,peft,trackio,accelerate

Option 2: Local / RunPod / Colab

Requires GPU with ≥24GB VRAM (e.g., A10G, A100, RTX 3090/4090):

pip install transformers trl torch datasets peft trackio accelerate
python train.py

Monitoring

Training metrics are logged to Trackio dashboard. Set TRACKIO_SPACE_ID environment variable to enable live tracking.

References

• SeaLLMs-v3: Open Foundation and Chat Multilingual LLMs for SEA Languages (arXiv:2407.19672)
• TRL SFT Trainer: https://huggingface.co/docs/trl/sft_trainer
• LoRA Without Regret: https://huggingface.co/docs/trl/lora_without_regret

License

• Base model: Apache 2.0
• Dataset: MIT
• This adapter: Apache 2.0