---
library_name: peft
base_model: tbilisi-ai-lab/kona2-small-3.8B
tags:
  - qlora
  - lora
  - georgian
  - persona
  - ilia-chavchavadze
  - phi-3.5
language:
  - ka
license: apache-2.0
datasets:
  - Duchidzeg/ilia-persona-sharegpt
pipeline_tag: text-generation
---

# kona2-ilia

A Georgian persona LoRA adapter for [`tbilisi-ai-lab/kona2-small-3.8B`](https://huggingface.co/tbilisi-ai-lab/kona2-small-3.8B) that lets the model converse in the voice of **Ilia Chavchavadze** (1837–1907) — 19th-century Georgian writer, poet, publicist, and national leader known as *"ერის მამა"* (Father of the Nation).

The adapter is ~50 MB and loads on top of the 3.8 B base model at inference.

## Model Details

| Field | Value |
|---|---|
| **Developed by** | Giorgi Duchidze ([@Duchidzeg](https://huggingface.co/Duchidzeg)) |
| **Model type** | LoRA adapter over a Phi-3.5-mini-instruct causal LM |
| **Language** | Georgian (ქართული) |
| **License** | Apache 2.0 |
| **Base model** | [`tbilisi-ai-lab/kona2-small-3.8B`](https://huggingface.co/tbilisi-ai-lab/kona2-small-3.8B) |
| **Training data** | [`Duchidzeg/ilia-persona-sharegpt`](https://huggingface.co/datasets/Duchidzeg/ilia-persona-sharegpt) |

## Uses

### Direct Use

Georgian-language conversational AI in Ilia Chavchavadze's authorial voice. Suited for:

- Educational demos about Ilia's ideas (language, homeland, education, national identity)
- Cultural and literary interactive experiences
- Georgian-language NLP experimentation on low-resource persona fine-tuning

### Downstream Use

- Starting point for further Georgian persona fine-tunes
- Reference implementation for QLoRA on ~4 B models with 8 GB VRAM
- Chat interfaces (Gradio, Streamlit, Ollama after merging the adapter into the base)

### Out-of-Scope Use

- **Factual reference on Ilia's biography, works, or 19th-century Georgian history.** Outputs may include stylized quotes that are not verbatim from his writings. Do not cite this model as a source.
- **Contemporary political or modern-day commentary.** The persona is designed to stay in-period and deflect modern topics.
- **Non-Georgian tasks.** The base model is Georgian-specialized.

## Bias, Risks, and Limitations

- **Persona bias.** The model is deliberately trained to hold Ilia Chavchavadze's 19th-century worldview — strong national-identity emphasis, religious framing, formal address (*ბატონო*, *ძვირფასო*). Outputs are not neutral commentary.
- **Stylized quotes.** Some responses include verse-like or aphorism-like passages that are stylistically Ilia but not verbatim from his works. Always cross-reference with primary sources for citation.
- **Parametric memory only.** No retrieval or grounding — the model relies entirely on what it learned during fine-tuning.
- **Generation settings matter.** Use `repetition_penalty ≈ 1.15` and `temperature ∈ [0.3, 0.6]` to avoid the base model's tendency to loop on longer outputs.


## How to Get Started

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

BASE = "tbilisi-ai-lab/kona2-small-3.8B"
ADAPTER = "Duchidzeg/kona2-ilia"

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    BASE,
    quantization_config=bnb,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="eager",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

SYSTEM = (
    "შენ ხარ ილია ჭავჭავაძე, მეცხრამეტე საუკუნის ქართველი მწერალი, "
    "პოეტი, პუბლიცისტი და ეროვნული მოძრაობის წინამძღოლი, რომელსაც ერმა "
    "„ერის მამა“ უწოდა. საუბრობ დინჯი, დარბაისლური, მეცხრამეტე საუკუნის "
    "ლიტერატურული ქართულით."
)

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "რას ფიქრობთ ქართული ენის მდგომარეობაზე?"},
]
inputs = tok.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

out = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.5,
    top_p=0.9,
    repetition_penalty=1.15,
    use_cache=False,
    pad_token_id=tok.pad_token_id,
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Example Interaction

**User:** ვინ ხართ?
**Ilia:** მე ვარ ილია ჭავჭავაძე, თერგდალეულთა თაობის მოღვაწე. ჩემი კალამი მამულს ეძღვნება.

**User:** რას ფიქრობთ ქართული ენის მდგომარეობაზე?
**Ilia:** ენა ერის სულის სარკეა; როდესაც ენა იტანჯება, ერიც იტანჯება. ჩვენი ენა დიდი ხანია იბრძვის, მაგრამ მისი გადარჩენა მხოლოდ განათლებული თაობების ხელით თუ იქნება შესაძლებელი.

**User:** რას ნიშნავს მამულის სიყვარული?
**Ilia:** მამულის სიყვარული ის არის, როცა ადამიანი თავის სიცოცხლეზე მეტად სამშობლოს კეთილდღეობას უფრთხის. ვინც ამას ვერ ხედავს, მას ერიც არ უყვარს და არც თავისი ღირსება გააჩნია.

## Training Details

### Training Data

[`Duchidzeg/ilia-persona-sharegpt`](https://huggingface.co/datasets/Duchidzeg/ilia-persona-sharegpt) — 3,154 ShareGPT-format conversations covering Ilia's essays, prose, and poetry, plus identity and theme reinforcement across categories: identity, character disambiguation, meta/AI probes, time-period grounding, language, homeland, education, own-works attribution, modern-topic deflection, and style.

### Method

QLoRA (4-bit NF4 with double quantization) with LoRA adapters on all linear projections. Loss is computed **only on assistant tokens** (system and user tokens masked to `-100`).

### LoRA Configuration

- Rank `r = 16`, `lora_alpha = 32`, `lora_dropout = 0.0`, `use_rslora = True`
- Target modules: `qkv_proj`, `o_proj`, `gate_up_proj`, `down_proj` (Phi-3 all-linear)
- Trainable parameters: **25,165,824** (0.63% of 3.97 B total)

### Preprocessing

- Chat template: ChatML (kona2's own via `apply_chat_template`)
- Max sequence length: 1024
- Response-only masking on `<|im_start|>assistant\n` … `<|im_end|>` spans

### Training Hyperparameters

| Parameter | Value |
|---|---|
| Precision | bf16 mixed (compute), NF4 4-bit (storage) |
| Per-device batch size | 1 |
| Gradient accumulation | 8 (effective batch = 8) |
| Epochs | 3 |
| Learning rate | 1.5e-4, cosine schedule |
| Warmup | ~10% of steps per epoch |
| Weight decay | 0.01 |
| Optimizer | `adamw_bnb_8bit` |
| Gradient checkpointing | on (non-reentrant) |
| NEFTune noise α | 5 |
| Seed | 3407 |

### Speeds, Sizes, Times

- Total training steps: **1,125** (3 × 375 update steps/epoch)
- Hardware: 1 × NVIDIA RTX 5060 (8 GB VRAM, Blackwell sm_120, WDDM)
- Wall time: ~100 minutes
- Peak VRAM during training: ~6 GB
- Adapter size: ~50 MB (safetensors, fp16)

## Evaluation

### Held-Out Loss

5% random split (158 samples), same response-only masking as training.

- **Best eval loss: 2.32**
- Training loss at best checkpoint: 2.40
- Best checkpoint selected via `load_best_model_at_end` on `eval_loss`

## Environmental Impact

- **Hardware:** 1 × NVIDIA RTX 5060 (145 W TDP)
- **Duration:** ~1.7 h
- **Cloud provider:** N/A (local training)
- **Region:** Tbilisi, Georgia
- **Estimated energy:** ~0.25 kWh per training run

## Technical Specifications

### Model Architecture

- **Base:** Phi-3.5-mini-instruct architecture — 32 layers, 32 attention heads, 3072 hidden dim, extended vocabulary to 51,997 tokens for Georgian
- **Objective:** Causal language modeling with response-only loss masking (SFT)
- **Adapter:** LoRA rank-16 with RSLoRA scaling on all linear projections

### Software

- Python 3.12.10
- PyTorch 2.11.0+cu128
- transformers 5.13.1
- peft 0.19.1
- bitsandbytes 0.49.2
- datasets 5.0.0
- accelerate 1.14.0

## Citation

```bibtex
@misc{duchidze2026kona2ilia,
  title  = {kona2-ilia: An Ilia Chavchavadze persona LoRA adapter for kona2-small-3.8B},
  author = {Duchidze, Giorgi},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Duchidzeg/kona2-ilia}},
}
```

## Model Card Authors

Giorgi Duchidze ([@Duchidzeg](https://huggingface.co/Duchidzeg))

## Contact

Open an issue or discussion on the [model repository](https://huggingface.co/Duchidzeg/kona2-ilia/discussions).