README.md

---
base_model:
- LVSTCK/domestic-yak-8B
datasets:
- LVSTCK/sft-mk
- LVSTCK/Open-Platypus-MK
- LVSTCK/ultrachat-sft-mk
language:
- mk
license: llama3.1
library_name: transformers
pipeline_tag: text-generation
tags:
- mkd
- mk
- macedonia
---

# 🐂 domestic-yak, a Macedonian LM (instruct version)

This repository contains the model of the paper [Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language](https://huggingface.co/papers/2506.09560).

Code: https://github.com/LVSTCK

## Model Summary
This is the **instruct-tuned version** of [domestic-yak-8B](https://huggingface.co/LVSTCK/domestic-yak-8B). It has been fine-tuned specifically to improve instruction-following capabilities in Macedonian. It was fine-tuned on the [sft-mk dataset](https://huggingface.co/datasets/LVSTCK/sft-mk) for three epochs. Building on the foundation of `domestic-yak-8B`, this version is optimized for generating coherent, task-specific responses to user queries, making it ideal for chatbots, virtual assistants, and other interactive applications.

### 📊 Results
The table below compares the performance of our model, domestic-yak-8B-instruct with 4 other models. As we can see our model is on par with Llama 70B, and even beats it on three of the benchmarks. It is also worth noting that this model is currently the best in the 8B parameter range.

The results were obtained using the [macedonian-llm-eval](https://github.com/LVSTCK/macedonian-llm-eval) benchmark.

![image/png](https://cdn-uploads.huggingface.co/production/uploads/633206606eae0bb0a01c8a82/YsjmNbiXgQuvn26pQupDF.png)

wn.png)

## 🔑 Key Details
- **Language:** Macedonian (`mk`)
- **Base Model:** [domestic-yak-8B](https://huggingface.co/LVSTCK/domestic-yak-8B)
- **Dataset:** ~100k samples across multiple categories (Question answering (QA), chat-like conversations, reasoning, essays, and code) consolidated from translating publicly available datasets and custom synthetic data. Dataset can be found [here](https://huggingface.co/datasets/LVSTCK/sft-mk).
- **Fine-tuning Objective:** Supervised fine-tuning (SFT) on Macedonian-specific instruction-following data


## Usage 
Pipeline automatically uses apply_chat_template which formats the input appropriately. The model was trained using the default Llama 3.1 [format](https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/).

```python
import transformers
import torch

model_id = "LVSTCK/domestic-yak-8B-instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "Ти си виртуелен асистент кој помага на корисници на македонски јазик. Одговарај на прашања на јасен, разбирлив и професионален начин. Користи правилна граматика и обиди се одговорите да бидат што е можно покорисни и релевантни."},
    {"role": "user", "content": "Кој е највисок врв во Македонија?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256, # You can increase this
    temperature=0.1,
)
print(outputs[0]["generated_text"][-1])
```

## 📬 Contact

For inquiries, feedback, or contributions, please feel free to reach out to the core team:

- [Stefan Krsteski](https://www.linkedin.com/in/stefan-krsteski-136abb235/) [📧](mailto:stefan.krsteski@gmail.com)
- [Matea Tashkovska](https://www.linkedin.com/in/matea-tashkovska-774603198/) [📧](mailto:matea_tas@yahoo.com)
- [Borjan Sazdov](https://www.linkedin.com/in/borjan-sazdov-4b2187211/) [📧](mailto:borjansazdov@yahoo.com)


## Citation
```
@article{krsteski2025towards,
  title={Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language},
  author={Krsteski, Stefan and Tashkovska, Matea and Sazdov, Borjan and Gjoreski, Hristijan and Gerazov, Branislav},
  journal={arXiv preprint arXiv:2506.09560},
  year={2025}
}
```