---
license: apache-2.0
datasets:
- alibayram/tr-books
language:
- tr
---

# 🧠 LLaMA-50M Turkish Books 

## Model Summary
| Property | Value |
|-----------|--------|
| **Architecture** | LLaMA (decoder-only transformer) |
| **Parameters** | ~50M |
| **Vocab size** | 32,768 |
| **Embedding dim** | 256 |
| **Hidden dim** | 2048 |
| **Layers** | 20 |
| **Attention heads** | 128 |
| **KV groups** | 64 |
| **Context length** | 256 |
| **Tokenizer** | [turkish_tokenizer (alibayram)](https://github.com/malibayram/turkish-tokenizer) |
| **Dataset** | [alibayram/tr_books](https://huggingface.co/datasets/alibayram/tr_books) |
| **Tokens seen** | 372,679,971 |
| **Epochs** | 2 |
| **Batch size** | 64 |
| **Language** | Turkish 🇹🇷 |

---

## Model Description

`llama-50m-turkish-books` is a **50 million parameter Turkish language model** trained from scratch on the `alibayram/tr_books` dataset.  
It was developed as a lightweight experimental model to explore **Turkish-specific tokenization and morphology-aware pretraining** using the custom [turkish_tokenizer](https://github.com/malibayram/turkish-tokenizer).

The model follows the **LLaMA-style causal transformer** architecture and was trained with a context length of 256 tokens over ~372M tokens in total.

---

### Training Environment
Hardware: NVIDIA B200 GPU ~7 hours

```python