---
language:
- ar
license: mit
tags:
- arabic
- nanochat
- causal-lm
- hpc
- research
- from-scratch
pipeline_tag: text-generation
library_name: pytorch
---

# nanochat-ar: Arabic LLM Proof Run on Aziz HPC

This repository contains a small Arabic nanochat proof-of-competence checkpoint trained from scratch on King Abdulaziz University Aziz HPC.

This is not a production Arabic LLM and does not claim state-of-the-art performance. The purpose is to document and release an end-to-end HPC training artifact: Arabic corpus processing, tokenizer training, base pretraining, evaluation, supervised fine-tuning, and checkpoint generation.

## What is included

- Arabic tokenizer: tokenizer/tokenizer.pkl and tokenizer/token_bytes.pt
- Base checkpoint: base_checkpoints/ar-d12-eager/model_000050.pt
- SFT checkpoint: chatsft_checkpoints/ar-d12-eager/model_000002.pt
- Metadata JSON files for both checkpoints
- Training/evaluation reports in reports/
- Final HPC scripts in scripts/
- 200-example SFT sample in data_samples/

Optimizer states and the full raw corpus are not uploaded.

## Training summary

| Stage | Result |
|---|---:|
| Arabic Wikipedia dump | 1,219,201 articles |
| Clean train split | 1,190,864 lines |
| Clean validation split | 12,028 lines |
| Tokenizer vocabulary | 32,768 |
| Base model | 12 layers, 286,261,730 parameters |
| Base checkpoint | ar-d12-eager, step 50 |
| Base validation BPB | 1.214387 |
| Base eval train BPB | 1.118719 |
| Base eval validation BPB | 1.214387 |
| SFT dataset | 2,000 generated Arabic instruction examples |
| SFT checkpoint | chatsft_checkpoints/ar-d12-eager, step 2 |
| SFT validation BPB | 1.0228 |

## How to load with nanochat

Clone karpathy/nanochat, install its environment, then use this repository as the NANOCHAT_BASE_DIR layout.

Commands:

    export NANOCHAT_BASE_DIR=/path/to/this/repo
    export NANOCHAT_DTYPE=float32
    cd /path/to/nanochat
    source .venv/bin/activate
    python -m scripts.chat_cli --device-type cpu --source sft --model-tag ar-d12-eager --step 2 --prompt 'مرحبا، من أنت؟'

For GPU inference, use an A100 or similar GPU and the standard nanochat chat CLI/web server.

## Limitations

- Very short proof training run: base model trained for only 50 steps.
- SFT is a compact generated instruction dataset, not a curated production instruction set.
- Chat quality is expected to be weak and repetitive.
- This release is best used as a reproducibility artifact and HPC workflow demonstration.

## Citation / Acknowledgement

This project used King Abdulaziz University Aziz HPC resources. Built on karpathy/nanochat under the MIT license.