pbarcelos1/nanochat-pt-latn-d12-sft

nanochat PT-Latn d12 — SFT fine-tuned on QA-PT (assistant-only loss mask)

This is a supervised fine-tuned (SFT) chat model. It was trained with an assistant-only loss mask on Portuguese QA data. It follows simple instruction / answer prompts using the nanochat chat template.

Forked from karpathy/nanochat and trained on ju-resplande/qa-pt (assistant tokens only).

Model details

Architecture	Decoder-only transformer (relu² FFN, RoPE, GQA, RMSNorm)
Layers	12
d_model	768
Attention heads	6 (KV heads: 6)
Sequence length	2048
Vocabulary	32,768 (byte-level BPE, PT-trained)
Trained steps	1,170
Final val bpb	0.2045
Training time	1.7 h on 1× NVIDIA RTX A6000

Usage

Load with nanochat:

import os
import torch
from huggingface_hub import snapshot_download
from nanochat.checkpoint_manager import build_model, find_last_step
from nanochat.engine import Engine

local_dir = snapshot_download(repo_id="pbarcelos1/nanochat-pt-latn-d12-sft")
os.environ["NANOCHAT_BASE_DIR"] = local_dir

device = torch.device("cuda:0")
step = find_last_step(local_dir)
model, tokenizer, meta = build_model(local_dir, step, device, phase="eval")

engine = Engine(model, tokenizer)
tokens = tokenizer("Era uma vez,", prepend="<|bos|>")
samples, _ = engine.generate_batch(tokens, num_samples=1, max_tokens=200, temperature=0.8)
print(tokenizer.decode(samples[0]))