---
language:
- pt
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: karpathy/nanochat
tags:
- nanochat
- nanochatpt
- portuguese
- pt-pt
- causal-lm
- pretraining
- european-portuguese
---

# inacio-d16v1

`inacio-d16v1` is the first public base model of the **nanochatpt** project, an effort to train compact language models from scratch with **European Portuguese (PT-PT)** as the primary language while retaining English competence.

The model is based on Andrej Karpathy's NanoChat architecture and was pretrained from scratch on a multilingual corpus deliberately biased towards Portuguese.

This release is a **base model**, intended for further supervised fine-tuning and research rather than direct conversational use.

## Model Architecture

- Architecture: NanoChat
- Decoder-only Transformer
- Parameters: **872,416,058**
- Layers: 16
- Hidden size: 1024
- Attention heads: 8
- KV heads: 8
- Context length: 2048 tokens
- Vocabulary: 65,536 tokens
- Activation: ReLU²
- Attention: Multi-Query Attention (MQA)
- Positional encoding: RoPE
- Window pattern: SSSL

## Pretraining Dataset

The model was trained on a custom corpus composed primarily of European Portuguese data.

Training data includes:

- DuarteoCarmo Bagaco (PT)
- Karpathy ClimbMix (EN)
- Europarl Portuguese
- Filtered PT-PT subset of CulturaX

The corpus intentionally mixes Portuguese 80% and English 20% to preserve multilingual capabilities while prioritizing PT-PT.

## Training

Training was performed on the Modal cloud platform.

Hardware:

- 4 × NVIDIA A100 SXM4 80GB

Training configuration:

- Depth: 16
- Batch size: 524,288 tokens
- Target token/parameter ratio: 12
- Total training tokens: 3.22B
- Total optimization steps: 6,144

Training completed successfully without divergence.

Final metrics:

- Minimum validation BPB: **0.769**
- CORE metric: **0.172**

## Training costs

- total 37$ - 30$ = 7$ (modal gives 30$ credit every months)

## Intended Uses

This model is intended for:

- continued pretraining
- supervised fine-tuning
- research
- experimentation
- educational purposes

It is **not intended** to be used directly as a chat assistant.

## Limitations

This is first complete training run of the nanochatpt project.

Although the model acquires a substantial amount of Portuguese linguistic knowledge during pretraining, it still exhibits important limitations typical of compact base models trained on relatively modest token budgets:

- factual inaccuracies
- arithmetic weaknesses
- reasoning limitations
- hallucinations
- incomplete world knowledge

These limitations are expected to improve with larger pretraining corpora and subsequent instruction tuning.

## Requeriments

nanochatpt or nanochat cloned to your machine, install requirements.
https://github.com/inaciose/nanochatpt

copy the model to your local folder
copy the tokenizer to your local folder

cache/modal
├── base_checkpoints
│   └── d16
│       ├── meta_0XXXXX.json
│       ├── model_0XXXXX.pt
│       ├── optim_0XXXXX_rank0.pt
│       ├── optim_0XXXXX_rank1.pt
│       ├── optim_0XXXXX_rank2.pt
│       └── optim_0XXXXX_rank3.pt
└── tokenizer
    ├── token_bytes.pt
    └── tokenizer.pkl

export NANOCHAT_BASE_DIR=cache/modal

## Project

nanochatpt is an personal open research project focused on training compact language models from scratch with European Portuguese as the primary language.

Future releases may include:

- larger pretraining corpora
- improved PT-PT datasets
- Portuguese evaluation benchmarks
- improved instruction tuning