---
language:
- pt
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: inaciose/inacio-d16v1
tags:
- nanochat
- nanochatpt
- portuguese
- pt-pt
- instruction-tuned
- sft
- conversational
- european-portuguese
---

# inacio-sft-d16v1

`inacio-sft-d16v1` is the first instruction-tuned model of the **nanochatpt** project. It is obtained by supervised fine-tuning (**SFT**) of the base model **inacio-d16v1**, with the objective of improving conversational abilities while maintaining European Portuguese (PT-PT) as the primary language.

This release represents the first end-to-end demonstration of the complete nanochatpt training pipeline, from tokenizer training to pretraining and supervised fine-tuning.

## Base Model

This model is fine-tuned from:

- **inaciose/inacio-d16v1**

The base model was pretrained from scratch on a corpus primarily composed of European Portuguese text, together with a smaller proportion of English.

## Model Architecture

- Architecture: NanoChat
- Decoder-only Transformer
- Parameters: **872,416,058**
- Layers: 16
- Hidden size: 1024
- Attention heads: 8
- KV heads: 8
- Context length: 2048 tokens
- Vocabulary: 65,536 tokens

## Supervised Fine-Tuning Dataset

The instruction tuning stage uses NanoChat's TaskMixture framework.

Training data consists of:

- CAIS MMLU (English) 
- OpenAI GSM8K (English) 
- inaciose/smol-smoltalk-pt-v1 (Mix European Portuguese 80%, English 20%)

The Portuguese conversational dataset **smol-smoltalk-pt-v1** was created specifically for this project as a temporary PT-PT replacement for HuggingFaceTB/smol-smoltalk.

It is derived from:

- AMALIA-LLM-0626-SFT-Dataset

Future releases will progressively replace the remaining English instruction datasets with Portuguese equivalents.

 https://huggingface.co/datasets/cais/mmlu
 https://huggingface.co/datasets/openai/gsm8k
 https://huggingface.co/datasets/inaciose/smol-smoltalk-pt-v1

## Training

Supervised fine-tuning was performed on the Modal cloud platform using:

- 4 × NVIDIA A100 SXM4 80GB

The model was initialized directly from the pretrained checkpoint of **inacio-d16v1**.

## Intended Uses

This model is intended for:

- conversational assistants
- experimentation
- educational purposes
- research on European Portuguese language models

## Current Capabilities

The model is capable of:

- maintaining multi-turn conversations
- following simple instructions
- generating structured answers
- responding primarily in Portuguese

Compared to the base model, it produces considerably more natural conversational responses and follows instruction formats learned during supervised fine-tuning.

## Current Limitations

This is the first public SFT release of nanochatpt.

The model still presents significant limitations, including:

- factual hallucinations
- weak mathematical reasoning
- inconsistent scientific explanations
- occasional repetition
- occasional generation of special conversation tokens (e.g. `<|assistant_end|>`)

These behaviours are expected for a first-generation instruction-tuned model trained on a relatively small Portuguese conversational corpus.

Future releases may improve both the instruction dataset and the amount of Portuguese pretraining data.

## Requeriments

nanochatpt or nanochat cloned to your machine, install requirements.
https://github.com/inaciose/nanochatpt

copy the model to your local folder
copy the tokenizer to your local folder

cache/modal
├── chatsft_checkpoints
│   └── d16
│       ├── meta_0XXXXX.json
│       ├── model_0XXXXX.pt
│       ├── optim_0XXXXX_rank0.pt
│       ├── optim_0XXXXX_rank1.pt
│       ├── optim_0XXXXX_rank2.pt
│       └── optim_0XXXXX_rank3.pt
└── tokenizer
    ├── token_bytes.pt
    └── tokenizer.pkl

export NANOCHAT_BASE_DIR=cache/modal

run the python -m scripts/chat_cli.py

## Project

nanochatpt is an personal open research project whose objective is to train compact language models from scratch with European Portuguese as the primary language.

Future work includes:

- larger pretraining corpora
- improved Portuguese instruction datasets
- Portuguese evaluation benchmarks
- additional supervised fine-tuning
- reinforcement learning alignment