---
license: apache-2.0
language:
  - en
pipeline_tag: text-generation
tags:
  - pytorch
  - causal-lm
  - from-scratch
  - custom_code
  - gqa
  - rope
  - instruct
---

# G0-nano-instruct

A 62M-parameter GPT trained **completely from scratch on a single 8GB-RAM device** (an NVIDIA Jetson) — no cloud cluster, no multi-GPU node. Pretraining and instruction fine-tuning both happened within that budget. This is the instruction-tuned (chat) checkpoint. See [`G0-nano-base`](https://huggingface.co/AZERDSQ/G0-nano-base) for the raw pretrained model.

## Architecture

Llama-style decoder-only transformer:

| | |
|---|---|
| Parameters | 62.1M (embeddings shared with LM head) |
| Layers | 12 |
| Hidden size | 640 |
| Attention | Grouped-Query Attention, 10 query heads / 2 KV heads, head_dim 64 |
| Position encoding | RoPE (θ=10000) |
| Feed-forward | SwiGLU, hidden 1728 |
| Normalization | RMSNorm |
| Context length | 1024 tokens (SFT'd at 512) |
| Vocabulary | 16,384 base + 4 chat tokens (`<\|user\|>`, `<\|assistant\|>`, `<\|end\|>`, `<\|system\|>`) |
| Precision | fp32 |

Pretrained on ~1.5B tokens of English web/book text, then instruction-tuned on a cleaned Alpaca dataset.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("AZERDSQ/G0-nano-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("AZERDSQ/G0-nano-instruct", trust_remote_code=True)

prompt = "<|user|>What is the capital of France?<|end|><|assistant|>"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60, do_sample=False)
print(tok.decode(out[0]))
```

`trust_remote_code=True` is required — this is a custom architecture (GQA + RoPE + SwiGLU), not one of the built-in `transformers` model types. Generation stops automatically at `<|end|>` (configured in `generation_config.json`).

## Benchmarks

Zero-shot, full test sets, via `lm-evaluation-harness`, checkpoint `G0-nano-instruct` (step 2375/2376). Compared against models at the closest available parameter scale — not against frontier LLMs, which sit in a different weight class entirely by construction (see Limitations).

![Benchmark comparison: G0-nano-instruct vs Pythia-70M, Pythia-160M, GPT-2-small](benchmarks.png)

G0-nano-instruct tracks Pythia-70M (closest parameter count) closely on common-sense tasks, and beats every model in this table on ARC-Easy and SciQ — likely a side effect of instruction tuning on a clean Q&A-formatted dataset (Alpaca) rather than raw scale. On knowledge-dense benchmarks (MMLU, GPQA) it sits at chance level, consistent with the ~2 bits/parameter ceiling on factual knowledge for a model this size ([Allen-Zhu & Li, ICLR'25](https://arxiv.org/abs/2404.05405)) — not a training failure, a hard capacity limit.

## Limitations

- 62M parameters caps factual knowledge hard — near-chance on MMLU/GPQA, expect confident, fluent, frequently wrong answers on knowledge-dense questions.
- 1024-token context (512 during SFT).
- English only, single-turn conversations only (one user turn, one assistant turn — the SFT data didn't cover multi-turn).
- The custom tokenizer/model code does not handle padded batched inference — single-sequence generation only.

## License

Apache 2.0. Weights only — this release does not include training code or data pipelines.