G0-nano-instruct

A 62M-parameter GPT trained completely from scratch on a single 8GB-RAM device (an NVIDIA Jetson) — no cloud cluster, no multi-GPU node. Pretraining and instruction fine-tuning both happened within that budget. This is the instruction-tuned (chat) checkpoint. See G0-nano-base for the raw pretrained model.

Architecture

Llama-style decoder-only transformer:

Parameters	62.1M (embeddings shared with LM head)
Layers	12
Hidden size	640
Attention	Grouped-Query Attention, 10 query heads / 2 KV heads, head_dim 64
Position encoding	RoPE (θ=10000)
Feed-forward	SwiGLU, hidden 1728
Normalization	RMSNorm
Context length	1024 tokens (SFT'd at 512)
Vocabulary	16,384 base + 4 chat tokens (<|user|>, <|assistant|>, <|end|>, <|system|>)
Precision	fp32

Pretrained on ~1.5B tokens of English web/book text, then instruction-tuned on a cleaned Alpaca dataset.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("AZERDSQ/G0-nano-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("AZERDSQ/G0-nano-instruct", trust_remote_code=True)

prompt = "<|user|>What is the capital of France?<|end|><|assistant|>"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60, do_sample=False)
print(tok.decode(out[0]))

trust_remote_code=True is required — this is a custom architecture (GQA + RoPE + SwiGLU), not one of the built-in transformers model types. Generation stops automatically at <|end|> (configured in generation_config.json).

Benchmarks

Zero-shot, full test sets, via lm-evaluation-harness, checkpoint G0-nano-instruct (step 2375/2376). Compared against models at the closest available parameter scale — not against frontier LLMs, which sit in a different weight class entirely by construction (see Limitations).

G0-nano-instruct tracks Pythia-70M (closest parameter count) closely on common-sense tasks, and beats every model in this table on ARC-Easy and SciQ — likely a side effect of instruction tuning on a clean Q&A-formatted dataset (Alpaca) rather than raw scale. On knowledge-dense benchmarks (MMLU, GPQA) it sits at chance level, consistent with the ~2 bits/parameter ceiling on factual knowledge for a model this size (Allen-Zhu & Li, ICLR'25) — not a training failure, a hard capacity limit.

Limitations

• 62M parameters caps factual knowledge hard — near-chance on MMLU/GPQA, expect confident, fluent, frequently wrong answers on knowledge-dense questions.
• 1024-token context (512 during SFT).
• English only, single-turn conversations only (one user turn, one assistant turn — the SFT data didn't cover multi-turn).
• The custom tokenizer/model code does not handle padded batched inference — single-sequence generation only.

License

Apache 2.0. Weights only — this release does not include training code or data pipelines.