---
license: mit
tags:
- gemma3
- safetensors
- transformers
- tinygemma
- tinystories
- validation
- test-suite
---

# TinyStories Gemma3 Text Validation Artifact

This directory contains a tiny Gemma 3 text-only model trained with official
Hugging Face Transformers classes.

It is intended for inference-engine validation, not for production language
quality.

## Official classes used

- `Gemma3TextConfig`
- `Gemma3ForCausalLM`
- `Trainer`

No custom Gemma 3 modeling code is used.

## Key validation targets

- `model_type = gemma3_text`
- `architectures = Gemma3ForCausalLM`
- local/global attention pattern through `layer_types`
- sliding-window attention
- full attention
- GQA
- per-head `q_norm` / `k_norm`
- Gemma3 four-norm decoder layer structure
- gated MLP: `silu(gate_proj(x)) * up_proj(x)`
- tied output head through `model.embed_tokens.weight`

## Tiny architecture

- vocab_size: 1024
- hidden_size: 128
- intermediate_size: 512
- num_hidden_layers: 6
- num_attention_heads: 4
- num_key_value_heads: 1
- head_dim: 32
- sliding_window: 32
- layer_types: ['sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention']

## Files

- `hf/`: Hugging Face model/tokenizer artifact
- `reference/reference.pt`: deterministic reference tensors
- `reference/reference.json`: JSON summary of reference logits
- `gemma3_text_config_dump.json`: normalized config dump
- `safetensors_keys.json`: tensor names and shapes
- `artifact_metadata.json`: generation metadata

## Usage

```python
import torch
from transformers import Gemma3ForCausalLM, PreTrainedTokenizerFast

model_dir = "hf"
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_dir)
model = Gemma3ForCausalLM.from_pretrained(model_dir)
model.eval()

ids = [tokenizer.bos_token_id] + tokenizer.encode("Once upon", add_special_tokens=False)
input_ids = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    out = model.generate(input_ids=input_ids, max_new_tokens=64, do_sample=False)

print(tokenizer.decode(out[0], skip_special_tokens=True))
```