Upload folder using huggingface_hub

README.md

@@ -3,32 +3,23 @@ license: mit
 tags:
   - educational
   - from-scratch
-  - character-level
+  - char-level
   - transformer
   - pytorch
 ---
 
-# SIL-v01 — Tiny From-Scratch Language Model
+# tinyshakespeare -- char tokenizer, n_embd=128
 
 **This is an educational project, not a practical language model.** It is a
-from-scratch, char-RNN-scale transformer built to learn the mechanics of
-language modeling end-to-end: tokenization → embeddings → the
-language-modeling objective → training loop → self-attention → subword
-tokenization. It is not intended for any downstream or production use —
+from-scratch transformer built to learn the mechanics of language modeling
+end-to-end. It is not intended for any downstream or production use --
 treat it as a worked example, not a tool.
 
-The published checkpoint here is the **best-performing configuration found
-in a controlled scaling sweep**: a 4-layer, 4-head, causal self-attention
-transformer with `n_embd=128`, trained on a character-level tokenizer.
-
 ## Model architecture
 
-Token + position embeddings → 4 stacked pre-norm transformer blocks
-(multi-head causal self-attention + feedforward, residual connections) →
-final LayerNorm → linear head to vocab logits. Same family as a minimal
-GPT, at a scale that trains on a CPU in minutes. Full hyperparameters are
-in `config.json`; nothing about the architecture is hardcoded in the loader
-— see `load_model.py`.
+Token + position embeddings -> n_layer stacked transformer blocks (pre-norm multi-head causal self-attention + feedforward, residual connections) -> final LayerNorm -> linear head to vocab logits. Full hyperparameters are in `config.json`;
+nothing about the architecture is hardcoded in the loader -- see
+`load_model.py`.
 
 | | |
 |---|---|
@@ -37,21 +28,11 @@ in `config.json`; nothing about the architecture is hardcoded in the loader
 | n_head | 4 |
 | n_layer | 4 |
 | block_size | 32 |
-| Tokenizer | character-level, vocab_size=65 |
+| Tokenizer | char-level, vocab_size=65 |
 
 ## Training corpus
 
-Trained on **TinyShakespeare** (~1.1M characters), the classic small corpus
-for character-level language modeling. It is fetched directly from Andrej
-Karpathy's `char-rnn` GitHub repository
-(`raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt`)
-— a concatenation of Shakespeare's plays, ultimately public-domain text, but
-sourced from that repo directly rather than downloaded from Project
-Gutenberg. (This project's data pipeline also includes a Project Gutenberg
-boilerplate-stripper for *other* corpora it experiments with — Dostoevsky,
-Nietzsche, Suetonius — but that path was not used for this checkpoint.)
-
-90/10 train/val split, held fixed across every run below.
+Trained on **TinyShakespeare** (~1.1M characters), fetched directly from `https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt`.
 
 ## Results
 
@@ -93,30 +74,30 @@ run here.
 
 | File | Purpose |
 |---|---|
-| `model.pt` | `state_dict()` of the trained `BigramLanguageModel` |
+| `model.safetensors` | `state_dict()` of the trained `BigramLanguageModel`, in safetensors format |
 | `config.json` | Every hyperparameter needed to reconstruct the architecture and tokenizer |
-| `tokenizer_char.json` | The character-level tokenizer's `stoi` vocabulary |
-| `bigram.py` | Model architecture source (vendored so this repo is self-sufficient) |
-| `tokenizer.py` | Character tokenizer source (vendored, same reason) |
+| `tokenizer_char.json` | The char-level tokenizer's vocabulary |
+| `bigram.py` | Model architecture source (vendored so this folder is self-sufficient) |
+| `tokenizer.py` | Tokenizer source (vendored, same reason) |
 | `load_model.py` | Reconstructs the model + tokenizer from `config.json` and generates a sample |
 
 ## Usage
 
 ```bash
-pip install torch
+pip install torch safetensors
 python load_model.py
 ```
 
-`load_model.py` reads `config.json` for every architectural parameter
-(`vocab_size`, `n_embd`, `n_head`, `n_layer`, `block_size`, `tokenizer_type`)
-— it does not assume or hardcode them — loads `model.pt` into a freshly
-constructed model, and samples 400 characters to prove the checkpoint and
-its config agree.
+`load_model.py` reads `config.json` for every architectural parameter --
+it does not assume or hardcode them -- loads `model.safetensors`
+into a freshly constructed model, and samples 400 characters to prove the
+checkpoint and its config agree.
 
 ## Limitations
 
-This is a ~65-character-vocabulary, 32-token-context toy model trained for
-8,000 steps on 1MB of text. It reproduces surface Shakespeare-ish texture
-(names, verse-like line breaks, archaic diction) but not coherent meaning,
-plot, or factual content. **Do not use this for anything beyond studying
-how these mechanics fit together.**
+This is a ~65-char-vocabulary,
+32-token-context toy model trained for
+8,000 steps. It reproduces surface
+Shakespeare-ish texture but not coherent meaning, plot, or factual
+content. **Do not use this for anything beyond studying how these
+mechanics fit together.**

config.json

@@ -8,11 +8,11 @@
   "block_size": 32,
   "tokenizer_type": "char",
   "tokenizer_file": "tokenizer_char.json",
-  "weights_file": "model.pt",
+  "weights_file": "model.safetensors",
   "training": {
     "dataset": "tinyshakespeare.txt",
     "batch_size": 32,
-    "learning_rate": 1e-3,
+    "learning_rate": 0.001,
     "max_iters": 8000,
     "seed": 1337
   },

load_model.py

@@ -1,11 +1,10 @@
 """
 Self-contained loader for this checkpoint.
 
-Reconstructs the model architecture and tokenizer entirely from config.json
-(no hardcoded hyperparameters here), loads the trained weights, and
-generates a sample -- proving the artifact set (config.json + model.pt +
-tokenizer_char.json + bigram.py + tokenizer.py) is sufficient on its own,
-with no dependency on the original training repo or its train.py.
+Reconstructs the model architecture and tokenizer entirely from
+config.json (no hardcoded hyperparameters here), loads the trained
+weights, and generates a sample -- proving this folder is sufficient on
+its own, with no dependency on the original training repo.
 
 Usage:
     python load_model.py
@@ -19,19 +18,28 @@ from pathlib import Path
 import torch
 
 from bigram import BigramLanguageModel
-from tokenizer import CharTokenizer
 
 HERE = Path(__file__).resolve().parent
 
 
+def _load_weights(path: Path) -> dict[str, torch.Tensor]:
+    if path.suffix == ".safetensors":
+        from safetensors.torch import load_file
+
+        return load_file(path)
+    return torch.load(path, map_location="cpu", weights_only=True)
+
+
 def load_model_and_tokenizer(dir_path: Path = HERE):
     config = json.loads((dir_path / "config.json").read_text())
 
-    if config["tokenizer_type"] != "char":
-        raise ValueError(
-            f"This loader only wires up 'char'; config says {config['tokenizer_type']!r}"
-        )
-    tokenizer = CharTokenizer.load(dir_path / config["tokenizer_file"])
+    if config["tokenizer_type"] == "char":
+        from tokenizer import CharTokenizer as TokenizerClass
+    elif config["tokenizer_type"] == "bpe":
+        from bpe_tokenizer import BPETokenizer as TokenizerClass
+    else:
+        raise ValueError(f"Unknown tokenizer_type: {config['tokenizer_type']!r}")
+    tokenizer = TokenizerClass.load(dir_path / config["tokenizer_file"])
 
     model = BigramLanguageModel(
         vocab_size=config["vocab_size"],
@@ -40,7 +48,7 @@ def load_model_and_tokenizer(dir_path: Path = HERE):
         n_head=config["n_head"],
         n_layer=config["n_layer"],
     )
-    state_dict = torch.load(dir_path / config["weights_file"], map_location="cpu")
+    state_dict = _load_weights(dir_path / config["weights_file"])
     model.load_state_dict(state_dict)
     model.eval()
 

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce06ec1406abaa871d90b8fcc2ff4d008047fceea39dff3c75c542f617ca75ef
+size 3326268