Upload folder using huggingface_hub

README.md

@@ -12,13 +12,14 @@ tags:
 
 # TinyModel Mixtral 2M Top-3 MoE (tinyllama4gpt2m) HF Validation Suite
 
-This repository provides an ultra-lightweight Mixtral model variant scaled down to a 2M class total parameter footprint. It is optimized specifically as a precise validation asset for custom inference environments, hardware tensor compilers, and execution scheduling verification.
+This repository provides an ultra-lightweight **Llama 4 format model variant** scaled down to a 2M class total parameter footprint, explicitly utilizing a **GPT-style tokenizer**.
 
-This asset is calibrated to a 1,024 token context window (1k) utilizing standard positional embeddings. It is designed to isolate and debug dynamic routing mechanics, expert tensor scheduling, and memory dispatch loops without prohibitive compute costs or training latency.
+### 🎯 Primary Validation Objective: GPT-Style Tokenizer Verification (Core Purpose)
+The foundational purpose of this entire suite is to isolate and verify the exact mathematical and structural behavior of a **GPT-style Byte-level BPE Tokenizer** (configured with `add_prefix_space=True`). Testing tokenizer compliance on large models introduces unnecessary complexity. This 2M parameter configuration allows developers to ensure their text-to-token transformation, token-to-text decoding, and prefix-space fusion rules perfectly match the reference implementation, with immediate visibility into alignment results.
 
-With this iteration, the model transitions from a standard Multi-Head Attention (MHA) layout to a **Grouped-Query Attention (GQA)** configuration (4 Query Heads to 2 Key/Value Heads). This specific structure is engineered to isolate, profile, and validate index mapping calculations, group-to-head allocations, and KV cache memory strides within custom attention architectures.
+**If your custom inference backend handles prefix spaces, boundary word fusions, or byte-level fallbacks incorrectly, the token IDs emitted here will immediately drift from the PyTorch reference baseline, isolating tokenization anomalies before tensor computations even begin.**
 
-It remains highly optimized for debugging sparse Mixture-of-Experts (MoE) features, including gating weight allocation, token scatter/gather loops, and expert output synthesis.
+In addition to tokenizer verification, this asset is calibrated to a 1,024 token context window utilizing **Llama 3 RoPE Scaling** (4.0x factor over a 256 base window), providing a comprehensive test bed for both text processing and advanced position embedding calculations.
 
 ---
 
@@ -28,19 +29,21 @@ It remains highly optimized for debugging sparse Mixture-of-Experts (MoE) featur
 Unquantized components formatted for direct instantiation inside the PyTorch `transformers` library ecosystem or compatible proprietary model parsers:
 
 * **`hf/model.safetensors`**: Raw unquantized matrix parameters containing all 5 expert sub-networks alongside the master router tensor (Gate) and GQA projection layers.
-* **`hf/config.json`**: Architectural specifications built around `MixtralConfig` criteria, explicitly enforcing `num_attention_heads: 4`, `num_key_value_heads: 2`, and `max_position_embeddings: 1024`.
+* **`hf/config.json`**: Architectural specifications built around `MixtralConfig` criteria, explicitly enforcing `num_attention_heads: 4`, `num_key_value_heads: 2`, `max_position_embeddings: 1024`, and the `llama3` type `rope_scaling` parameters.
 * **`hf/generation_config.json`**: Standard generation defaults for greedy search boundaries.
-* **`hf/tokenizer.json`**: The core Byte-level BPE tokenizer layout containing vocabulary indices, pre-tokenization rules, and the merges map.
+* **`hf/tokenizer.json`**: The core Byte-level BPE tokenizer layout (configured with GPT-style `add_prefix_space=True`) containing vocabulary indices, pre-tokenization rules, and the merges map.
 * **`hf/tokenizer.model`**: A structural dummy file provided exclusively to maintain complete Llama/Mixtral asset footprint compatibility with legacy reference loaders.
 * **`hf/tokenizer_config.json`**: Metadata managing tokenization classes to guarantee correct handling of prefix spacing and automatic `<s>` (BOS) injection properly on the execution backend.
 
 ---
 
-## 🎯 Purpose & Design Philosophy (Verification Targets)
+## 📂 Purpose & Design Philosophy (Verification Targets)
 
 This checkpoint is engineered strictly as a deterministic validation test asset for computing platforms and custom inference environments. 
 
 Due to the compact vocabulary layout (4,000 tokens) and highly localized layer structure, it provides an ideal environment to isolate and profile specific compute structures:
+* **GPT-Style Tokenization Mechanics**: Validates that the word-boundary space management (`add_prefix_space=True`) and byte-level fallback merging match the GPT-2/Llama ecosystem exactly. This isolates subtokens layout anomalies before text data interacts with embedding layers.
+* **Llama 3 RoPE Scaling Verification**: Validating multi-band frequency adjustments (`factor=4.0`, `low_freq_factor=1.0`, `high_freq_factor=4.0`, `original_max_position_embeddings=256`). This verifies whether the custom inference engine correctly bifurcates dimensions into high, medium, and low frequency bands and scales them accurately across an expanded 1,024-token sequence.
 * **GQA Routing & Index Mapping**: Verifying the group indexing logic where 4 query heads resolve to 2 distinct key/value head pairs, exposing stride offsets and boundary errors in attention loops.
 * **Dynamic Routing Isolation**: Validating Top-3 gating allocation vectors and tracking row-index distribution matrices inside custom execution topologies.
 * **Scatter/Gather Verification**: Profiling the memory dispatch loops that split token matrices into independent expert segments and synthesize them back into the main residual stream.
@@ -48,35 +51,37 @@ Due to the compact vocabulary layout (4,000 tokens) and highly localized layer s
 
 ---
 
-## 🚀 Usage Examples
+## 📂 Usage Examples
 
 ### Loading Hugging Face Formats via Python
 
-Because the configuration parameters are seamlessly matched with the standard Transformers schema, you can invoke the classes using automated loaders by pointing directly to the local directory path.
+Because the configuration parameters are seamlessly matched with the standard Transformers schema, you can invoke the classes using automated loaders by pointing directly to the Hugging Face repository and subfolder.
 
 ```python
 import torch
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
-# Path to the local directory containing your 'hf' folder
-model_path = "./tinyllama4gpt2m/hf"
+# Target repository and subfolder configuration
+repo_id = "shibatch/tinyllama4gpt2m"
+subfolder = "hf"
 
-print("Loading MoE GQA configuration and tokenizer layers...")
-tokenizer = AutoTokenizer.from_pretrained(model_path)
-model = AutoModelForCausalLM.from_pretrained(model_path)
+print("Loading MoE GQA configuration and GPT-style tokenizer layers from Hugging Face...")
+tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=subfolder)
+model = AutoModelForCausalLM.from_pretrained(repo_id, subfolder=subfolder)
 
 device = "cuda" if torch.cuda.is_available() else "cpu"
 model = model.to(device)
 model.eval()
 
 prompt = "Once upon"
+# Tokenize using the loaded GPT-style configuration
 inputs = tokenizer(prompt, return_tensors="pt").to(device)
 
-print("Running inference loop (Validating Top-3 sparse routing and 4:2 GQA matrices)...")
+print("Running inference loop (Validating GPT-style Tokenizer, Top-3 routing, GQA, and Llama3 RoPE Scaling)...")
 with torch.no_grad():
     outputs = model.generate(
         **inputs, 
-        max_new_tokens=20, 
+        max_new_tokens=100, 
         do_sample=False
     )
     
@@ -90,22 +95,21 @@ print("Generated:", generated_text)
 
 ---
 
-## 📝 Model Specifications
+## 📂 Model Specifications
 
 * **Architecture:** Mixtral (`MixtralForCausalLM`)
 * **Dataset:** Language Modeling Corpora
 * **Total Parameters (`num_local_experts` = 5):** 2M class footprint
 * **Active Parameters (`num_experts_per_tok` = 3):** 1.18M active during dispatch
-* **Vocabulary Size (`vocab_size`):** 4,000 (Byte-level BPE with decoder tracking)
+* **Vocabulary Size (`vocab_size`):** 4,000 (Byte-level BPE with strict GPT-style `add_prefix_space=True` configuration)
 * **Hidden Size (`hidden_size`):** 96
 * **Number of Hidden Layers (`num_hidden_layers`):** 2
 * **Number of Attention Heads (`num_heads` / `num_kv_heads`):** 4 / 2 *(Grouped-Query Attention layout)*
 * **Individual Expert Internal Dimension (`intermediate_size`):** 192 *(SwiGLU structure)*
 * **Max Position Embeddings (`max_position_embeddings`):** 1,024
+* **RoPE Scaling (`rope_scaling`):** `{"type": "llama3", "factor": 4.0, "low_freq_factor": 1.0, "high_freq_factor": 4.0, "original_max_position_embeddings": 256}`
 * **RMS Norm Epsilon (`rms_norm_eps`):** 1e-5
 
-## 📜 License
+## 📂 License
 
 * **License:** **MIT License**. You are completely free to duplicate, modify, distribute, and utilize these assets across any commercial, personal, or educational environments.
-
-```

hf/config.json

@@ -22,8 +22,13 @@
   "pad_token_id": 2,
   "rms_norm_eps": 1e-05,
   "rope_parameters": {
+    "factor": 4.0,
+    "high_freq_factor": 4.0,
+    "low_freq_factor": 1.0,
+    "original_max_position_embeddings": 256,
     "rope_theta": 1000000.0,
-    "rope_type": "default"
+    "rope_type": "llama3",
+    "type": "llama3"
   },
   "router_aux_loss_coef": 0.001,
   "router_jitter_noise": 0.0,

hf/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2c0869d11b1b66c318a99866c3ebbda871f82b92c09de5c8d86d1dee1ce2ba64
+oid sha256:c1c9ffa9b6fb878519124dd655bc1dc344516e494d29c462b3d3a52e76eda2bb
 size 5516176