Continued Pretraining • 50M Parameters • 5K Context
 Supra-1.5-50M-Base-exp is a continued-pretrained 50M parameter Llama-style base model derived from `SupraLabs/Supra-50M-Base`. The context window has been expanded from 1,024 to 5,120 tokens using RoPE scaling and full-weight continued pretraining. This specific repository has been modified to natively support ChatML. The tokenizer and embedding layers have been updated to include `<|im_start|>` and `<|im_end|>` as single special tokens. ## Architecture & Updates | Specification | Value | |--------------|--------| | Architecture | `LlamaForCausalLM` | | Parameters | ~50M | | Vocabulary Size | 32,002 (Expanded for ChatML) | | Hidden Size | 512 | | Layers | 12 | | Attention Heads | 8 | | KV Heads | 4 | | Context Length | 5,120 tokens | | Tokenizer | BPE tokenizer (ChatML Jinja template applied) | | Native EOS Token | `<|im_end|>` | ## Vocabulary Expansion & Initialization The model's 3B token CPT corpus included raw ChatML text. As a result, the base model originally learned the tags `<|im_start|>` and `<|im_end|>` as sequences of 7 separate subwords. To convert these tags into single tokens without destroying the learned pre-trained representations, we used subword-mean initialization: 1. Extracted the attention embeddings for the original 7-token sequences. 2. Averaged the weights for each sequence. 3. Assigned these mean weights to the new single-token IDs. This ensures stable loss at the start of fine-tuning and allows inference frameworks to natively stop generation when `<|im_end|>` is predicted. ## Continued Pretraining Objective This is a base model, not an instruct model. Training used packed raw text with standard causal language-modeling loss: - `labels = input_ids` - all non-pad tokens are trained - no response-only masking - no system/user/assistant masking - no LoRA adapters in the default run ## Data Mix The training mix prepared for the CPT run was: - 3,000,000,062 CPT tokens - 30% Tool Calling - 30% ChatML Conversations - 25% Factual Text (articles, essays, blogs) - 15% Math & Logic Questions ## Fine-Tuning Guide This model is intended for Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), including reasoning-style (R1) alignments. You can use standard Hugging Face tools and call `tokenizer.apply_chat_template()` directly on your datasets. **LoRA Configuration Note:** Because the vocabulary size was expanded from 32,000 to 32,002, you must train the embedding layers during fine-tuning so the model can update the new ChatML tokens. Add the following to your PEFT/LoRA configuration: ```python modules_to_save=["embed_tokens", "lm_head"]