Initial upload: weightless (strict-mode) FlashNorm checkpoint

README.md

@@ -0,0 +1,86 @@
+---
+license: apache-2.0
+base_model: HuggingFaceTB/SmolLM2-135M
+tags:
+  - flashnorm
+  - transformer-tricks
+  - efficient-inference
+  - weightless-rmsnorm
+pipeline_tag: text-generation
+---
+
+# SmolLM2-135M-FlashNorm-strict
+
+**Weightless (strict-mode) FlashNorm checkpoint** of [HuggingFaceTB/SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M).
+
+Mathematically equivalent to the source model. The per-channel normalization weight tensors (`input_layernorm.weight`, `post_attention_layernorm.weight`, `model.norm.weight`) have been folded into the following linear layers and then removed from the state dict entirely.
+
+> **This checkpoint does NOT load in stock vLLM today.** vLLM's weight loader raises a `ValueError` because the norm weight tensors are absent. Issue tracking the loader patch: TBD. Use [open-machine/SmolLM2-135M-FlashNorm](https://huggingface.co/open-machine/SmolLM2-135M-FlashNorm) (the compat variant) for a drop-in checkpoint that loads in stock vLLM today.
+
+This repo exists as a concrete test vector for the upstream patch that would let vLLM accept weightless RMSNorm models.
+
+## What is FlashNorm (weightless)?
+
+An exact reformulation of `RMSNorm -> Linear`:
+
+- **Fold** the per-channel normalization weight `g` into the following linear layer: `W_star = W @ diag(g)`.
+- After folding, the RMSNorm layer has no learnable per-channel scale. It just divides by `rms(x)`.
+- The resulting model computes the same output as the original, by Proposition 1 of the FlashNorm paper.
+
+This repo is a "weightless" variant: the `g` tensor itself is absent from the safetensors, because after the fold the runtime value of `g` is always all-ones (the multiplicative identity). Deleting the tensor saves a small amount of disk space and makes explicit that the runtime never needs to multiply by `g`.
+
+See the [paper](https://github.com/OpenMachine-ai/transformer-tricks/blob/main/tex/flashNorm.tex) (Section 3.1 and Proposition 1) and the [transformer-tricks](https://github.com/OpenMachine-ai/transformer-tricks) repo for details.
+
+## What's different from the source checkpoint
+
+| Tensor | Source | Compat variant | This (strict) |
+|---|---|---|---|
+| `model.layers.*.input_layernorm.weight` | learned per-channel `g` | all ones | **absent** |
+| `model.layers.*.self_attn.{q,k,v}_proj.weight` | `W` | `W @ diag(g_input_layernorm)` | `W @ diag(g_input_layernorm)` |
+| `model.layers.*.post_attention_layernorm.weight` | learned per-channel `g` | all ones | **absent** |
+| `model.layers.*.mlp.{gate,up}_proj.weight` | `W` | `W @ diag(g_post_attention_layernorm)` | `W @ diag(g_post_attention_layernorm)` |
+| `model.norm.weight` | learned per-channel `g` | all ones | **absent** |
+
+All dtype conventions match the source (`bfloat16`). Mathematical identity to the source model holds by construction.
+
+## Usage
+
+### Via `transformer_tricks`
+
+The `transformer_tricks` package can regenerate this checkpoint locally from the source:
+
+```python
+import transformer_tricks as tt
+tt.flashify_repo('HuggingFaceTB/SmolLM2-135M', strict=True)
+```
+
+### Via HuggingFace Transformers
+
+HuggingFace Transformers will load this checkpoint with a warning that norm weights were not initialized from the checkpoint, and will default them to the module's init value (ones for `LlamaRMSNorm`). Under this path, the output is correct.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained('open-machine/SmolLM2-135M-FlashNorm-strict')
+model = AutoModelForCausalLM.from_pretrained('open-machine/SmolLM2-135M-FlashNorm-strict')
+
+ids = tok('Once upon a time there was', return_tensors='pt').input_ids
+out = model.generate(ids, max_new_tokens=50, do_sample=False)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+### Via vLLM
+
+**Not yet supported.** vLLM's weight loader validates that all declared `nn.Parameter` tensors are present in the safetensors and raises `ValueError` when norm weights are absent.
+
+Tracking issue for upstream patch: TBD (to be linked once filed).
+
+Until the patch lands, use [open-machine/SmolLM2-135M-FlashNorm](https://huggingface.co/open-machine/SmolLM2-135M-FlashNorm) (compat variant) which keeps the norm tensors as all-ones and loads in stock vLLM unchanged.
+
+## Verification
+
+Generated from the compat variant by deleting the 61 norm weight tensors (30 layers x 2 norms each + 1 final `model.norm`). All other tensors are byte-identical to the compat checkpoint; inference outputs are therefore identical when the loader defaults absent norm weights to ones.
+
+## License
+
+Apache-2.0, inherited from the source model.

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 576,
+  "initializer_range": 0.041666666666666664,
+  "intermediate_size": 1536,
+  "is_llama_config": true,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 9,
+  "num_hidden_layers": 30,
+  "num_key_value_heads": 3,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_interleaved": false,
+  "rope_parameters": {
+    "rope_theta": 100000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.4",
+  "use_cache": true,
+  "vocab_size": 49152
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1049412dc1bfef8ad396b910fb2243375aee9b9ad346790d95fd69a0e09a3afb
+size 268983424

special_tokens_map.json

@@ -0,0 +1,42 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,167 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 8192,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}