RWKV-7 0.1B, converted from BlinkDL/rwkv-7-world for the in-tree transformers implementation

README.md

@@ -0,0 +1,103 @@
+---
+license: apache-2.0
+language:
+- en
+- zh
+library_name: transformers
+pipeline_tag: text-generation
+base_model: BlinkDL/rwkv-7-world
+tags:
+- rwkv
+- rwkv7
+- goose
+- linear-attention
+- recurrent
+---
+
+# RWKV-7 "Goose" 0.1B — converted for the in-tree `transformers` implementation
+
+This is **not a new model**. It is a format conversion of
+[`BlinkDL/rwkv-7-world`](https://huggingface.co/BlinkDL/rwkv-7-world) →
+`RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth`, laid out as a `transformers`
+directory so that the `rwkv7` implementation can load it with `from_pretrained`.
+All credit for the weights belongs to **BlinkDL / the RWKV project**; they are
+redistributed here under the Apache-2.0 licence they were released under.
+
+## Why this exists
+
+RWKV-7 weights are published in two layouts, and until now neither loaded into the
+in-tree implementation:
+
+- the reference `.pth` — the same parameter names, but a bare `torch.save` of a flat
+  dict rather than a `transformers` directory;
+- the `fla` layout — a `transformers` directory, but a `trust_remote_code` repo whose
+  modelling code and tensor names come from `flash-linear-attention`, so it loads its
+  own implementation.
+
+The conversion is a rename, not a transformation. Every one of the 399 tensors in the
+source `.pth` is **bit-identical** here; the only additions are three all-zero
+placeholders for layer 0's value-residual LoRA, which that layer never reads (layer 0
+*produces* `v_first` rather than mixing towards it) and which exist so the state dict
+is rectangular. `bfloat16`, the dtype the source is stored in.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# The RWKV world tokenizer; this repo ships weights only.
+tokenizer = AutoTokenizer.from_pretrained("RWKV/RWKV7-Goose-World2.8-0.1B-HF", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("Hakureirm/rwkv7-0.1b-hf", dtype=torch.bfloat16)
+
+inputs = tokenizer("The Eiffel Tower is located in the city of", return_tensors="pt")
+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=20)[0]))
+```
+
+RWKV-7 is attention-free and fully recurrent: the state is a fixed-size matrix per
+head, so there is no KV cache, memory is constant in context length, and each new
+token costs the same as the first.
+
+## Verification
+
+Checked against **BlinkDL's own runtime** — the `rwkv` package at `cpu fp32` with
+`RWKV_V7_ON=1`, not against the implementation being loaded here, which would be
+self-certifying, and not against `fla`, whose RWKV layer prints a warning on import
+saying it is potentially buggy and that results should be cross-checked against the
+official repository.
+
+Greedy, 20 tokens, from "The Eiffel Tower is located in the city of":
+
+| loaded as | tokens matching the reference runtime |
+|---|---|
+| `float32` | **20/20** |
+| `bfloat16` | **20/20** |
+
+Both produce `" Paris, France. It is the tallest building in the world and is the world's tallest"`.
+
+Logits agree with the reference to `7.06e-05` absolute and `1.97e-06` relative, with
+argmax agreement on every prompt position.
+
+0.1B is a useful size to check against for a reason beyond its download: it is 768 wide
+with `head_dim` 64, i.e. twelve heads of width 64, so the head *count* and the head
+*width* differ. At larger widths where both are 64, a quantity indexed by the wrong one
+of them agrees by coincidence.
+
+## Reproducing the conversion
+
+```bash
+huggingface-cli download BlinkDL/rwkv-7-world RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth --local-dir .
+python src/transformers/models/rwkv7/convert_rwkv7_checkpoint_to_hf.py \
+    --checkpoint RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
+    --flavour native --dtype bfloat16 --output_dir ./rwkv7-0.1b-hf
+```
+
+`--flavour fla` reads the safetensors layout instead. The converter compares the
+checkpoint's tensor shapes against the ones the config implies and refuses a
+disagreement, so a config that names a different model fails rather than producing
+something that loads and generates noise.
+
+## Citation
+
+The model is RWKV-7 "Goose" by Bo Peng (BlinkDL) and the RWKV community. Reference
+implementation: [BlinkDL/RWKV-LM](https://github.com/BlinkDL/RWKV-LM).

config.json

@@ -0,0 +1,33 @@
+{
+  "a_low_rank_dim": 64,
+  "architectures": [
+    "Rwkv7ForCausalLM"
+  ],
+  "bos_token_id": 0,
+  "decay_low_rank_dim": 64,
+  "deep_embed_size": 768,
+  "dtype": "bfloat16",
+  "eos_token_id": 0,
+  "gate_low_rank_dim": 128,
+  "head_dim": 64,
+  "hidden_act": "sqrelu",
+  "hidden_ratio": 4.0,
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "rwkv7",
+  "norm_bias": true,
+  "norm_eps": 1e-05,
+  "num_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "sparse_channel_mix": false,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.0.dev0",
+  "use_cache": true,
+  "use_deep_embed": false,
+  "v_low_rank_dim": 32,
+  "vocab_size": 65536,
+  "wkv_implementation": "eager",
+  "wkv_state_dtype": "float32"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:843e408a28782988b7be383991b49dba632559d85455d0658dfeb522da6b59cc
+size 382208592