---
license: apache-2.0
language:
- en
- zh
library_name: transformers
pipeline_tag: text-generation
base_model: BlinkDL/rwkv-7-world
tags:
- rwkv
- rwkv7
- goose
- linear-attention
- recurrent
---

# RWKV-7 "Goose" World 1.5B — ready for the in-tree `transformers` implementation

This is **not a new model**. It is a format conversion of
[`BlinkDL/rwkv-7-world`](https://huggingface.co/BlinkDL/rwkv-7-world) →
`RWKV-x070-World-1.5B-v3-20250127-ctx4096.pth`, laid out as a `transformers` directory so the `rwkv7`
implementation can load it with `from_pretrained`. All credit for the weights
belongs to **BlinkDL / the RWKV project**; they are redistributed here under the
Apache-2.0 licence they were released under.

The conversion is a rename, not a transformation: every one of the 795 tensors
in the source `.pth` is **bit-identical** here (verified tensor by tensor). The only
additions are three all-zero placeholders for layer 0's value-residual LoRA, which
that layer never reads (layer 0 *produces* `v_first` rather than mixing towards it).
`bfloat16`, the dtype the source is stored in. Training context length 4096.

## Usage

The implementation ships inside this repo (`auto_map` remote code), so stock
`transformers` is all you need — no fork, no extra package:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("RWKV/RWKV7-Goose-World2.8-0.1B-HF", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Hakureirm/rwkv7-1.5b-hf", trust_remote_code=True, dtype=torch.bfloat16)

inputs = tokenizer("The Eiffel Tower is located in the city of", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=20)[0]))
```

(The tokenizer is the RWKV "world" tokenizer, shared by every model in the family,
so the 0.1B repo above is just a convenient place to load it from.)

The same model class is also open as an in-tree PR
([rwkv-rs/transformers-rwkv#2](https://github.com/rwkv-rs/transformers-rwkv/pull/2));
this repo is the way to use it today on released transformers.

## Runs on

Pure-PyTorch portable path, no CUDA-only code on it. The implementation is verified
greedy 20/20 against BlinkDL's own runtime on **CPU**, **Apple Silicon MPS** and
**CUDA** (see [`Hakureirm/rwkv7-0.1b-hf`](https://huggingface.co/Hakureirm/rwkv7-0.1b-hf)
for that matrix and the logit-level comparison). Optional Triton kernels engage only
on CUDA and fall back to the portable path everywhere else.

RWKV-7 is attention-free and fully recurrent: the state is a fixed-size matrix per
head, so there is no KV cache, memory is constant in context length, and each new
token costs the same as the first.

## What was verified for this size

- all 795 source tensors **bit-identical** after conversion, none missing, none extra
  beyond the three zero placeholders named above;
- the converter's shape check: every tensor matches the shape the inferred config
  implies (24 layers, hidden 2048, 32 heads x 64, ffn 8192, LoRA ranks w/a/v/g 96/96/64/256);
- a bf16 CUDA greedy smoke run. `"The Eiffel Tower is located in the city of"` continues:

  > The Eiffel Tower is located in the city of Paris, France. It is the most famous landmark in the world and is a

The logit-level check against BlinkDL's own runtime was run on the 0.1B conversion
(same converter, same implementation), not repeated per size.

## Reproducing the conversion

```bash
huggingface-cli download BlinkDL/rwkv-7-world RWKV-x070-World-1.5B-v3-20250127-ctx4096.pth --local-dir .
python src/transformers/models/rwkv7/convert_rwkv7_checkpoint_to_hf.py \
    --checkpoint RWKV-x070-World-1.5B-v3-20250127-ctx4096.pth \
    --flavour native --dtype bfloat16 --output_dir ./rwkv7-1.5b-hf
```

The converter compares the checkpoint's tensor shapes against the ones the config
implies and refuses a disagreement, so a config that names a different model fails
rather than producing something that loads and generates noise.

## Citation

The model is RWKV-7 "Goose" by Bo Peng (BlinkDL) and the RWKV community. Reference
implementation: [BlinkDL/RWKV-LM](https://github.com/BlinkDL/RWKV-LM).