---
license: apache-2.0
language:
- en
- zh
library_name: transformers
pipeline_tag: text-generation
base_model: BlinkDL/rwkv-7-world
tags:
- rwkv
- rwkv7
- goose
- linear-attention
- recurrent
---

# RWKV-7 "Goose" 0.1B — converted for the in-tree `transformers` implementation

This is **not a new model**. It is a format conversion of
[`BlinkDL/rwkv-7-world`](https://huggingface.co/BlinkDL/rwkv-7-world) →
`RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth`, laid out as a `transformers`
directory so that the `rwkv7` implementation can load it with `from_pretrained`.
All credit for the weights belongs to **BlinkDL / the RWKV project**; they are
redistributed here under the Apache-2.0 licence they were released under.

## Why this exists

RWKV-7 weights are published in two layouts, and until now neither loaded into the
in-tree implementation:

- the reference `.pth` — the same parameter names, but a bare `torch.save` of a flat
  dict rather than a `transformers` directory;
- the `fla` layout — a `transformers` directory, but a `trust_remote_code` repo whose
  modelling code and tensor names come from `flash-linear-attention`, so it loads its
  own implementation.

The conversion is a rename, not a transformation. Every one of the 399 tensors in the
source `.pth` is **bit-identical** here; the only additions are three all-zero
placeholders for layer 0's value-residual LoRA, which that layer never reads (layer 0
*produces* `v_first` rather than mixing towards it) and which exist so the state dict
is rectangular. `bfloat16`, the dtype the source is stored in.

## Usage

The implementation ships inside this repo (`auto_map` remote code), so stock
`transformers` is all you need — no fork, no extra package:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("RWKV/RWKV7-Goose-World2.8-0.1B-HF", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Hakureirm/rwkv7-0.1b-hf", trust_remote_code=True, dtype=torch.bfloat16)

inputs = tokenizer("The Eiffel Tower is located in the city of", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=20)[0]))
```

The same model class is also open as an in-tree PR
([rwkv-rs/transformers-rwkv#2](https://github.com/rwkv-rs/transformers-rwkv/pull/2));
this repo is the way to use it today on released transformers.

## Runs on

Pure-PyTorch portable path, no CUDA-only code on it. Verified, greedy 20/20 against
BlinkDL's own runtime in each case: **CPU** (fp32/bf16, transformers 5.12.1),
**Apple Silicon MPS** (fp32 and fp16), **CUDA** (the full test suite). Optional Triton
kernels engage only on CUDA and fall back to the portable path everywhere else.

RWKV-7 is attention-free and fully recurrent: the state is a fixed-size matrix per
head, so there is no KV cache, memory is constant in context length, and each new
token costs the same as the first.

## Verification

Checked against **BlinkDL's own runtime** — the `rwkv` package at `cpu fp32` with
`RWKV_V7_ON=1`, not against the implementation being loaded here, which would be
self-certifying, and not against `fla`, whose RWKV layer prints a warning on import
saying it is potentially buggy and that results should be cross-checked against the
official repository.

Greedy, 20 tokens, from "The Eiffel Tower is located in the city of":

| loaded as | tokens matching the reference runtime |
|---|---|
| `float32` | **20/20** |
| `bfloat16` | **20/20** |

Both produce `" Paris, France. It is the tallest building in the world and is the world's tallest"`.

Logits agree with the reference to `7.06e-05` absolute and `1.97e-06` relative, with
argmax agreement on every prompt position.

0.1B is a useful size to check against for a reason beyond its download: it is 768 wide
with `head_dim` 64, i.e. twelve heads of width 64, so the head *count* and the head
*width* differ. At larger widths where both are 64, a quantity indexed by the wrong one
of them agrees by coincidence.

## Reproducing the conversion

```bash
huggingface-cli download BlinkDL/rwkv-7-world RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth --local-dir .
python src/transformers/models/rwkv7/convert_rwkv7_checkpoint_to_hf.py \
    --checkpoint RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
    --flavour native --dtype bfloat16 --output_dir ./rwkv7-0.1b-hf
```

`--flavour fla` reads the safetensors layout instead. The converter compares the
checkpoint's tensor shapes against the ones the config implies and refuses a
disagreement, so a config that names a different model fails rather than producing
something that loads and generates noise.

## Citation

The model is RWKV-7 "Goose" by Bo Peng (BlinkDL) and the RWKV community. Reference
implementation: [BlinkDL/RWKV-LM](https://github.com/BlinkDL/RWKV-LM).