---
library_name: rwkv
pipeline_tag: text-generation
inference: false
license: apache-2.0
base_model: BlinkDL/rwkv7-g1
datasets:
- mikuhhn1239/novel-agent-sft-dataset
language:
- zh
tags:
- rwkv
- rwkv-7
- g1h
- pytorch
- causal-lm
- chinese-novels
- narrative-analysis
---

# RWKV Novel Agent G1h 13B (10,240 context)

This is an experimental Chinese novel-agent supervised fine-tune of
RWKV-7 G1h 13.3B. It targets novel continuation/instruction following and
structured narrative tasks such as speaker attribution, narrative-type
classification, and scene-boundary detection.

This release is the full-pass checkpoint after one nominal packed-data pass
over the local packed training stream. Use it for research and evaluation
rather than production.

## Model details

| Item | Value |
|---|---|
| Architecture | RWKV-7 `x070` (G1h) |
| Parameters | 13,270,298,624 |
| Layers / width | 61 / 4,096 |
| Model / FFN width | 4,096 / 16,384 |
| Head size | 64 |
| Vocabulary | 65,536 (`rwkv_vocab_v20230424`) |
| Training segment length | 10,240 tokens |
| Checkpoint format | Raw BF16 PyTorch state dict (`.pth`) |
| Checkpoint size | 26,540,868,232 bytes (24.72 GiB) |
| SHA-256 | `e6f9ae8379acb451063a84eee0f2e1d3a735cce6b4ca446a7971e198adc4c793` |

The starting checkpoint was
`BlinkDL/rwkv7-g1/rwkv7-g1h-13.3b-20260710-ctx10240.pth`. This repository does
not currently include a Transformers config or Safetensors conversion.

## Prompt format

Use the fine-tuning prompt format below. Keep the single spaces after role
labels.

```text
System: 你是一个中文小说归因辅助助手。只能从候选角色中选择，并只输出 JSON。

User: 候选角色：甲、乙
上下文：……

Assistant:
```

Generation should stop on token `0` (EOD) or the text sequence
`\n\nUser:`. Consume the stop sequence instead of displaying it.

## Inference

The official `rwkv` runtime can load this raw checkpoint directly. One GPU with
about 30 GiB of available VRAM is recommended for BF16 inference.

```bash
python -m pip install "torch>=2.5" "rwkv==0.8.32" "tokenizers>=0.13.2" ninja
```

```python
import os

os.environ["RWKV_V7_ON"] = "1"
os.environ["RWKV_JIT_ON"] = "1"
os.environ["RWKV_CUDA_ON"] = "1"

from rwkv.model import RWKV
from rwkv.utils import PIPELINE, PIPELINE_ARGS

# The RWKV-v7 loader appends ".pth", so omit the suffix here.
model = RWKV(model="/path/to/rwkv-final", strategy="cuda bf16")
pipeline = PIPELINE(model, "rwkv_vocab_v20230424")

prompt = """System: 场景边界检测。仅输出 {"boundaries":[N]}

User: [P1] 雨停了。\n[P2] 三天后，他回到故乡。\n[P3] 他推开老宅的门。

Assistant:"""

args = PIPELINE_ARGS(
    temperature=0,
    top_p=0,
    alpha_frequency=0,
    alpha_presence=0,
    token_stop=[0],
    chunk_len=256,
)
text = pipeline.generate(prompt, token_count=128, args=args)
text = text.split("\n\nUser:", 1)[0]
print(text)
```

CUDA inference JIT-compiles the RWKV-7 extension on first use, so a working
CUDA toolkit, C++ compiler, and `ninja` executable are required. The model path
passed to `RWKV` must omit the `.pth` suffix.

For deterministic JSON-style evaluation, use greedy decoding:
`temperature=0`, `top_p=0`, `alpha_frequency=0`, `alpha_presence=0`,
`chunk_len=256`, stop token `[0]`, and stop text `\n\nUser:`.

## Training

The published checkpoint was produced in two stages:

| Stage | Optimizer steps | Nominal token presentations | Notes |
|---|---:|---:|---|
| Pilot | 977 | 120,053,760 | Initial SFT run |
| Full-pass continuation | 2,385 | 293,068,800 | Continued from pilot weights with fresh optimizer state |
| Combined | 3,362 | 413,122,560 | One nominal packed-data pass |

The full-pass continuation used 6 GPUs, BF16, DeepSpeed ZeRO-2, global batch
12 sequences (122,880 tokens/update), assistant-only loss masking, constant
learning rate `5e-7` after 10 warmup steps, weight decay `0.01`, and gradient
clipping `0.5`.

The packed training stream contains 131,920 conversations and 413,123,655
tokens, of which 329,536,148 were marked as supervised assistant targets.
Because packed sampling used the training script's magic-prime schedule, "one
nominal pass" should not be read as every raw token being seen exactly once.

Training data was derived from
[`mikuhhn1239/novel-agent-sft-dataset`](https://huggingface.co/datasets/mikuhhn1239/novel-agent-sft-dataset)
at revision `5d3040d21f51b3ce90b9396b058e552c47f43cd5` and includes continuation,
instruction, attribution, narrative-analysis, and scene-boundary examples.

## Evaluation

The checkpoint was evaluated with `rwkv==0.8.32`, PyTorch `2.7.1+cu128`, BF16
CUDA inference, and greedy decoding on NVIDIA A800 80 GB GPUs. Prompts were
rendered with the same local packing code used for training, ending at
`Assistant:`. Invalid JSON counted as wrong or empty predictions.

Local held-out test split:

| Item | Value |
|---|---|
| File | `datasets/staging/novel-agent-sft-v1/test.jsonl` |
| SHA-256 | `6855803b3555839275e5cfabf4ddcd76172e41b0dfaacfe300870b0b293124e6` |
| Cases | 264 total: 30 attribution, 39 narrative, 195 scene |
| Versions | attribution v3.2, narrative v3.2, scene v4/v4.1 |
| Decode caps | attribution 64 tokens, scene 64 tokens, narrative 1280 tokens |

### Local benchmark comparison

| Metric | Dataset README reference | Pilot | Full-pass | Delta |
|---|---:|---:|---:|---:|
| Attribution v3.2 best-candidate accuracy | 86.7% | 83.3% | 86.7% | +3.3 pp |
| Narrative v3.2 micro label accuracy | 69.5% | 66.5% | 65.7% | -0.9 pp |
| Scene v4/v4.1 micro F1 | 30.5%* | 21.4% | 20.7% | -0.7 pp |

Additional full-pass metrics:

| Task | JSON valid | Schema valid | Secondary metrics |
|---|---:|---:|---|
| Attribution | 100.0% | 100.0% | uncertain accuracy 66.7%, joint accuracy 56.7% |
| Narrative | 100.0% | 100.0% | document exact match 0.0% |
| Scene | 100.0% | 100.0% | precision 16.7%, recall 27.2%, exact-set accuracy 19.0% |

*The dataset README's published scene F1 is for its v4-590 evaluation setup,
not this local merged v4/v4.1 held-out split, so it is a directional reference
rather than a strict apples-to-apples baseline.

The last recorded training loss is not an evaluation metric and is intentionally
omitted.

## Intended use

- Research on Chinese fiction generation and continuation.
- Experiments in speaker attribution, narrative classification, and scene
  boundary detection.
- Further fine-tuning and evaluation of RWKV-7 recurrent inference.

## Limitations and risks

- This is an experimental SFT checkpoint; output quality may be unstable.
- The configured 10,240-token training segment length is not evidence of
  reliable performance across that full length.
- Generated JSON may be malformed or violate the requested schema.
- The model has not been evaluated for factuality, bias, memorization, harmful
  content, or prompt-injection resistance.
- Training data contains third-party Chinese novel text. Per-book rights and
  provenance are not fully established, so users must assess copyright,
  privacy, and downstream-use obligations for their jurisdiction and use case.
- Do not rely on generated text for legal, medical, financial, safety-critical,
  or other high-impact decisions.

## License and attribution

The upstream RWKV-7 G1h checkpoint is distributed under Apache-2.0, and this
repository retains that model-repository license metadata. The dataset's
Apache-2.0 label does not establish rights to every underlying novel. The data
rights caveat above remains applicable to this derivative checkpoint.

RWKV implementation and base-model credit:
[`BlinkDL/RWKV-LM`](https://github.com/BlinkDL/RWKV-LM) and
[`BlinkDL/rwkv7-g1`](https://huggingface.co/BlinkDL/rwkv7-g1).