Spaces:
Running
Running
zhan1206 commited on
Commit ·
ff4d952
1
Parent(s): 17159b7
feat: train SentencePiece tokenizer + update README
Browse files- README.md +75 -88
- data/tokenizer_train.txt +0 -0
- models/tokenizer.py +1 -1
- scripts/train_tokenizer.py +2 -0
- tokenizers/tokenizer.model +0 -0
- tokenizers/tokenizer.vocab +500 -0
README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
|
| 2 |
|
| 3 |
**集百家之长，铸六边形开源大模型**
|
| 4 |
|
|
@@ -6,36 +6,34 @@
|
|
| 6 |
[]()
|
| 7 |
[]()
|
| 8 |
|
| 9 |
-
##
|
| 10 |
|
| 11 |
Fusion 是一套面向**纯本地训练与推理**的开源大语言模型方案。它不只交付模型权重，更交付一整套可在消费级硬件上自行定制、微调、甚至复现预训练的全链路工具。
|
| 12 |
|
| 13 |
**核心理念：用户主权** - 所有训练数据清洗、模型微调、推理部署均在本地完成，无需依赖任何云端服务。
|
| 14 |
|
| 15 |
-
##
|
| 16 |
|
| 17 |
-
###
|
| 18 |
- 长序列切为定长块，块内高秩潜空间 + 块间极低秩潜向量
|
| 19 |
-
-
|
| 20 |
- 在 24 GB 显卡上即可对 14B 模型进行长文档微调与推理
|
| 21 |
|
| 22 |
-
###
|
| 23 |
-
- 通过 `
|
| 24 |
-
-
|
| 25 |
-
-
|
| 26 |
-
- 一个模型同时拥有
|
| 27 |
|
| 28 |
-
###
|
| 29 |
- 中文达到 Qwen 同等深度，英文自然度对标 LLaMA 3.1
|
| 30 |
- 根除"翻译腔"，中英独立文化人格
|
| 31 |
-
- 中英比例 1:1，绝不混合 token
|
| 32 |
|
| 33 |
-
###
|
| 34 |
-
- 使用开源教师模型改写
|
| 35 |
-
- 生成风格统一、论证清晰的教学文本
|
| 36 |
-
- 8B 模型知识面比肩传统 70B
|
| 37 |
|
| 38 |
-
##
|
| 39 |
|
| 40 |
### 硬件要求
|
| 41 |
|
|
@@ -54,46 +52,65 @@ cd fusion-llm
|
|
| 54 |
pip install -r requirements.txt
|
| 55 |
```
|
| 56 |
|
| 57 |
-
### 快速
|
|
|
|
|
|
|
| 58 |
|
| 59 |
```bash
|
| 60 |
-
|
| 61 |
-
|
|
|
|
|
|
|
| 62 |
|
| 63 |
-
#
|
| 64 |
-
|
|
|
|
|
|
|
| 65 |
```
|
| 66 |
|
| 67 |
-
###
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
|
| 69 |
```python
|
| 70 |
-
from
|
| 71 |
|
| 72 |
-
|
| 73 |
-
model =
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
-
|
| 76 |
-
response = model.generate("解释量子纠缠", thinking_depth=0)
|
| 77 |
|
| 78 |
-
|
| 79 |
-
|
| 80 |
```
|
| 81 |
|
| 82 |
-
##
|
| 83 |
|
| 84 |
```
|
| 85 |
fusion-llm/
|
| 86 |
-
├── models/ # 模型架构（SBLA注意力、Thinking Dial）
|
| 87 |
-
├── train/ # 训练脚本（
|
| 88 |
-
├── data_pipeline/ # 数据处理（
|
| 89 |
-
├── inference/ # 推理部署（
|
| 90 |
-
├──
|
| 91 |
-
├──
|
| 92 |
-
├──
|
| 93 |
-
|
|
|
|
| 94 |
```
|
| 95 |
|
| 96 |
-
##
|
| 97 |
|
| 98 |
### 1. SBLA 注意力机制
|
| 99 |
|
|
@@ -105,9 +122,6 @@ class SlidingBlockLatentAttention(nn.Module):
|
|
| 105 |
- 块内：高秩潜空间（保留细节）
|
| 106 |
- 块间：极低秩潜向量（传递上下文）
|
| 107 |
"""
|
| 108 |
-
def forward(self, x, block_size=512, ...):
|
| 109 |
-
# 实现详见 models/sbla_attention.py
|
| 110 |
-
...
|
| 111 |
```
|
| 112 |
|
| 113 |
### 2. Thinking Dial 控制
|
|
@@ -116,69 +130,42 @@ class SlidingBlockLatentAttention(nn.Module):
|
|
| 116 |
# 训练时标注 think_rank
|
| 117 |
{"text": "证明勾股定理", "think_rank": 3}
|
| 118 |
|
| 119 |
-
# 推理时
|
| 120 |
-
model.generate(prompt, thinking_depth=2) # 0-3 级别
|
| 121 |
-
```
|
| 122 |
-
|
| 123 |
-
### 3. 双母语数据清洗
|
| 124 |
-
|
| 125 |
-
```python
|
| 126 |
-
# data_pipeline/bilingual_filter.py
|
| 127 |
-
filter = BilingualTrueFilter(
|
| 128 |
-
lang="zh",
|
| 129 |
-
filters=["remove_machine_translation", "remove_clickbait"]
|
| 130 |
-
)
|
| 131 |
-
clean_data = filter.process(raw_data)
|
| 132 |
```
|
| 133 |
|
| 134 |
-
##
|
| 135 |
-
|
| 136 |
-
| 模型 | 中文能力 | 英文能力 | 推理能力 | 长文本 | 显存占用 |
|
| 137 |
-
|------|---------|---------|---------|--------|---------|
|
| 138 |
-
| Qwen-8B | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | 32K | 高 |
|
| 139 |
-
| LLaMA-8B | ⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | 8K | 中 |
|
| 140 |
-
| **Fusion-8B** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | **32K** (可扩展256K) | **低** |
|
| 141 |
|
| 142 |
-
|
| 143 |
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
- [推理部署](docs/inference.md) - Ollama、vLLM、API
|
| 148 |
-
- [硬件估算](docs/hardware.md) - 各型号显卡配置建议
|
| 149 |
-
|
| 150 |
-
## 🤝 贡献指南
|
| 151 |
-
|
| 152 |
-
我们欢迎任何形式的贡献！
|
| 153 |
-
|
| 154 |
-
1. Fork 本仓库
|
| 155 |
-
2. 创建特性分支 (`git checkout -b feature/AmazingFeature`)
|
| 156 |
-
3. 提交更改 (`git commit -m 'Add some AmazingFeature'`)
|
| 157 |
-
4. 推送到分支 (`git push origin feature/AmazingFeature`)
|
| 158 |
-
5. 开启 Pull Request
|
| 159 |
|
| 160 |
-
|
|
|
|
|
|
|
|
|
|
| 161 |
|
| 162 |
-
##
|
| 163 |
|
| 164 |
本项目采用 **Apache License 2.0** - 详见 [LICENSE](LICENSE)
|
| 165 |
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
|
| 171 |
-
##
|
| 172 |
|
| 173 |
Fusion 项目受到以下开源项目的启发：
|
| 174 |
|
| 175 |
- [DeepSeek](https://github.com/deepseek-ai) - MLA 注意力机制
|
| 176 |
-
- [LLaMA](https://github.com/meta-llama/llama) - 基础Transformer架构
|
| 177 |
- [Qwen](https://github.com/QwenLM/Qwen) - 中文能力标杆
|
| 178 |
- [Transformers](https://github.com/huggingface/transformers) - 训练框架
|
| 179 |
- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - 分布式训练
|
| 180 |
|
| 181 |
-
##
|
| 182 |
|
| 183 |
- 项目作者：zhan1206
|
| 184 |
- GitHub：[@zhan1206](https://github.com/zhan1206)
|
|
|
|
| 1 |
+
# Fusion - 六边形开源大模型
|
| 2 |
|
| 3 |
**集百家之长，铸六边形开源大模型**
|
| 4 |
|
|
|
|
| 6 |
[]()
|
| 7 |
[]()
|
| 8 |
|
| 9 |
+
## 项目简介
|
| 10 |
|
| 11 |
Fusion 是一套面向**纯本地训练与推理**的开源大语言模型方案。它不只交付模型权重，更交付一整套可在消费级硬件上自行定制、微调、甚至复现预训练的全链路工具。
|
| 12 |
|
| 13 |
**核心理念：用户主权** - 所有训练数据清洗、模型微调、推理部署均在本地完成，无需依赖任何云端服务。
|
| 14 |
|
| 15 |
+
## 核心特性
|
| 16 |
|
| 17 |
+
### 滑动分块潜注意力（SBLA）
|
| 18 |
- 长序列切为定长块，块内高秩潜空间 + 块间极低秩潜向量
|
| 19 |
+
- 支持 32K 上下文窗口，KV 缓存仅为传统 GQA 的 1/8（SBLA 架构可扩展至 256K）
|
| 20 |
- 在 24 GB 显卡上即可对 14B 模型进行长文档微调与推理
|
| 21 |
|
| 22 |
+
### 动态推理强度调节器（Thinking Dial）
|
| 23 |
+
- 通过 `think_rank`（0-3）控制推理深度
|
| 24 |
+
- 0：直接作答（闲聊、翻译）
|
| 25 |
+
- 3：长思维链模式（数学、代码调试）
|
| 26 |
+
- 一个模型同时拥有快响应与深推理
|
| 27 |
|
| 28 |
+
### 双母语独立训练
|
| 29 |
- 中文达到 Qwen 同等深度，英文自然度对标 LLaMA 3.1
|
| 30 |
- 根除"翻译腔"，中英独立文化人格
|
|
|
|
| 31 |
|
| 32 |
+
### 教科书级知识蒸馏（T-KD）
|
| 33 |
+
- 使用开源教师模型改写高信誉源
|
| 34 |
+
- 生成风格统一、论证清晰的教学文本
|
|
|
|
| 35 |
|
| 36 |
+
## 快速开始
|
| 37 |
|
| 38 |
### 硬件要求
|
| 39 |
|
|
|
|
| 52 |
pip install -r requirements.txt
|
| 53 |
```
|
| 54 |
|
| 55 |
+
### Mini 模型快速训练
|
| 56 |
+
|
| 57 |
+
Mini 模型是一个小型字符级模型，用于快速验证训练流程，无需 GPU。
|
| 58 |
|
| 59 |
```bash
|
| 60 |
+
python train/train_mini.py
|
| 61 |
+
```
|
| 62 |
+
|
| 63 |
+
训练产物保存在 `output/mini_model/`。
|
| 64 |
|
| 65 |
+
### LoRA 微调（需要 GPU）
|
| 66 |
+
|
| 67 |
+
```bash
|
| 68 |
+
python train/lora_finetune.py --model_size 8B
|
| 69 |
```
|
| 70 |
|
| 71 |
+
### 全参微调（需要 GPU）
|
| 72 |
+
|
| 73 |
+
```bash
|
| 74 |
+
python train/full_finetune.py --model_size 8B
|
| 75 |
+
```
|
| 76 |
+
|
| 77 |
+
### 推理
|
| 78 |
|
| 79 |
```python
|
| 80 |
+
from models.fusion_model import FusionModel, FusionConfig
|
| 81 |
|
| 82 |
+
config = FusionConfig(vocab_size=10000, hidden_size=256, num_layers=2)
|
| 83 |
+
model = FusionModel(config)
|
| 84 |
+
model.eval()
|
| 85 |
+
|
| 86 |
+
# 基础推理
|
| 87 |
+
input_ids = torch.tensor([[1, 2, 3]]) # 替换为实际 token IDs
|
| 88 |
+
with torch.no_grad():
|
| 89 |
+
outputs = model(input_ids)
|
| 90 |
+
```
|
| 91 |
|
| 92 |
+
或使用推理控制板：
|
|
|
|
| 93 |
|
| 94 |
+
```bash
|
| 95 |
+
python inference/dashboard.py
|
| 96 |
```
|
| 97 |
|
| 98 |
+
## 项目结构
|
| 99 |
|
| 100 |
```
|
| 101 |
fusion-llm/
|
| 102 |
+
├── models/ # 模型架构（SBLA注意力、Thinking Dial、FusionModel）
|
| 103 |
+
├── train/ # 训练脚本（LoRA微调、全参微调、DPO对齐、Mini模型）
|
| 104 |
+
├── data_pipeline/ # 数据处理（双语过滤、T-KD蒸馏）
|
| 105 |
+
├── inference/ # 推理部署（Dashboard、DyQuant量化、Ollama）
|
| 106 |
+
├── tokenizers/ # SentencePiece tokenizer 模型文件
|
| 107 |
+
├── configs/ # 配置文件模板（0.5B/1.5B/8B/14B/mini）
|
| 108 |
+
├── scripts/ # 工具脚本（tokenizer训练、数据去重）
|
| 109 |
+
├── tests/ # 单元测试
|
| 110 |
+
└── requirements.txt # Python 依赖
|
| 111 |
```
|
| 112 |
|
| 113 |
+
## 核心技术
|
| 114 |
|
| 115 |
### 1. SBLA 注意力机制
|
| 116 |
|
|
|
|
| 122 |
- 块内：高秩潜空间（保留细节）
|
| 123 |
- 块间：极低秩潜向量（传递上下文）
|
| 124 |
"""
|
|
|
|
|
|
|
|
|
|
| 125 |
```
|
| 126 |
|
| 127 |
### 2. Thinking Dial 控制
|
|
|
|
| 130 |
# 训练时标注 think_rank
|
| 131 |
{"text": "证明勾股定理", "think_rank": 3}
|
| 132 |
|
| 133 |
+
# 推理时通过 ThinkingDialProcessor 注入 thinking depth token
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
```
|
| 135 |
|
| 136 |
+
### 3. SentencePiece Tokenizer
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
|
| 138 |
+
项目使用 SentencePiece 训练专用 tokenizer，支持中英双语和 Fusion 特殊 token：
|
| 139 |
|
| 140 |
+
```bash
|
| 141 |
+
# 用自定义语料训练 tokenizer
|
| 142 |
+
python scripts/train_tokenizer.py --input data/tokenizer_train.txt --vocab_size 100000 --output tokenizers/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
|
| 144 |
+
# 用内置 sample data 快速测试
|
| 145 |
+
python scripts/train_tokenizer.py --create_sample_data --input data/tokenizer_train.txt
|
| 146 |
+
python scripts/train_tokenizer.py --input data/tokenizer_train.txt --vocab_size 500 --output tokenizers/
|
| 147 |
+
```
|
| 148 |
|
| 149 |
+
## 许可证
|
| 150 |
|
| 151 |
本项目采用 **Apache License 2.0** - 详见 [LICENSE](LICENSE)
|
| 152 |
|
| 153 |
+
- 可商用
|
| 154 |
+
- 可修改
|
| 155 |
+
- 可私有部署
|
| 156 |
+
- 无附加条款
|
| 157 |
|
| 158 |
+
## 致谢
|
| 159 |
|
| 160 |
Fusion 项目受到以下开源项目的启发：
|
| 161 |
|
| 162 |
- [DeepSeek](https://github.com/deepseek-ai) - MLA 注意力机制
|
| 163 |
+
- [LLaMA](https://github.com/meta-llama/llama) - 基础 Transformer 架构
|
| 164 |
- [Qwen](https://github.com/QwenLM/Qwen) - 中文能力标杆
|
| 165 |
- [Transformers](https://github.com/huggingface/transformers) - 训练框架
|
| 166 |
- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - 分布式训练
|
| 167 |
|
| 168 |
+
## 联系方式
|
| 169 |
|
| 170 |
- 项目作者：zhan1206
|
| 171 |
- GitHub：[@zhan1206](https://github.com/zhan1206)
|
data/tokenizer_train.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
models/tokenizer.py
CHANGED
|
@@ -9,7 +9,7 @@ Usage:
|
|
| 9 |
tokenizer = get_tokenizer("gpt2") # placeholder
|
| 10 |
tokenizer = get_tokenizer("fusion", vocab_size=100000) # future: SentencePiece
|
| 11 |
|
| 12 |
-
Author:
|
| 13 |
Project: Fusion-LLM
|
| 14 |
License: Apache 2.0
|
| 15 |
"""
|
|
|
|
| 9 |
tokenizer = get_tokenizer("gpt2") # placeholder
|
| 10 |
tokenizer = get_tokenizer("fusion", vocab_size=100000) # future: SentencePiece
|
| 11 |
|
| 12 |
+
Author: zhan1206
|
| 13 |
Project: Fusion-LLM
|
| 14 |
License: Apache 2.0
|
| 15 |
"""
|
scripts/train_tokenizer.py
CHANGED
|
@@ -67,6 +67,8 @@ def train_tokenizer(input_path: str, vocab_size: int, output_dir: str, model_typ
|
|
| 67 |
byte_fallback=True, # Important for multilingual
|
| 68 |
split_by_unicode_script=True,
|
| 69 |
allow_whitespace_only_pieces=True,
|
|
|
|
|
|
|
| 70 |
)
|
| 71 |
|
| 72 |
model_path = os.path.join(output_dir, "tokenizer.model")
|
|
|
|
| 67 |
byte_fallback=True, # Important for multilingual
|
| 68 |
split_by_unicode_script=True,
|
| 69 |
allow_whitespace_only_pieces=True,
|
| 70 |
+
normalization_rule_name='identity', # Avoid nmt_nfkc precompiled charsmap issues
|
| 71 |
+
|
| 72 |
)
|
| 73 |
|
| 74 |
model_path = os.path.join(output_dir, "tokenizer.model")
|
tokenizers/tokenizer.model
ADDED
|
Binary file (7.63 kB). View file
|
|
|
tokenizers/tokenizer.vocab
ADDED
|
@@ -0,0 +1,500 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<unk> 0
|
| 2 |
+
<s> 0
|
| 3 |
+
</s> 0
|
| 4 |
+
<pad> 0
|
| 5 |
+
<|pad|> 0
|
| 6 |
+
<|start|> 0
|
| 7 |
+
<|end|> 0
|
| 8 |
+
<|think_depth_0|> 0
|
| 9 |
+
<|think_depth_1|> 0
|
| 10 |
+
<|think_depth_2|> 0
|
| 11 |
+
<|think_depth_3|> 0
|
| 12 |
+
<0x00> 0
|
| 13 |
+
<0x01> 0
|
| 14 |
+
<0x02> 0
|
| 15 |
+
<0x03> 0
|
| 16 |
+
<0x04> 0
|
| 17 |
+
<0x05> 0
|
| 18 |
+
<0x06> 0
|
| 19 |
+
<0x07> 0
|
| 20 |
+
<0x08> 0
|
| 21 |
+
<0x09> 0
|
| 22 |
+
<0x0A> 0
|
| 23 |
+
<0x0B> 0
|
| 24 |
+
<0x0C> 0
|
| 25 |
+
<0x0D> 0
|
| 26 |
+
<0x0E> 0
|
| 27 |
+
<0x0F> 0
|
| 28 |
+
<0x10> 0
|
| 29 |
+
<0x11> 0
|
| 30 |
+
<0x12> 0
|
| 31 |
+
<0x13> 0
|
| 32 |
+
<0x14> 0
|
| 33 |
+
<0x15> 0
|
| 34 |
+
<0x16> 0
|
| 35 |
+
<0x17> 0
|
| 36 |
+
<0x18> 0
|
| 37 |
+
<0x19> 0
|
| 38 |
+
<0x1A> 0
|
| 39 |
+
<0x1B> 0
|
| 40 |
+
<0x1C> 0
|
| 41 |
+
<0x1D> 0
|
| 42 |
+
<0x1E> 0
|
| 43 |
+
<0x1F> 0
|
| 44 |
+
<0x20> 0
|
| 45 |
+
<0x21> 0
|
| 46 |
+
<0x22> 0
|
| 47 |
+
<0x23> 0
|
| 48 |
+
<0x24> 0
|
| 49 |
+
<0x25> 0
|
| 50 |
+
<0x26> 0
|
| 51 |
+
<0x27> 0
|
| 52 |
+
<0x28> 0
|
| 53 |
+
<0x29> 0
|
| 54 |
+
<0x2A> 0
|
| 55 |
+
<0x2B> 0
|
| 56 |
+
<0x2C> 0
|
| 57 |
+
<0x2D> 0
|
| 58 |
+
<0x2E> 0
|
| 59 |
+
<0x2F> 0
|
| 60 |
+
<0x30> 0
|
| 61 |
+
<0x31> 0
|
| 62 |
+
<0x32> 0
|
| 63 |
+
<0x33> 0
|
| 64 |
+
<0x34> 0
|
| 65 |
+
<0x35> 0
|
| 66 |
+
<0x36> 0
|
| 67 |
+
<0x37> 0
|
| 68 |
+
<0x38> 0
|
| 69 |
+
<0x39> 0
|
| 70 |
+
<0x3A> 0
|
| 71 |
+
<0x3B> 0
|
| 72 |
+
<0x3C> 0
|
| 73 |
+
<0x3D> 0
|
| 74 |
+
<0x3E> 0
|
| 75 |
+
<0x3F> 0
|
| 76 |
+
<0x40> 0
|
| 77 |
+
<0x41> 0
|
| 78 |
+
<0x42> 0
|
| 79 |
+
<0x43> 0
|
| 80 |
+
<0x44> 0
|
| 81 |
+
<0x45> 0
|
| 82 |
+
<0x46> 0
|
| 83 |
+
<0x47> 0
|
| 84 |
+
<0x48> 0
|
| 85 |
+
<0x49> 0
|
| 86 |
+
<0x4A> 0
|
| 87 |
+
<0x4B> 0
|
| 88 |
+
<0x4C> 0
|
| 89 |
+
<0x4D> 0
|
| 90 |
+
<0x4E> 0
|
| 91 |
+
<0x4F> 0
|
| 92 |
+
<0x50> 0
|
| 93 |
+
<0x51> 0
|
| 94 |
+
<0x52> 0
|
| 95 |
+
<0x53> 0
|
| 96 |
+
<0x54> 0
|
| 97 |
+
<0x55> 0
|
| 98 |
+
<0x56> 0
|
| 99 |
+
<0x57> 0
|
| 100 |
+
<0x58> 0
|
| 101 |
+
<0x59> 0
|
| 102 |
+
<0x5A> 0
|
| 103 |
+
<0x5B> 0
|
| 104 |
+
<0x5C> 0
|
| 105 |
+
<0x5D> 0
|
| 106 |
+
<0x5E> 0
|
| 107 |
+
<0x5F> 0
|
| 108 |
+
<0x60> 0
|
| 109 |
+
<0x61> 0
|
| 110 |
+
<0x62> 0
|
| 111 |
+
<0x63> 0
|
| 112 |
+
<0x64> 0
|
| 113 |
+
<0x65> 0
|
| 114 |
+
<0x66> 0
|
| 115 |
+
<0x67> 0
|
| 116 |
+
<0x68> 0
|
| 117 |
+
<0x69> 0
|
| 118 |
+
<0x6A> 0
|
| 119 |
+
<0x6B> 0
|
| 120 |
+
<0x6C> 0
|
| 121 |
+
<0x6D> 0
|
| 122 |
+
<0x6E> 0
|
| 123 |
+
<0x6F> 0
|
| 124 |
+
<0x70> 0
|
| 125 |
+
<0x71> 0
|
| 126 |
+
<0x72> 0
|
| 127 |
+
<0x73> 0
|
| 128 |
+
<0x74> 0
|
| 129 |
+
<0x75> 0
|
| 130 |
+
<0x76> 0
|
| 131 |
+
<0x77> 0
|
| 132 |
+
<0x78> 0
|
| 133 |
+
<0x79> 0
|
| 134 |
+
<0x7A> 0
|
| 135 |
+
<0x7B> 0
|
| 136 |
+
<0x7C> 0
|
| 137 |
+
<0x7D> 0
|
| 138 |
+
<0x7E> 0
|
| 139 |
+
<0x7F> 0
|
| 140 |
+
<0x80> 0
|
| 141 |
+
<0x81> 0
|
| 142 |
+
<0x82> 0
|
| 143 |
+
<0x83> 0
|
| 144 |
+
<0x84> 0
|
| 145 |
+
<0x85> 0
|
| 146 |
+
<0x86> 0
|
| 147 |
+
<0x87> 0
|
| 148 |
+
<0x88> 0
|
| 149 |
+
<0x89> 0
|
| 150 |
+
<0x8A> 0
|
| 151 |
+
<0x8B> 0
|
| 152 |
+
<0x8C> 0
|
| 153 |
+
<0x8D> 0
|
| 154 |
+
<0x8E> 0
|
| 155 |
+
<0x8F> 0
|
| 156 |
+
<0x90> 0
|
| 157 |
+
<0x91> 0
|
| 158 |
+
<0x92> 0
|
| 159 |
+
<0x93> 0
|
| 160 |
+
<0x94> 0
|
| 161 |
+
<0x95> 0
|
| 162 |
+
<0x96> 0
|
| 163 |
+
<0x97> 0
|
| 164 |
+
<0x98> 0
|
| 165 |
+
<0x99> 0
|
| 166 |
+
<0x9A> 0
|
| 167 |
+
<0x9B> 0
|
| 168 |
+
<0x9C> 0
|
| 169 |
+
<0x9D> 0
|
| 170 |
+
<0x9E> 0
|
| 171 |
+
<0x9F> 0
|
| 172 |
+
<0xA0> 0
|
| 173 |
+
<0xA1> 0
|
| 174 |
+
<0xA2> 0
|
| 175 |
+
<0xA3> 0
|
| 176 |
+
<0xA4> 0
|
| 177 |
+
<0xA5> 0
|
| 178 |
+
<0xA6> 0
|
| 179 |
+
<0xA7> 0
|
| 180 |
+
<0xA8> 0
|
| 181 |
+
<0xA9> 0
|
| 182 |
+
<0xAA> 0
|
| 183 |
+
<0xAB> 0
|
| 184 |
+
<0xAC> 0
|
| 185 |
+
<0xAD> 0
|
| 186 |
+
<0xAE> 0
|
| 187 |
+
<0xAF> 0
|
| 188 |
+
<0xB0> 0
|
| 189 |
+
<0xB1> 0
|
| 190 |
+
<0xB2> 0
|
| 191 |
+
<0xB3> 0
|
| 192 |
+
<0xB4> 0
|
| 193 |
+
<0xB5> 0
|
| 194 |
+
<0xB6> 0
|
| 195 |
+
<0xB7> 0
|
| 196 |
+
<0xB8> 0
|
| 197 |
+
<0xB9> 0
|
| 198 |
+
<0xBA> 0
|
| 199 |
+
<0xBB> 0
|
| 200 |
+
<0xBC> 0
|
| 201 |
+
<0xBD> 0
|
| 202 |
+
<0xBE> 0
|
| 203 |
+
<0xBF> 0
|
| 204 |
+
<0xC0> 0
|
| 205 |
+
<0xC1> 0
|
| 206 |
+
<0xC2> 0
|
| 207 |
+
<0xC3> 0
|
| 208 |
+
<0xC4> 0
|
| 209 |
+
<0xC5> 0
|
| 210 |
+
<0xC6> 0
|
| 211 |
+
<0xC7> 0
|
| 212 |
+
<0xC8> 0
|
| 213 |
+
<0xC9> 0
|
| 214 |
+
<0xCA> 0
|
| 215 |
+
<0xCB> 0
|
| 216 |
+
<0xCC> 0
|
| 217 |
+
<0xCD> 0
|
| 218 |
+
<0xCE> 0
|
| 219 |
+
<0xCF> 0
|
| 220 |
+
<0xD0> 0
|
| 221 |
+
<0xD1> 0
|
| 222 |
+
<0xD2> 0
|
| 223 |
+
<0xD3> 0
|
| 224 |
+
<0xD4> 0
|
| 225 |
+
<0xD5> 0
|
| 226 |
+
<0xD6> 0
|
| 227 |
+
<0xD7> 0
|
| 228 |
+
<0xD8> 0
|
| 229 |
+
<0xD9> 0
|
| 230 |
+
<0xDA> 0
|
| 231 |
+
<0xDB> 0
|
| 232 |
+
<0xDC> 0
|
| 233 |
+
<0xDD> 0
|
| 234 |
+
<0xDE> 0
|
| 235 |
+
<0xDF> 0
|
| 236 |
+
<0xE0> 0
|
| 237 |
+
<0xE1> 0
|
| 238 |
+
<0xE2> 0
|
| 239 |
+
<0xE3> 0
|
| 240 |
+
<0xE4> 0
|
| 241 |
+
<0xE5> 0
|
| 242 |
+
<0xE6> 0
|
| 243 |
+
<0xE7> 0
|
| 244 |
+
<0xE8> 0
|
| 245 |
+
<0xE9> 0
|
| 246 |
+
<0xEA> 0
|
| 247 |
+
<0xEB> 0
|
| 248 |
+
<0xEC> 0
|
| 249 |
+
<0xED> 0
|
| 250 |
+
<0xEE> 0
|
| 251 |
+
<0xEF> 0
|
| 252 |
+
<0xF0> 0
|
| 253 |
+
<0xF1> 0
|
| 254 |
+
<0xF2> 0
|
| 255 |
+
<0xF3> 0
|
| 256 |
+
<0xF4> 0
|
| 257 |
+
<0xF5> 0
|
| 258 |
+
<0xF6> 0
|
| 259 |
+
<0xF7> 0
|
| 260 |
+
<0xF8> 0
|
| 261 |
+
<0xF9> 0
|
| 262 |
+
<0xFA> 0
|
| 263 |
+
<0xFB> 0
|
| 264 |
+
<0xFC> 0
|
| 265 |
+
<0xFD> 0
|
| 266 |
+
<0xFE> 0
|
| 267 |
+
<0xFF> 0
|
| 268 |
+
▁ -2.44612
|
| 269 |
+
s -2.94286
|
| 270 |
+
l -3.33855
|
| 271 |
+
a -3.77109
|
| 272 |
+
t -3.82259
|
| 273 |
+
e -4.00136
|
| 274 |
+
o -4.03383
|
| 275 |
+
p -4.30998
|
| 276 |
+
f -4.40349
|
| 277 |
+
▁mode -4.40746
|
| 278 |
+
▁t -4.60183
|
| 279 |
+
▁language -4.62083
|
| 280 |
+
k -4.62094
|
| 281 |
+
- -4.62445
|
| 282 |
+
tion -4.7007
|
| 283 |
+
ge -4.72992
|
| 284 |
+
ce -4.85343
|
| 285 |
+
ra -4.86504
|
| 286 |
+
re -4.8947
|
| 287 |
+
计算机 -4.89926
|
| 288 |
+
的 -4.90304
|
| 289 |
+
▁learning -4.91083
|
| 290 |
+
▁computer -4.91117
|
| 291 |
+
▁A -4.92103
|
| 292 |
+
he -4.92372
|
| 293 |
+
ta -4.92573
|
| 294 |
+
u -4.99341
|
| 295 |
+
ch -5.00304
|
| 296 |
+
ine -5.15633
|
| 297 |
+
en -5.18019
|
| 298 |
+
te -5.18026
|
| 299 |
+
m -5.18134
|
| 300 |
+
ed -5.18892
|
| 301 |
+
i -5.19778
|
| 302 |
+
n -5.23153
|
| 303 |
+
c -5.23766
|
| 304 |
+
学习 -5.29817
|
| 305 |
+
语言 -5.30085
|
| 306 |
+
是 -5.30174
|
| 307 |
+
P -5.3058
|
| 308 |
+
▁N -5.30606
|
| 309 |
+
大 -5.30708
|
| 310 |
+
机 -5.30758
|
| 311 |
+
▁a -5.30847
|
| 312 |
+
人工智能 -5.30859
|
| 313 |
+
模型 -5.30988
|
| 314 |
+
rom -5.31076
|
| 315 |
+
中 -5.31088
|
| 316 |
+
任务 -5.31127
|
| 317 |
+
现 -5.31167
|
| 318 |
+
语言模型 -5.31206
|
| 319 |
+
▁learn -5.31216
|
| 320 |
+
L -5.31437
|
| 321 |
+
技术 -5.31487
|
| 322 |
+
使 -5.31626
|
| 323 |
+
▁Transformer -5.31634
|
| 324 |
+
模 -5.31696
|
| 325 |
+
b -5.31828
|
| 326 |
+
自然语言处理 -5.31847
|
| 327 |
+
深度学习 -5.32159
|
| 328 |
+
ow -5.32251
|
| 329 |
+
了 -5.32341
|
| 330 |
+
人 -5.32361
|
| 331 |
+
▁pro -5.32703
|
| 332 |
+
ex -5.32706
|
| 333 |
+
▁ne -5.34816
|
| 334 |
+
▁cor -5.35055
|
| 335 |
+
ific -5.35407
|
| 336 |
+
▁de -5.35822
|
| 337 |
+
nd -5.39694
|
| 338 |
+
mo -5.4074
|
| 339 |
+
r -5.41932
|
| 340 |
+
ur -5.43363
|
| 341 |
+
▁u -5.48994
|
| 342 |
+
is -5.49832
|
| 343 |
+
ar -5.53788
|
| 344 |
+
da -5.55829
|
| 345 |
+
▁ar -5.59715
|
| 346 |
+
▁m -5.65626
|
| 347 |
+
tu -5.66034
|
| 348 |
+
▁d -5.72945
|
| 349 |
+
tur -5.754
|
| 350 |
+
er -5.76365
|
| 351 |
+
chi -5.78109
|
| 352 |
+
es -5.78847
|
| 353 |
+
gen -5.83188
|
| 354 |
+
in -5.84722
|
| 355 |
+
ence -5.84828
|
| 356 |
+
at -5.87637
|
| 357 |
+
iz -5.92451
|
| 358 |
+
on -5.95904
|
| 359 |
+
ec -5.96185
|
| 360 |
+
ent -5.97889
|
| 361 |
+
w -5.97954
|
| 362 |
+
ing -5.98389
|
| 363 |
+
ti -5.98673
|
| 364 |
+
学 -5.98833
|
| 365 |
+
一 -5.98834
|
| 366 |
+
个 -5.98834
|
| 367 |
+
分 -5.98834
|
| 368 |
+
支 -5.98834
|
| 369 |
+
科 -5.98834
|
| 370 |
+
重 -5.98834
|
| 371 |
+
要 -5.98835
|
| 372 |
+
语 -5.99244
|
| 373 |
+
料 -5.99246
|
| 374 |
+
练 -5.99246
|
| 375 |
+
规 -5.99246
|
| 376 |
+
过 -5.99246
|
| 377 |
+
通 -5.99246
|
| 378 |
+
训 -5.99247
|
| 379 |
+
预 -5.99247
|
| 380 |
+
知 -5.99247
|
| 381 |
+
识 -5.99247
|
| 382 |
+
ci -5.99501
|
| 383 |
+
v -5.99642
|
| 384 |
+
D -5.99741
|
| 385 |
+
y -5.99741
|
| 386 |
+
▁T -5.99776
|
| 387 |
+
从 -5.99998
|
| 388 |
+
器 -5.99998
|
| 389 |
+
够 -5.99998
|
| 390 |
+
据 -5.99998
|
| 391 |
+
数 -5.99998
|
| 392 |
+
能 -5.99998
|
| 393 |
+
下 -6.00078
|
| 394 |
+
定 -6.00078
|
| 395 |
+
应 -6.00078
|
| 396 |
+
游 -6.00078
|
| 397 |
+
特 -6.00078
|
| 398 |
+
适 -6.00078
|
| 399 |
+
微 -6.00079
|
| 400 |
+
调 -6.00079
|
| 401 |
+
代 -6.00158
|
| 402 |
+
件 -6.00158
|
| 403 |
+
制 -6.00158
|
| 404 |
+
力 -6.00158
|
| 405 |
+
心 -6.00158
|
| 406 |
+
意 -6.00158
|
| 407 |
+
核 -6.00158
|
| 408 |
+
注 -6.00158
|
| 409 |
+
组 -6.00158
|
| 410 |
+
助 -6.00556
|
| 411 |
+
帮 -6.00556
|
| 412 |
+
理 -6.00556
|
| 413 |
+
类 -6.00557
|
| 414 |
+
解 -6.00557
|
| 415 |
+
or -6.00794
|
| 416 |
+
在 -6.00816
|
| 417 |
+
文 -6.00816
|
| 418 |
+
出 -6.00816
|
| 419 |
+
成 -6.00816
|
| 420 |
+
本 -6.00816
|
| 421 |
+
生 -6.00816
|
| 422 |
+
色 -6.00816
|
| 423 |
+
表 -6.00816
|
| 424 |
+
M -6.00977
|
| 425 |
+
学习使 -6.00979
|
| 426 |
+
▁comp -6.01112
|
| 427 |
+
F -6.01157
|
| 428 |
+
动 -6.0154
|
| 429 |
+
及 -6.0154
|
| 430 |
+
开 -6.0154
|
| 431 |
+
推 -6.0154
|
| 432 |
+
普 -6.0154
|
| 433 |
+
源 -6.0154
|
| 434 |
+
底 -6.01781
|
| 435 |
+
�� -6.01781
|
| 436 |
+
改 -6.01781
|
| 437 |
+
变 -6.01782
|
| 438 |
+
域 -6.01782
|
| 439 |
+
构 -6.01782
|
| 440 |
+
架 -6.01782
|
| 441 |
+
领 -6.01782
|
| 442 |
+
ning -6.02384
|
| 443 |
+
多 -6.02818
|
| 444 |
+
层 -6.02818
|
| 445 |
+
拟 -6.02818
|
| 446 |
+
来 -6.02818
|
| 447 |
+
用 -6.02818
|
| 448 |
+
神 -6.02818
|
| 449 |
+
经 -6.02818
|
| 450 |
+
络 -6.02818
|
| 451 |
+
网 -6.02818
|
| 452 |
+
脑 -6.02818
|
| 453 |
+
rn -6.02895
|
| 454 |
+
I -6.03309
|
| 455 |
+
O -6.0331
|
| 456 |
+
rati -6.03329
|
| 457 |
+
▁s -6.0389
|
| 458 |
+
ran -6.04547
|
| 459 |
+
an -6.05951
|
| 460 |
+
d -6.06651
|
| 461 |
+
z -6.1136
|
| 462 |
+
po -6.12669
|
| 463 |
+
la -6.16874
|
| 464 |
+
tr -6.19337
|
| 465 |
+
ns -6.23596
|
| 466 |
+
ers -6.24996
|
| 467 |
+
pe -6.44462
|
| 468 |
+
le -6.48899
|
| 469 |
+
ne -6.54229
|
| 470 |
+
el -6.61066
|
| 471 |
+
ea -6.67065
|
| 472 |
+
me -6.87077
|
| 473 |
+
▁la -6.91637
|
| 474 |
+
▁i -7.05299
|
| 475 |
+
nce -7.16604
|
| 476 |
+
g -7.59023
|
| 477 |
+
nt -7.68809
|
| 478 |
+
ion -7.75095
|
| 479 |
+
▁l -7.82733
|
| 480 |
+
h -8.89587
|
| 481 |
+
A -11.5543
|
| 482 |
+
x -11.9128
|
| 483 |
+
N -11.9523
|
| 484 |
+
T -12.6852
|
| 485 |
+
深 -16.3599
|
| 486 |
+
度 -16.36
|
| 487 |
+
自 -16.3601
|
| 488 |
+
然 -16.3602
|
| 489 |
+
处 -16.3603
|
| 490 |
+
术 -16.3604
|
| 491 |
+
技 -16.3605
|
| 492 |
+
务 -16.3606
|
| 493 |
+
任 -16.3607
|
| 494 |
+
智 -16.3608
|
| 495 |
+
工 -16.3609
|
| 496 |
+
计 -16.361
|
| 497 |
+
算 -16.3611
|
| 498 |
+
型 -16.3612
|
| 499 |
+
习 -16.3613
|
| 500 |
+
言 -16.3614
|