Upload 9 files

base patch32 256 fp32 onnx

siglip_base_text.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9b1674071b771e513cbc3ed4736bf838cd2477bdecb96101b2a83c6c103b560
+size 1129500913

siglip_base_vision.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f64443505c3d385b27d9e8480779a334143a4f4d243a7fbe9ed66dc0c58b3067
+size 378501505

tokenizer_vocab/README.md

@@ -0,0 +1,82 @@
+# SigLIP2 Tokenizer 词汇表
+
+## 文件说明
+
+- **vocab_token_to_id.json**: 完整词汇表 (token → ID)
+  - 格式: {"token": id}
+  - 条目数: 256,000
+
+- **vocab_id_to_token.json**: 反向词汇表 (ID → token)
+  - 格式: {"id": "token"}
+  - 条目数: 256,000
+
+- **special_tokens.json**: 特殊tokens
+  - PAD: <pad> (ID: 0)
+  - EOS: <eos> (ID: 1)
+  - BOS: <bos> (ID: 2)
+  - UNK: <unk> (ID: 3)
+
+- **common_words.json**: 常用词的token ID映射
+  - 包含常见的名词、动词、形容词等
+
+## 使用方法
+
+### Python中加载词汇表
+
+```python
+import json
+
+# 加载token -> id映射
+with open('vocab_token_to_id.json', 'r', encoding='utf-8') as f:
+    vocab = json.load(f)
+
+# 查询特定token的ID
+token_id = vocab.get('cat')
+print(f"'cat' 的ID: {token_id}")
+
+# 加载id -> token映射
+with open('vocab_id_to_token.json', 'r', encoding='utf-8') as f:
+    id_to_token = json.load(f)
+
+# 查询特定ID的token
+token = id_to_token.get('4401')
+print(f"ID 4401 的token: {token}")
+```
+
+### 查询词的token ID
+
+```python
+from transformers import AutoProcessor
+
+processor = AutoProcessor.from_pretrained('google/siglip2-base-patch32-256')
+tokenizer = processor.tokenizer
+
+# 方法1: 直接查询词汇表
+vocab = tokenizer.get_vocab()
+token_id = vocab.get('cat')
+
+# 方法2: 使用encode
+token_ids = tokenizer.encode('cat', add_special_tokens=False)
+print(f"'cat' 的token IDs: {token_ids}")
+```
+
+## 词汇表统计
+
+- 总词汇量: 256,000
+- 特殊tokens: 4 (PAD, EOS, BOS, UNK)
+- Tokenizer类型: GemmaTokenizerFast
+- 模型: google/siglip2-base-patch32-256
+
+## 注意事项
+
+1. **空格处理**: 很多token前面有 `▁` 符号，表示该token前面有空格
+   - 例如: `▁cat` 表示 " cat" (前面有空格)
+   - 而 `cat` 表示 "cat" (前面没有空格)
+
+2. **子词分割**: 长词可能被分割成多个token
+   - 例如: "bicycle" 可能被分割成 ["▁bi", "cycle"]
+
+3. **大小写敏感**: tokenizer区分大小写
+   - "Cat" 和 "cat" 是不同的token
+
+4. **数字**: 数字通常有独立的token ID

tokenizer_vocab/STATUS.md

@@ -0,0 +1,165 @@
+# Tokenizer词汇表状态报告
+
+## ✅ 生成状态
+
+**状态**: 已成功生成并验证  
+**生成时间**: 最新  
+**验证状态**: ✅ 所有测试通过
+
+## 📊 词汇表统计
+
+| 项目 | 数值 |
+|------|------|
+| 总词汇量 | 256,000 tokens |
+| 特殊tokens | 4 个 (PAD, EOS, BOS, UNK) |
+| 常用词 | 47 个 |
+| Tokenizer类型 | GemmaTokenizerFast |
+| 模型 | google/siglip2-base-patch32-256 |
+
+## 📁 文件清单
+
+| 文件 | 大小 | 状态 | 说明 |
+|------|------|------|------|
+| vocab_token_to_id.json | 5.4 MB | ✅ | Token → ID 完整映射 |
+| vocab_id_to_token.json | 5.9 MB | ✅ | ID → Token 反向映射 |
+| special_tokens.json | 0.2 KB | ✅ | 特殊tokens定义 |
+| common_words.json | 2.7 KB | ✅ | 47个常用词 |
+| README.md | 2.1 KB | ✅ | 基本使用说明 |
+| USAGE.md | - | ✅ | 详细使用指南 |
+| STATUS.md | - | ✅ | 本文件 |
+
+## ✅ 验证结果
+
+### 文件完整性
+- ✅ 所有必需文件存在
+- ✅ 文件大小正常
+- ✅ JSON格式正确
+
+### 数据正确性
+- ✅ 词汇表大小: 256,000 tokens
+- ✅ 关键tokens验证通过:
+  - `'cat'` → 4991
+  - `'▁cat'` → 4401
+  - `'dog'` → 12240
+  - `'▁dog'` → 5929
+  - `'a'` → 235250
+
+### 双向映射一致性
+- ✅ token_to_id 和 id_to_token 大小一致
+- ✅ 100个样本抽查全部一致
+- ✅ 反向映射验证通过
+
+### 特殊Tokens
+- ✅ PAD token: `<pad>` (ID: 0)
+- ✅ EOS token: `<eos>` (ID: 1)
+- ✅ BOS token: `<bos>` (ID: 2)
+- ✅ UNK token: `<unk>` (ID: 3)
+
+### 常用词
+- ✅ 47个常用词全部验证通过
+- ✅ ID映射正确
+
+## 🚀 使用方法
+
+### 快速加载
+
+```python
+import json
+
+# 加载词汇表
+with open('models/tokenizer_vocab/vocab_token_to_id.json', 'r') as f:
+    vocab = json.load(f)
+
+# 查询
+print(vocab['cat'])  # 4991
+```
+
+### 运行示例
+
+```bash
+# 加载和使用示例
+python examples/load_vocab_from_models.py
+
+# 验证词汇表
+python tools/verify_vocab.py
+```
+
+### 搜索功能
+
+```bash
+# 搜索特定词
+python tools/extract_tokenizer_vocab.py --search "cat"
+python tools/extract_tokenizer_vocab.py --search "bicycle"
+```
+
+## 🔄 重新生成
+
+如需重新生成词汇表：
+
+```bash
+# 重新生成到models文件夹
+python tools/extract_tokenizer_vocab.py --output models/tokenizer_vocab
+
+# 验证生成结果
+python tools/verify_vocab.py
+```
+
+## 📚 相关文档
+
+- **基本说明**: `README.md`
+- **详细指南**: `USAGE.md`
+- **完整文档**: `../../docs/how_to_get_tokenizer_vocab.md`
+- **快速参考**: `../../docs/tokenizer_quick_reference.md`
+- **技术细节**: `../../docs/tokenization_technical_details.md`
+
+## 🛠️ 相关工具
+
+- **提取工具**: `tools/extract_tokenizer_vocab.py`
+- **验证工具**: `tools/verify_vocab.py`
+- **加载示例**: `examples/load_vocab_from_models.py`
+- **使用示例**: `examples/use_tokenizer_vocab.py`
+- **演示脚本**: `examples/tokenization_demo.py`
+
+## 📝 更新日志
+
+### 最新版本
+- ✅ 成功生成256,000个tokens的完整词汇表
+- ✅ 双向映射验证通过
+- ✅ 所有测试通过
+- ✅ 文档完整
+
+## ⚠️ 注意事项
+
+1. **空格前缀**: 词前的 `▁` 表示有空格
+   - `'cat'` (4991) - 无空格前缀
+   - `'▁cat'` (4401) - 有空格前缀
+
+2. **大小写**: tokenizer区分大小写
+   - `'cat'` ≠ `'Cat'` ≠ `'CAT'`
+
+3. **文件大小**: 完整词汇表文件较大 (5-6 MB)
+   - 如需轻量级版本，使用 `common_words.json`
+
+4. **编码**: 所有JSON文件使用UTF-8编码
+
+## 🎯 质量保证
+
+- ✅ 自动化验证脚本
+- ✅ 完整性检查
+- ✅ 一致性验证
+- ✅ 示例代码测试
+- ✅ 文档完整
+
+## 📞 支持
+
+如遇问题：
+1. 运行验证脚本: `python tools/verify_vocab.py`
+2. 查看文档: `USAGE.md` 或 `README.md`
+3. 重新生成: `python tools/extract_tokenizer_vocab.py --output models/tokenizer_vocab`
+
+---
+
+**最后验证时间**: 刚刚  
+**验证状态**: ✅ 全部通过  
+**词汇表版本**: SigLIP2 (google/siglip2-base-patch32-256)
+

tokenizer_vocab/coco_vocab.json

@@ -0,0 +1,1321 @@
+{
+  "person": {
+    "forms": {
+      "person": 6717,
+      "▁person": 1552,
+      "Person": 7175,
+      "▁Person": 6839
+    },
+    "encode": {
+      "token_ids": [
+        6717
+      ],
+      "tokens": [
+        "person"
+      ]
+    },
+    "id": 6717,
+    "token": "person"
+  },
+  "bicycle": {
+    "forms": {
+      "bicycle": 134234,
+      "▁bicycle": 34334,
+      "Bicycle": 178030,
+      "▁Bicycle": 91466
+    },
+    "encode": {
+      "token_ids": [
+        134234
+      ],
+      "tokens": [
+        "bicycle"
+      ]
+    },
+    "id": 134234,
+    "token": "bicycle"
+  },
+  "car": {
+    "forms": {
+      "car": 2269,
+      "▁car": 1226,
+      "Car": 3726,
+      "▁Car": 2244
+    },
+    "encode": {
+      "token_ids": [
+        2269
+      ],
+      "tokens": [
+        "car"
+      ]
+    },
+    "id": 2269,
+    "token": "car"
+  },
+  "motorcycle": {
+    "forms": {
+      "motorcycle": 165924,
+      "▁motorcycle": 33743,
+      "Motorcycle": 206757,
+      "▁Motorcycle": 82796
+    },
+    "encode": {
+      "token_ids": [
+        165924
+      ],
+      "tokens": [
+        "motorcycle"
+      ]
+    },
+    "id": 165924,
+    "token": "motorcycle"
+  },
+  "airplane": {
+    "forms": {
+      "airplane": 169566,
+      "▁airplane": 44737,
+      "Airplane": 229137,
+      "▁Airplane": 154228
+    },
+    "encode": {
+      "token_ids": [
+        169566
+      ],
+      "tokens": [
+        "airplane"
+      ]
+    },
+    "id": 169566,
+    "token": "airplane"
+  },
+  "bus": {
+    "forms": {
+      "bus": 6710,
+      "▁bus": 1986,
+      "Bus": 14617,
+      "▁Bus": 13632
+    },
+    "encode": {
+      "token_ids": [
+        6710
+      ],
+      "tokens": [
+        "bus"
+      ]
+    },
+    "id": 6710,
+    "token": "bus"
+  },
+  "train": {
+    "forms": {
+      "train": 8297,
+      "▁train": 6588,
+      "Train": 32348,
+      "▁Train": 23098
+    },
+    "encode": {
+      "token_ids": [
+        8297
+      ],
+      "tokens": [
+        "train"
+      ]
+    },
+    "id": 8297,
+    "token": "train"
+  },
+  "truck": {
+    "forms": {
+      "truck": 56791,
+      "▁truck": 9564,
+      "Truck": 81410,
+      "▁Truck": 39473
+    },
+    "encode": {
+      "token_ids": [
+        56791
+      ],
+      "tokens": [
+        "truck"
+      ]
+    },
+    "id": 56791,
+    "token": "truck"
+  },
+  "boat": {
+    "forms": {
+      "boat": 29045,
+      "▁boat": 11390,
+      "Boat": 78863,
+      "▁Boat": 40927
+    },
+    "encode": {
+      "token_ids": [
+        29045
+      ],
+      "tokens": [
+        "boat"
+      ]
+    },
+    "id": 29045,
+    "token": "boat"
+  },
+  "traffic light": {
+    "encode": {
+      "token_ids": [
+        67557,
+        2611
+      ],
+      "tokens": [
+        "traffic",
+        "▁light"
+      ]
+    }
+  },
+  "fire hydrant": {
+    "encode": {
+      "token_ids": [
+        7838,
+        6432,
+        5977
+      ],
+      "tokens": [
+        "fire",
+        "▁hyd",
+        "rant"
+      ]
+    }
+  },
+  "stop sign": {
+    "encode": {
+      "token_ids": [
+        7282,
+        2035
+      ],
+      "tokens": [
+        "stop",
+        "▁sign"
+      ]
+    }
+  },
+  "parking meter": {
+    "encode": {
+      "token_ids": [
+        74811,
+        16198
+      ],
+      "tokens": [
+        "parking",
+        "▁meter"
+      ]
+    }
+  },
+  "bench": {
+    "forms": {
+      "bench": 31683,
+      "▁bench": 15475,
+      "Bench": 51453,
+      "▁Bench": 41285
+    },
+    "encode": {
+      "token_ids": [
+        31683
+      ],
+      "tokens": [
+        "bench"
+      ]
+    },
+    "id": 31683,
+    "token": "bench"
+  },
+  "bird": {
+    "forms": {
+      "bird": 18424,
+      "▁bird": 13170,
+      "Bird": 45124,
+      "▁Bird": 23912
+    },
+    "encode": {
+      "token_ids": [
+        18424
+      ],
+      "tokens": [
+        "bird"
+      ]
+    },
+    "id": 18424,
+    "token": "bird"
+  },
+  "cat": {
+    "forms": {
+      "cat": 4991,
+      "▁cat": 4401,
+      "Cat": 9843,
+      "▁Cat": 6828
+    },
+    "encode": {
+      "token_ids": [
+        4991
+      ],
+      "tokens": [
+        "cat"
+      ]
+    },
+    "id": 4991,
+    "token": "cat"
+  },
+  "dog": {
+    "forms": {
+      "dog": 12240,
+      "▁dog": 5929,
+      "Dog": 21401,
+      "▁Dog": 12935
+    },
+    "encode": {
+      "token_ids": [
+        12240
+      ],
+      "tokens": [
+        "dog"
+      ]
+    },
+    "id": 12240,
+    "token": "dog"
+  },
+  "horse": {
+    "forms": {
+      "horse": 38689,
+      "▁horse": 10468,
+      "Horse": 62788,
+      "▁Horse": 29572
+    },
+    "encode": {
+      "token_ids": [
+        38689
+      ],
+      "tokens": [
+        "horse"
+      ]
+    },
+    "id": 38689,
+    "token": "horse"
+  },
+  "sheep": {
+    "forms": {
+      "sheep": 111417,
+      "▁sheep": 21304,
+      "Sheep": 103092,
+      "▁Sheep": 64756
+    },
+    "encode": {
+      "token_ids": [
+        111417
+      ],
+      "tokens": [
+        "sheep"
+      ]
+    },
+    "id": 111417,
+    "token": "sheep"
+  },
+  "cow": {
+    "forms": {
+      "cow": 14706,
+      "▁cow": 13910,
+      "Cow": 44111,
+      "▁Cow": 20845
+    },
+    "encode": {
+      "token_ids": [
+        14706
+      ],
+      "tokens": [
+        "cow"
+      ]
+    },
+    "id": 14706,
+    "token": "cow"
+  },
+  "elephant": {
+    "forms": {
+      "elephant": 135290,
+      "▁elephant": 38816,
+      "Elephant": 139690,
+      "▁Elephant": 74431
+    },
+    "encode": {
+      "token_ids": [
+        135290
+      ],
+      "tokens": [
+        "elephant"
+      ]
+    },
+    "id": 135290,
+    "token": "elephant"
+  },
+  "bear": {
+    "forms": {
+      "bear": 21033,
+      "▁bear": 8718,
+      "Bear": 24561,
+      "▁Bear": 20323
+    },
+    "encode": {
+      "token_ids": [
+        21033
+      ],
+      "tokens": [
+        "bear"
+      ]
+    },
+    "id": 21033,
+    "token": "bear"
+  },
+  "zebra": {
+    "forms": {
+      "zebra": 139048,
+      "▁zebra": 64801,
+      "Zebra": 161784,
+      "▁Zebra": 131247
+    },
+    "encode": {
+      "token_ids": [
+        139048
+      ],
+      "tokens": [
+        "zebra"
+      ]
+    },
+    "id": 139048,
+    "token": "zebra"
+  },
+  "giraffe": {
+    "forms": {
+      "▁giraffe": 117771,
+      "Giraffe": 233208
+    },
+    "encode": {
+      "token_ids": [
+        126258,
+        10303
+      ],
+      "tokens": [
+        "gira",
+        "ffe"
+      ]
+    }
+  },
+  "backpack": {
+    "forms": {
+      "backpack": 203210,
+      "▁backpack": 26408,
+      "▁Backpack": 100536
+    },
+    "encode": {
+      "token_ids": [
+        203210
+      ],
+      "tokens": [
+        "backpack"
+      ]
+    },
+    "id": 203210,
+    "token": "backpack"
+  },
+  "umbrella": {
+    "forms": {
+      "umbrella": 201328,
+      "▁umbrella": 43562,
+      "▁Umbrella": 135457
+    },
+    "encode": {
+      "token_ids": [
+        201328
+      ],
+      "tokens": [
+        "umbrella"
+      ]
+    },
+    "id": 201328,
+    "token": "umbrella"
+  },
+  "handbag": {
+    "forms": {
+      "▁handbag": 86119
+    },
+    "encode": {
+      "token_ids": [
+        3345,
+        6214
+      ],
+      "tokens": [
+        "hand",
+        "bag"
+      ]
+    }
+  },
+  "tie": {
+    "forms": {
+      "tie": 6290,
+      "▁tie": 10985,
+      "Tie": 64403,
+      "▁Tie": 34135
+    },
+    "encode": {
+      "token_ids": [
+        6290
+      ],
+      "tokens": [
+        "tie"
+      ]
+    },
+    "id": 6290,
+    "token": "tie"
+  },
+  "suitcase": {
+    "forms": {
+      "▁suitcase": 79957
+    },
+    "encode": {
+      "token_ids": [
+        18555,
+        1533
+      ],
+      "tokens": [
+        "suit",
+        "case"
+      ]
+    }
+  },
+  "frisbee": {
+    "encode": {
+      "token_ids": [
+        70219,
+        19476
+      ],
+      "tokens": [
+        "fris",
+        "bee"
+      ]
+    }
+  },
+  "skis": {
+    "forms": {
+      "skis": 211141,
+      "▁skis": 107791
+    },
+    "encode": {
+      "token_ids": [
+        211141
+      ],
+      "tokens": [
+        "skis"
+      ]
+    },
+    "id": 211141,
+    "token": "skis"
+  },
+  "snowboard": {
+    "forms": {
+      "▁snowboard": 91337
+    },
+    "encode": {
+      "token_ids": [
+        35514,
+        3334
+      ],
+      "tokens": [
+        "snow",
+        "board"
+      ]
+    }
+  },
+  "sports ball": {
+    "encode": {
+      "token_ids": [
+        32811,
+        5744
+      ],
+      "tokens": [
+        "sports",
+        "▁ball"
+      ]
+    }
+  },
+  "kite": {
+    "forms": {
+      "kite": 58060,
+      "▁kite": 72215,
+      "Kite": 221887,
+      "▁Kite": 121582
+    },
+    "encode": {
+      "token_ids": [
+        58060
+      ],
+      "tokens": [
+        "kite"
+      ]
+    },
+    "id": 58060,
+    "token": "kite"
+  },
+  "baseball bat": {
+    "encode": {
+      "token_ids": [
+        111021,
+        5085
+      ],
+      "tokens": [
+        "baseball",
+        "▁bat"
+      ]
+    }
+  },
+  "baseball glove": {
+    "encode": {
+      "token_ids": [
+        111021,
+        53535
+      ],
+      "tokens": [
+        "baseball",
+        "▁glove"
+      ]
+    }
+  },
+  "skateboard": {
+    "forms": {
+      "▁skateboard": 88403
+    },
+    "encode": {
+      "token_ids": [
+        126209,
+        3334
+      ],
+      "tokens": [
+        "skate",
+        "board"
+      ]
+    }
+  },
+  "surfboard": {
+    "forms": {
+      "▁surfboard": 211764
+    },
+    "encode": {
+      "token_ids": [
+        67948,
+        3334
+      ],
+      "tokens": [
+        "surf",
+        "board"
+      ]
+    }
+  },
+  "tennis racket": {
+    "encode": {
+      "token_ids": [
+        90092,
+        115124
+      ],
+      "tokens": [
+        "tennis",
+        "▁racket"
+      ]
+    }
+  },
+  "bottle": {
+    "forms": {
+      "bottle": 77455,
+      "▁bottle": 12989,
+      "Bottle": 113074,
+      "▁Bottle": 58299
+    },
+    "encode": {
+      "token_ids": [
+        77455
+      ],
+      "tokens": [
+        "bottle"
+      ]
+    },
+    "id": 77455,
+    "token": "bottle"
+  },
+  "wine glass": {
+    "encode": {
+      "token_ids": [
+        48758,
+        5570
+      ],
+      "tokens": [
+        "wine",
+        "▁glass"
+      ]
+    }
+  },
+  "cup": {
+    "forms": {
+      "cup": 10125,
+      "▁cup": 7190,
+      "Cup": 45490,
+      "▁Cup": 10701
+    },
+    "encode": {
+      "token_ids": [
+        10125
+      ],
+      "tokens": [
+        "cup"
+      ]
+    },
+    "id": 10125,
+    "token": "cup"
+  },
+  "fork": {
+    "forms": {
+      "fork": 49240,
+      "▁fork": 24943,
+      "Fork": 72844,
+      "▁Fork": 50819
+    },
+    "encode": {
+      "token_ids": [
+        49240
+      ],
+      "tokens": [
+        "fork"
+      ]
+    },
+    "id": 49240,
+    "token": "fork"
+  },
+  "knife": {
+    "forms": {
+      "knife": 47674,
+      "▁knife": 19193,
+      "Knife": 70855,
+      "▁Knife": 76352
+    },
+    "encode": {
+      "token_ids": [
+        47674
+      ],
+      "tokens": [
+        "knife"
+      ]
+    },
+    "id": 47674,
+    "token": "knife"
+  },
+  "spoon": {
+    "forms": {
+      "spoon": 169224,
+      "▁spoon": 35306,
+      "Spoon": 161623,
+      "▁Spoon": 103888
+    },
+    "encode": {
+      "token_ids": [
+        169224
+      ],
+      "tokens": [
+        "spoon"
+      ]
+    },
+    "id": 169224,
+    "token": "spoon"
+  },
+  "bowl": {
+    "forms": {
+      "bowl": 77093,
+      "▁bowl": 14581,
+      "Bowl": 153549,
+      "▁Bowl": 29270
+    },
+    "encode": {
+      "token_ids": [
+        77093
+      ],
+      "tokens": [
+        "bowl"
+      ]
+    },
+    "id": 77093,
+    "token": "bowl"
+  },
+  "banana": {
+    "forms": {
+      "banana": 68092,
+      "▁banana": 31985,
+      "Banana": 75581,
+      "▁Banana": 73639
+    },
+    "encode": {
+      "token_ids": [
+        68092
+      ],
+      "tokens": [
+        "banana"
+      ]
+    },
+    "id": 68092,
+    "token": "banana"
+  },
+  "apple": {
+    "forms": {
+      "apple": 22447,
+      "▁apple": 15491,
+      "Apple": 22732,
+      "▁Apple": 9865
+    },
+    "encode": {
+      "token_ids": [
+        22447
+      ],
+      "tokens": [
+        "apple"
+      ]
+    },
+    "id": 22447,
+    "token": "apple"
+  },
+  "sandwich": {
+    "forms": {
+      "sandwich": 159965,
+      "▁sandwich": 34176,
+      "Sandwich": 150174,
+      "▁Sandwich": 77980
+    },
+    "encode": {
+      "token_ids": [
+        159965
+      ],
+      "tokens": [
+        "sandwich"
+      ]
+    },
+    "id": 159965,
+    "token": "sandwich"
+  },
+  "orange": {
+    "forms": {
+      "orange": 22103,
+      "▁orange": 10436,
+      "Orange": 36565,
+      "▁Orange": 20723
+    },
+    "encode": {
+      "token_ids": [
+        22103
+      ],
+      "tokens": [
+        "orange"
+      ]
+    },
+    "id": 22103,
+    "token": "orange"
+  },
+  "broccoli": {
+    "forms": {
+      "▁broccoli": 74824,
+      "Broccoli": 227006,
+      "▁Broccoli": 152084
+    },
+    "encode": {
+      "token_ids": [
+        5220,
+        49279
+      ],
+      "tokens": [
+        "bro",
+        "ccoli"
+      ]
+    }
+  },
+  "carrot": {
+    "forms": {
+      "carrot": 176463,
+      "▁carrot": 64058,
+      "Carrot": 151432,
+      "▁Carrot": 146032
+    },
+    "encode": {
+      "token_ids": [
+        176463
+      ],
+      "tokens": [
+        "carrot"
+      ]
+    },
+    "id": 176463,
+    "token": "carrot"
+  },
+  "hot dog": {
+    "encode": {
+      "token_ids": [
+        5074,
+        5929
+      ],
+      "tokens": [
+        "hot",
+        "▁dog"
+      ]
+    }
+  },
+  "pizza": {
+    "forms": {
+      "pizza": 46722,
+      "▁pizza": 14967,
+      "Pizza": 44738,
+      "▁Pizza": 33787
+    },
+    "encode": {
+      "token_ids": [
+        46722
+      ],
+      "tokens": [
+        "pizza"
+      ]
+    },
+    "id": 46722,
+    "token": "pizza"
+  },
+  "donut": {
+    "forms": {
+      "donut": 166772,
+      "▁donut": 92350,
+      "Donut": 164820,
+      "▁Donut": 190951
+    },
+    "encode": {
+      "token_ids": [
+        166772
+      ],
+      "tokens": [
+        "donut"
+      ]
+    },
+    "id": 166772,
+    "token": "donut"
+  },
+  "cake": {
+    "forms": {
+      "cake": 23144,
+      "▁cake": 11514,
+      "Cake": 62932,
+      "▁Cake": 33524
+    },
+    "encode": {
+      "token_ids": [
+        23144
+      ],
+      "tokens": [
+        "cake"
+      ]
+    },
+    "id": 23144,
+    "token": "cake"
+  },
+  "chair": {
+    "forms": {
+      "chair": 26454,
+      "▁chair": 9205,
+      "Chair": 28774,
+      "▁Chair": 11893
+    },
+    "encode": {
+      "token_ids": [
+        26454
+      ],
+      "tokens": [
+        "chair"
+      ]
+    },
+    "id": 26454,
+    "token": "chair"
+  },
+  "couch": {
+    "forms": {
+      "couch": 132772,
+      "▁couch": 34675,
+      "Couch": 187889,
+      "▁Couch": 106871
+    },
+    "encode": {
+      "token_ids": [
+        132772
+      ],
+      "tokens": [
+        "couch"
+      ]
+    },
+    "id": 132772,
+    "token": "couch"
+  },
+  "potted plant": {
+    "encode": {
+      "token_ids": [
+        11403,
+        691,
+        4829
+      ],
+      "tokens": [
+        "pot",
+        "ted",
+        "▁plant"
+      ]
+    }
+  },
+  "bed": {
+    "forms": {
+      "bed": 2907,
+      "▁bed": 3609,
+      "Bed": 20401,
+      "▁Bed": 12741
+    },
+    "encode": {
+      "token_ids": [
+        2907
+      ],
+      "tokens": [
+        "bed"
+      ]
+    },
+    "id": 2907,
+    "token": "bed"
+  },
+  "dining table": {
+    "encode": {
+      "token_ids": [
+        97229,
+        3037
+      ],
+      "tokens": [
+        "dining",
+        "▁table"
+      ]
+    }
+  },
+  "toilet": {
+    "forms": {
+      "toilet": 134695,
+      "▁toilet": 19384,
+      "Toilet": 141901,
+      "▁Toilet": 73399
+    },
+    "encode": {
+      "token_ids": [
+        134695
+      ],
+      "tokens": [
+        "toilet"
+      ]
+    },
+    "id": 134695,
+    "token": "toilet"
+  },
+  "tv": {
+    "forms": {
+      "tv": 9540,
+      "▁tv": 9445,
+      "Tv": 41430,
+      "▁Tv": 74855
+    },
+    "encode": {
+      "token_ids": [
+        9540
+      ],
+      "tokens": [
+        "tv"
+      ]
+    },
+    "id": 9540,
+    "token": "tv"
+  },
+  "laptop": {
+    "forms": {
+      "laptop": 73556,
+      "▁laptop": 15690,
+      "Laptop": 117635,
+      "▁Laptop": 63054
+    },
+    "encode": {
+      "token_ids": [
+        73556
+      ],
+      "tokens": [
+        "laptop"
+      ]
+    },
+    "id": 73556,
+    "token": "laptop"
+  },
+  "mouse": {
+    "forms": {
+      "mouse": 17536,
+      "▁mouse": 11937,
+      "Mouse": 13857,
+      "▁Mouse": 27968
+    },
+    "encode": {
+      "token_ids": [
+        17536
+      ],
+      "tokens": [
+        "mouse"
+      ]
+    },
+    "id": 17536,
+    "token": "mouse"
+  },
+  "remote": {
+    "forms": {
+      "remote": 25888,
+      "▁remote": 11196,
+      "Remote": 23947,
+      "▁Remote": 28028
+    },
+    "encode": {
+      "token_ids": [
+        25888
+      ],
+      "tokens": [
+        "remote"
+      ]
+    },
+    "id": 25888,
+    "token": "remote"
+  },
+  "keyboard": {
+    "forms": {
+      "keyboard": 22959,
+      "▁keyboard": 20434,
+      "Keyboard": 32731,
+      "▁Keyboard": 46891
+    },
+    "encode": {
+      "token_ids": [
+        22959
+      ],
+      "tokens": [
+        "keyboard"
+      ]
+    },
+    "id": 22959,
+    "token": "keyboard"
+  },
+  "cell phone": {
+    "encode": {
+      "token_ids": [
+        2896,
+        5248
+      ],
+      "tokens": [
+        "cell",
+        "▁phone"
+      ]
+    }
+  },
+  "microwave": {
+    "forms": {
+      "▁microwave": 40335,
+      "Microwave": 179681,
+      "▁Microwave": 93639
+    },
+    "encode": {
+      "token_ids": [
+        2324,
+        29733
+      ],
+      "tokens": [
+        "mic",
+        "rowave"
+      ]
+    }
+  },
+  "oven": {
+    "forms": {
+      "oven": 112964,
+      "▁oven": 18403,
+      "Oven": 164360,
+      "▁Oven": 91129
+    },
+    "encode": {
+      "token_ids": [
+        112964
+      ],
+      "tokens": [
+        "oven"
+      ]
+    },
+    "id": 112964,
+    "token": "oven"
+  },
+  "toaster": {
+    "forms": {
+      "toaster": 234116,
+      "▁toaster": 126789,
+      "Toaster": 215011
+    },
+    "encode": {
+      "token_ids": [
+        234116
+      ],
+      "tokens": [
+        "toaster"
+      ]
+    },
+    "id": 234116,
+    "token": "toaster"
+  },
+  "sink": {
+    "forms": {
+      "sink": 60770,
+      "▁sink": 22340,
+      "Sink": 60705,
+      "▁Sink": 87026
+    },
+    "encode": {
+      "token_ids": [
+        60770
+      ],
+      "tokens": [
+        "sink"
+      ]
+    },
+    "id": 60770,
+    "token": "sink"
+  },
+  "refrigerator": {
+    "forms": {
+      "▁refrigerator": 46154,
+      "▁Refrigerator": 188820
+    },
+    "encode": {
+      "token_ids": [
+        170546,
+        1291
+      ],
+      "tokens": [
+        "refriger",
+        "ator"
+      ]
+    }
+  },
+  "book": {
+    "forms": {
+      "book": 2545,
+      "▁book": 2870,
+      "Book": 7229,
+      "▁Book": 6757
+    },
+    "encode": {
+      "token_ids": [
+        2545
+      ],
+      "tokens": [
+        "book"
+      ]
+    },
+    "id": 2545,
+    "token": "book"
+  },
+  "clock": {
+    "forms": {
+      "clock": 14153,
+      "▁clock": 12177,
+      "Clock": 30458,
+      "▁Clock": 40095
+    },
+    "encode": {
+      "token_ids": [
+        14153
+      ],
+      "tokens": [
+        "clock"
+      ]
+    },
+    "id": 14153,
+    "token": "clock"
+  },
+  "vase": {
+    "forms": {
+      "vase": 123002,
+      "▁vase": 34557,
+      "Vase": 206225,
+      "▁Vase": 94699
+    },
+    "encode": {
+      "token_ids": [
+        123002
+      ],
+      "tokens": [
+        "vase"
+      ]
+    },
+    "id": 123002,
+    "token": "vase"
+  },
+  "scissors": {
+    "forms": {
+      "scissors": 163803,
+      "▁scissors": 59283,
+      "Scissors": 147376,
+      "▁Scissors": 109049
+    },
+    "encode": {
+      "token_ids": [
+        163803
+      ],
+      "tokens": [
+        "scissors"
+      ]
+    },
+    "id": 163803,
+    "token": "scissors"
+  },
+  "teddy bear": {
+    "encode": {
+      "token_ids": [
+        139341,
+        8718
+      ],
+      "tokens": [
+        "teddy",
+        "▁bear"
+      ]
+    }
+  },
+  "hair drier": {
+    "encode": {
+      "token_ids": [
+        28499,
+        134028
+      ],
+      "tokens": [
+        "hair",
+        "▁drier"
+      ]
+    }
+  },
+  "toothbrush": {
+    "forms": {
+      "▁toothbrush": 103255
+    },
+    "encode": {
+      "token_ids": [
+        12521,
+        31453
+      ],
+      "tokens": [
+        "tooth",
+        "brush"
+      ]
+    }
+  }
+}

tokenizer_vocab/common_words.json

@@ -0,0 +1,190 @@
+{
+  "a": {
+    "token": "a",
+    "id": 235250
+  },
+  "the": {
+    "token": "the",
+    "id": 1175
+  },
+  "an": {
+    "token": "an",
+    "id": 481
+  },
+  "cat": {
+    "token": "cat",
+    "id": 4991
+  },
+  "dog": {
+    "token": "dog",
+    "id": 12240
+  },
+  "bird": {
+    "token": "bird",
+    "id": 18424
+  },
+  "fish": {
+    "token": "fish",
+    "id": 11578
+  },
+  "person": {
+    "token": "person",
+    "id": 6717
+  },
+  "man": {
+    "token": "man",
+    "id": 1066
+  },
+  "woman": {
+    "token": "woman",
+    "id": 14172
+  },
+  "child": {
+    "token": "child",
+    "id": 3647
+  },
+  "car": {
+    "token": "car",
+    "id": 2269
+  },
+  "bike": {
+    "token": "bike",
+    "id": 31674
+  },
+  "bicycle": {
+    "token": "bicycle",
+    "id": 134234
+  },
+  "bus": {
+    "token": "bus",
+    "id": 6710
+  },
+  "train": {
+    "token": "train",
+    "id": 8297
+  },
+  "house": {
+    "token": "house",
+    "id": 6111
+  },
+  "building": {
+    "token": "building",
+    "id": 18802
+  },
+  "tree": {
+    "token": "tree",
+    "id": 9197
+  },
+  "flower": {
+    "token": "flower",
+    "id": 18697
+  },
+  "red": {
+    "token": "red",
+    "id": 854
+  },
+  "blue": {
+    "token": "blue",
+    "id": 8796
+  },
+  "green": {
+    "token": "green",
+    "id": 9740
+  },
+  "yellow": {
+    "token": "yellow",
+    "id": 22006
+  },
+  "black": {
+    "token": "black",
+    "id": 6944
+  },
+  "white": {
+    "token": "white",
+    "id": 7384
+  },
+  "big": {
+    "token": "big",
+    "id": 7773
+  },
+  "small": {
+    "token": "small",
+    "id": 10498
+  },
+  "large": {
+    "token": "large",
+    "id": 13210
+  },
+  "tiny": {
+    "token": "tiny",
+    "id": 35282
+  },
+  "running": {
+    "token": "running",
+    "id": 23655
+  },
+  "walking": {
+    "token": "walking",
+    "id": 61916
+  },
+  "sitting": {
+    "token": "sitting",
+    "id": 116373
+  },
+  "standing": {
+    "token": "standing",
+    "id": 8898
+  },
+  "eating": {
+    "token": "eating",
+    "id": 71409
+  },
+  "drinking": {
+    "token": "drinking",
+    "id": 139848
+  },
+  "sleeping": {
+    "token": "sleeping",
+    "id": 113608
+  },
+  "photo": {
+    "token": "photo",
+    "id": 5817
+  },
+  "image": {
+    "token": "image",
+    "id": 2502
+  },
+  "picture": {
+    "token": "picture",
+    "id": 12933
+  },
+  "of": {
+    "token": "of",
+    "id": 559
+  },
+  "in": {
+    "token": "in",
+    "id": 473
+  },
+  "on": {
+    "token": "on",
+    "id": 477
+  },
+  "at": {
+    "token": "at",
+    "id": 482
+  },
+  "with": {
+    "token": "with",
+    "id": 3041
+  },
+  "and": {
+    "token": "and",
+    "id": 639
+  },
+  "or": {
+    "token": "or",
+    "id": 483
+  }
+}

tokenizer_vocab/special_tokens.json

@@ -0,0 +1,10 @@
+{
+  "pad_token": "<pad>",
+  "pad_token_id": 0,
+  "eos_token": "<eos>",
+  "eos_token_id": 1,
+  "bos_token": "<bos>",
+  "bos_token_id": 2,
+  "unk_token": "<unk>",
+  "unk_token_id": 3
+}