mimelens-001 cell: tiny/bpe-64k/s2

README.md

@@ -1,17 +1,22 @@
 ---
 license: mit
 library_name: transformers
+language:
+  - en
 tags:
   - file-type-detection
   - mime-classification
   - binary-content
+  - binary-analysis
   - position-agnostic
   - libmagic
+  - forensics
+  - packet-inspection
   - bpe
+  - byte-pair-encoding
   - mimelens
-language: en
 base_model: mjbommar/binary-tokenizer-001-64k
-pipeline_tag: feature-extraction
+pipeline_tag: text-classification
 model-index:
   - name: mimelens-001-tiny-bpe-64k-s2
     results:
@@ -19,8 +24,8 @@ model-index:
           type: feature-extraction
           name: MIME-125 classification (libmagic 125-class taxonomy)
         dataset:
-          name: magic-bpe magic-frags (4 KB head of 64 KB random chunks, n=4,096)
-          type: mjbommar/magic-bpe-stratified
+          name: magic-frags (4 KB head of 64 KB random chunks, n=4,096)
+          type: custom
         metrics:
           - name: top-1 accuracy
             type: accuracy
@@ -33,95 +38,133 @@ model-index:
             value: 0.6754
         source:
           name: "MimeLens paper (Bommarito 2026), Appendix A"
-          url: https://github.com/mjbommar/binary-embedding-paper
+          url: https://github.com/mjbommar/mimelens-training
 ---
 
-# MimeLens-001 / tiny / bpe-64k / s2
+# mimelens-001-tiny-bpe-64k-s2
 
-**One cell from the [MimeLens-001](https://huggingface.co/mjbommar/mimelens-001) family** — `3.15` M backbone params, `bpe-64k` input pipeline, seed `2`. Pretrained MLM-only on 33 GB of position-arbitrary binary content for fine-grained file-content-type classification under [libmagic](https://github.com/file/file)'s 125-class MIME taxonomy.
+A 3.15M-backbone-parameter BERT-style encoder for position-agnostic file-content-type detection from binary data. It reads a byte window taken from *any* offset in a file (the first ~1{,}022 tokens of whatever you pass) and produces a 256-dimensional embedding that classifiers map to one of [libmagic](https://github.com/file/file)'s 125 MIME labels. Designed for inputs where you only have a chunk: a forensic-carved fragment, a random disk-block read, a streaming HTTP upload, a single network packet payload.
+
+- **🔗 Model**: [`mjbommar/mimelens-001-tiny-bpe-64k-s2`](https://huggingface.co/mjbommar/mimelens-001-tiny-bpe-64k-s2)
+- **👥 Family**: [`mjbommar/mimelens-001`](https://huggingface.co/mjbommar/mimelens-001) (36 released cells: 28 parent + 8 short-sequence)
+- **🔤 Tokenizer**: [`mjbommar/binary-tokenizer-001-64k`](https://huggingface.co/mjbommar/binary-tokenizer-001-64k)
+- **📄 Paper**: *MimeLens: Position-Agnostic Content-Type Detection for Binary Fragments* (Bommarito 2026)
+- **💻 Training code**: [`mjbommar/mimelens-training`](https://github.com/mjbommar/mimelens-training)
+- **📊 Pretraining corpus**: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized) plus magic-corpus extracts, packed binaries, a [`glaurung`](https://github.com/mjbommar/glaurung)-sourced binary corpus, and Windows drivers (33 GB stratified; the full corpus is not redistributable)
+
+---
+
+## What MimeLens does
+
+MimeLens classifies file content type from a byte window taken at any offset, not just the header of a complete file.
+
+Existing tools assume whole-file access at a known offset:
+
+- [`libmagic`](https://github.com/file/file) and [Apache Tika](https://tika.apache.org/) match handcrafted magic-byte signatures, almost always anchored at the file head.
+- [Magika](https://github.com/google/magika) (Google) is a small (~1 M-parameter) feedforward network over three 512-byte windows (head, middle, tail) of a known-bounded file.
+- TrID, PRONOM/Siegfried/DROID similarly require a complete file.
+
+These break down on a fragment. MimeLens is pretrained MLM-only on 1024-token windows sampled *uniformly at random* across files and 64 KB fragments, with no privileged head-of-file position. One checkpoint handles streaming, partial-arrival, mid-file, packet-payload, and forensic-carved inputs uniformly. The trade-off is CPU latency (roughly two orders of magnitude slower than Magika at the medium size; hardware-dependent) in exchange for libmagic's 125-class taxonomy plus position arbitrariness.
+
+The family ships 28 parent cells (3 sizes × 4 vocabs × 2-3 seeds at seq\_len=1024) plus an 8-cell short-sequence extension (medium tier × 4 vocabs × 2 seeds at seq\_len=256). This README documents one of them.
+
+> **Short-sequence sibling available.** If your inputs are sub-KB (DNS payloads, sub-MTU packets, small forensic fragments), use `mjbommar/mimelens-001-tiny-bpe-64k-s2-seq256` instead. Same architecture, 4× shorter context, ~5× lower CPU latency, BPE-cell accuracy ties or beats this cell on the magic-files probe-fit. See paper Appendix B.5.
 
-A single 4 KB byte buffer in (of which the first 1,022 body tokens are consumed), one of libmagic's 125 MIME labels out, regardless of where in a source file the buffer came from.
 
-For the family overview, decision tree (which cell to load?), and full cube results, see [`mjbommar/mimelens-001`](https://huggingface.co/mjbommar/mimelens-001).
 
-## How to use
+---
+
+## Overview
+
+- **This cell**: `tiny` tier, `bpe-64k` input pipeline, seed `2`
+- **Backbone**: 3.15M parameters (4 layers, hidden 256, 4 attention heads, head dim 64, RoPE, RMSNorm, no biases, no dropout)
+- **Input vocabulary**: `bpe-64k`. 65,536-entry binary BPE tokenizer (binary-tokenizer-001-64k), ~2.09 bytes/token. Reads ~2,134 bytes of the 4 KB buffer.
+- **Output**: 256-dim mean-pooled body-token embedding
+- **Label space**: [libmagic](https://github.com/file/file) 125-class MIME taxonomy (full list in paper Appendix)
+- **Pretraining**: MLM-only, 30% mask ratio, 33 GB stratified multi-source binary corpus, 22,888 gradient updates, single RTX 4060 Ti, ~2.7 h wall-clock
+- **License**: MIT
+
+## Headline benchmarks (this cell)
+
+| Benchmark | Value |
+|---|---|
+| MIME-125 top-1 (magic-frags, 4 KB head, n=4,096)            | **0.732** |
+| MIME-125 macro-F1 (magic-frags, 4 KB head)                  | 0.609 |
+| kNN R@1 (magic-frags, 3,147-file gallery / 949 queries)     | 0.675 |
+
+Full evaluation (within-cube bootstrap CIs, adversarial sweep, calibration, real-network curves, disk-block matrix, baselines against libmagic 5.46 and TrID 2.24) is in the [paper](https://github.com/mjbommar/mimelens-training).
+
+---
+
+## Quick start
+
+This cell publishes the encoder only (no classifier head baked in). Use it to extract embeddings, then fit a probe, run kNN over a labelled gallery, or fine-tune a head:
 
 ```python
 import torch
-from transformers import AutoModel
-from tokenizers import Tokenizer
+from transformers import AutoModel, AutoTokenizer
 
-repo = "mjbommar/mimelens-001-tiny-bpe-64k-s2"
+repo  = "mjbommar/mimelens-001-tiny-bpe-64k-s2"
 model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
-tok = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-64k")
-cfg = model.config
+tok   = AutoTokenizer.from_pretrained(repo)
 
-# BPE cell: encode 4 KB of raw bytes via the published binary-BPE tokenizer.
 window = open("path/to/file", "rb").read(4096)
-body = tok.encode(window.decode("latin-1")).ids[:1022]
-ids = [cfg.cls_token_id] + body + [cfg.sep_token_id]
-input_ids = torch.tensor([ids])
-
+inputs = tok(window.decode("latin-1"), max_length=1024, truncation=True,
+             padding="max_length", return_tensors="pt")
 with torch.no_grad():
-    out = model(input_ids=input_ids, attention_mask=torch.ones_like(input_ids))
-
-embedding = out.pooler_output     # (1, 256)  mean-pooled body-token embedding
-# Downstream: a frozen LR probe, a kNN over a labeled gallery, or fine-tune a classification head.
-# See the paper for the standard evaluation protocol.
+    embedding = model(**inputs).pooler_output         # (1, 256)
 ```
 
-## What this cell is
+The pre-fit LR probe weights for this cell are not bundled here. The deployed cells and per-size winners (e.g. `mimelens-001-medium-bpe-16k-s1`) ship a baked classifier head for a one-line `pipeline()` path.
 
-- **Family**: [MimeLens-001](https://huggingface.co/mjbommar/mimelens-001) — 28 pretrained checkpoints across 3 sizes × 4 vocabularies × 2 seeds, plus one matched-tokens-seen ablation.
-- **Size**: `tiny` — 3.15 M backbone params, 4 layers, hidden 256, 4 attention heads, head dim 64.
-- **Input pipeline**: `bpe-64k` (65{,}536-entry binary BPE tokenizer (from binary-tokenizer-001-64k), ~2.09 bytes per token on the corpus.).
-- **Seed**: `2` (1 of 2 for this (size, vocab) combination).
-- **Pretraining**: 22,888 gradient updates, MLM-only, 30% mask ratio, 1024-token windows sampled uniformly at random across files and 64 KB fragments. AdamW + cosine LR (peak 5e-4, 2,000-step warmup, 10% floor), bf16 mixed precision, single RTX 4060 Ti.
-- **License**: MIT.
 
-## Evaluation
+---
 
-Numbers below are for **this specific cell** on the `magic-frags` held-out test set (4 KB head of 64 KB random chunks, n=4,096). The within-cube comparison (3 sizes × 4 vocabs × 2-3 seeds, bootstrap CIs, adversarial sweep, calibration, real-network and disk-block validations) is in the [paper](https://github.com/mjbommar/binary-embedding-paper).
+## Choosing a window
 
-| Benchmark | This cell |
-|---|---|
-| MIME-125 top-1 (magic-frags 4 KB head, n=4,096)        | **0.732** |
-| MIME-125 macro-F1 (magic-frags 4 KB head)              | 0.609 |
-| kNN R@1 (magic-frags, 3,147-file gallery / 949 queries) | 0.675 |
+The model reads the first ~1{,}022 tokens of whatever you pass — a prefix of the buffer (for this BPE cell, whatever tokenizes to ~1{,}022 tokens, typically the first ~1.5--2.5 KB), not the whole window.
+
+- **Magic-byte / compressed types** (PNG, ZIP, GZIP, JPEG): a **short head window (256 B--1 KB) classifies better than 4 KB**. A long high-entropy body dilutes the header signal within the fixed token budget, and the model returns `application/octet-stream` on a mostly-opaque window — correct behaviour for genuinely high-entropy input, not a bug.
+- **Fragments / packets**: you cannot choose the offset, so pass what you have. This is the regime MimeLens is built for.
+
+---
 
 ## Recommended deployment regimes
 
 See the family hub README ([`mjbommar/mimelens-001`](https://huggingface.co/mjbommar/mimelens-001)) for the regime decision tree.
 
+---
+
 ## Training
 
-This cell is one point of the pre-registered 3 × 4 × 2 factorial cube described in the [MimeLens paper](https://github.com/mjbommar/binary-embedding-paper). Salient details:
+This cell is one point of the 3 × 4 × 2 factorial cube described in the paper.
 
-- **33 GB stratified multi-source binary corpus** (binary-30k + magic-frags + glaurung + Windows drivers).
-- **Position-arbitrary windowing**: 1024-token windows sampled uniformly at random across files and 64 KB fragments — no privileged "head of file" position. This is what makes MimeLens work on streaming / partial / random-offset inputs that whole-file detectors were not designed for.
-- **MLM-only** objective, 30% mask ratio (BERT replacement schedule: 80% `[MASK]`, 10% random, 10% original); tied input/output embeddings.
-- **Mean-pool over body tokens** for downstream tasks; the BERT-style `cls_pool` linear projection is *not* used because under MLM-only training it receives no gradient and remains at random init across all 28 cube cells (paper §3.4 verifies this).
-- **Wall-clock**: ~2.7 h on a single RTX 4060 Ti.
+- **Corpus** (33 GB, stratified multi-source): [`binary-30k`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized) (assorted ELF/PE/Mach-O), magic-frags (random 64 KB chunks across libmagic's full corpus), assorted packed/raw binaries, a [`glaurung`](https://github.com/mjbommar/glaurung)-sourced binary corpus, Windows drivers.
+- **Position-arbitrary windowing**: 1024-token windows sampled uniformly at random across files and 64 KB fragments. **No privileged "head of file" position.** This is the design choice that makes MimeLens work on streaming / partial / random-offset inputs.
+- **Objective**: MLM with 30% mask ratio (BERT replacement schedule: 80% `[MASK]`, 10% random, 10% original); tied input/output embeddings.
+- **Pooling**: mean-pool over body tokens for downstream tasks. The BERT-style `cls_pool` linear projection is *not* used: under MLM-only training it receives no gradient and remains byte-identical to its random initialisation across all 28 cube cells (paper §3.4 verifies this; left in the saved weights for architectural completeness only).
+- **Optimisation**: AdamW + cosine LR (peak 5e-4, 2,000-step warmup, 10% floor), bf16 mixed precision, gradient clipping at $\|g\|_2 \leq 1$, effective batch 128 at sequence length 1024, 22,888 gradient updates.
+- **Hardware**: single RTX 4060 Ti (16 GB), ~2.7 h wall-clock for this cell.
 
-## Honest caveats
+---
 
-- This is one cell of a 28-cell cube. Within-cube comparisons in the paper come with bootstrap CIs at n=2 seeds; some marginal orderings (byte vs bpe-16k at the top of medium) are within seed noise and should be read as ties.
+## Caveats
+
+- This is one cell of a 28-cell parent cube (36 released cells including the 8-cell short-sequence extension). Within-cube comparisons in the paper carry bootstrap CIs at n=2 seeds; some marginal orderings (byte vs bpe-16k at the largest size) are within seed noise and should be read as ties.
 - The training corpus is one 33 GB stratified multi-source binary sample. Results may not transfer to substantially different corpora.
-- All numbers are computed on data derived from a single labelling pipeline (libmagic-pinned via the [magic-bpe](https://github.com/mjbommar/magic-bpe) project). Cross-validation against PRONOM, Siegfried, DROID, or IANA reference files is a documented limitation.
-- CPU latency at the `medium` size is ~348× slower than Magika; for sub-millisecond whole-file triage on broad categories, Magika is purpose-built and is the right default. MimeLens occupies a different point on the deployment surface (position-arbitrary inputs + libmagic's 125-class taxonomy) rather than a drop-in replacement.
-- End-to-end fine-tuning on the production label distribution may shift these numbers and should be evaluated before deployment. The frozen-probe numbers reported above are not claimed as a lower bound on fine-tuned performance.
+- All numbers are computed on data labelled by a single pipeline (libmagic-pinned). Cross-validation against PRONOM, Siegfried, DROID, or IANA reference files is a documented limitation.
+- CPU latency at the `medium` size is ~155× slower than Magika v1.1 on a desktop CPU (hardware-dependent). For sub-millisecond whole-file triage on broad categories, Magika is purpose-built and is the right tool. MimeLens occupies a different point on the deployment surface (position-arbitrary inputs + libmagic's 125-class taxonomy), not a drop-in replacement.
+- End-to-end fine-tuning on the production label distribution may shift these numbers and should be evaluated before deployment. The frozen-probe numbers above are not claimed as a lower bound on fine-tuned performance.
+
+---
 
 ## Citation
 
 ```bibtex
 @misc{bommarito2026mimelens,
-  title  = {MimeLens: Pretrained Encoders for Fine-Grained Content-Type Detection},
+  title  = {MimeLens: Position-Agnostic Content-Type Detection for Binary Fragments},
   author = {Bommarito II, Michael J.},
   year   = {2026},
-  note   = {https://github.com/mjbommar/binary-embedding-paper},
+  note   = {https://github.com/mjbommar/mimelens-training},
 }
-```
-
-## Acknowledgments
-
-Thanks to the [magic-bpe](https://github.com/mjbommar/magic-bpe) project and the [binary-tokenizer-001](https://huggingface.co/mjbommar/binary-tokenizer-001-64k) family for the labelled corpus and BPE tokenizers this work builds on, and to the [Magika](https://github.com/google/magika) team for releasing a public package that made the §3 calibration possible.
+```

config.json

@@ -22,7 +22,7 @@
   "cls_token_id": 4,
   "sep_token_id": 5,
   "mask_token_id": 6,
-  "byte_offset": 5,
+  "byte_offset": 7,
   "cls_pool_dim": 256,
   "mimelens_cell_id": "tiny/bpe-64k/s2",
   "mimelens_vocab_pipeline": "bpe-64k",

configuration_mimelens.py

@@ -28,8 +28,9 @@ class MimeLensConfig(PretrainedConfig):
     paper repository (https://github.com/mjbommar/binary-embedding-paper).
 
     Args:
-        vocab_size: int — full vocabulary including 5 special tokens. byte
-            cells: 261 (256 bytes + 5 specials). BPE cells: 4101 / 16391 / 65543.
+        vocab_size: int — full vocabulary including 7 special tokens (start, end,
+            pad, unk, cls, sep, mask). byte cells: 263 (256 bytes + 7 specials).
+            BPE cells: 4103 / 16391 / 65543.
         hidden_size: int — transformer model dimension (256 / 384 / 512 for
             tiny / small / medium).
         num_hidden_layers: int — layer count (4 / 8 / 12 for tiny / small /
@@ -46,7 +47,7 @@ class MimeLensConfig(PretrainedConfig):
         pad_token_id / cls_token_id / sep_token_id / mask_token_id: int —
             special-token indices, matching binary_embedding.constants.
         byte_offset: int — for byte cells, ord(b)+byte_offset gives the token
-            id. Fixed at 5 (after the 5 special tokens). Unused for BPE cells.
+            id. Fixed at 7 (after the 7 special tokens). Unused for BPE cells.
         cls_pool_dim: int — output dim of the cls_pool layer. Note: this layer
             receives no gradient under MLM-only training (see paper §3.4); the
             mean-pool over body tokens is the trained pooling, not cls_pool.
@@ -82,7 +83,7 @@ class MimeLensConfig(PretrainedConfig):
         cls_token_id: int = 4,
         sep_token_id: int = 5,
         mask_token_id: int = 6,
-        byte_offset: int = 5,
+        byte_offset: int = 7,
         cls_pool_dim: int = 256,
         initializer_range: float = 0.02,
         mimelens_cell_id: str = "medium/bpe-16k/s1",

modeling_mimelens.py

@@ -32,7 +32,7 @@ import torch
 import torch.nn as nn
 import torch.nn.functional as F
 from transformers import PreTrainedModel
-from transformers.modeling_outputs import BaseModelOutputWithPooling
+from transformers.modeling_outputs import BaseModelOutputWithPooling, SequenceClassifierOutput
 
 from .configuration_mimelens import MimeLensConfig
 
@@ -272,3 +272,103 @@ class MimeLensModel(PreTrainedModel):
         attention_mask = torch.tensor([attn], dtype=torch.long, device=device)
         with torch.inference_mode():
             return self(input_ids, attention_mask=attention_mask).pooler_output
+
+
+class MimeLensForSequenceClassification(PreTrainedModel):
+    """MimeLens encoder + a 125-class libmagic-MIME classifier head.
+
+    Lets users do, in one line:
+
+        from transformers import pipeline
+        clf = pipeline("text-classification",
+                       model="mjbommar/mimelens-001-medium-bpe-16k-s1",
+                       trust_remote_code=True)
+        clf(open("some.bin", "rb").read(4096).decode("latin-1"))
+        # → [{"label": "text/x-python", "score": 0.91}, ...]
+
+    The classifier head is the same logistic-regression probe the paper
+    reports on the magic-files corpus, re-fit on the full 4,096-file
+    labelled set and baked into `model.safetensors` as `classifier.weight`
+    and `classifier.bias`. Labels live in `config.id2label` / `config.label2id`.
+
+    For embedding-only use, load via `AutoModel.from_pretrained(...)` instead,
+    which returns mean-pooled embeddings and ignores the classifier head.
+    """
+
+    config_class = MimeLensConfig
+    base_model_prefix = "mimelens"
+
+    def __init__(self, config: MimeLensConfig):
+        super().__init__(config)
+        self.config = config
+        self.num_labels = getattr(config, "num_labels", 125)
+        # The encoder body, identical to MimeLensModel — same parameter names so
+        # the encoder weights load from the same safetensors keys.
+        self.embed = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.layers = nn.ModuleList([Layer(config) for _ in range(config.num_hidden_layers)])
+        self.final_norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.cls_pool = nn.Linear(config.hidden_size, config.cls_pool_dim, bias=False)
+        # The 125-way classifier head.
+        self.classifier = nn.Linear(config.hidden_size, self.num_labels)
+
+        self._rope_cache: Optional[tuple[torch.Tensor, torch.Tensor]] = None
+        self._rope_cache_meta: Optional[tuple[torch.device, torch.dtype, int]] = None
+        self.post_init()
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+    def _get_rope(self, seq_len: int, device: torch.device, dtype: torch.dtype):
+        meta = (device, dtype, seq_len)
+        if self._rope_cache_meta != meta:
+            self._rope_cache = _build_rope_cache(seq_len, self.config.head_dim,
+                                                 self.config.rope_theta,
+                                                 device=device, dtype=dtype)
+            self._rope_cache_meta = meta
+        return self._rope_cache
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        return_dict: bool = True,
+    ):
+        B, S = input_ids.shape
+        x = self.embed(input_ids)
+
+        if attention_mask is None:
+            attention_mask = torch.ones(B, S, device=input_ids.device, dtype=torch.long)
+        attn_mask = attention_mask.to(x.dtype)
+        attn_mask = (1.0 - attn_mask).masked_fill((1.0 - attn_mask).bool(),
+                                                    torch.finfo(x.dtype).min)
+        attn_mask = attn_mask.view(B, 1, 1, S)
+
+        cos, sin = self._get_rope(S, device=x.device, dtype=x.dtype)
+        for layer in self.layers:
+            x = layer(x, cos, sin, attn_mask)
+        x = self.final_norm(x)
+
+        lens = attention_mask.sum(dim=1, keepdim=True)
+        positions = torch.arange(S, device=x.device).unsqueeze(0)
+        body_mask = (positions >= 1) & (positions < (lens - 1))
+        body_mask_f = body_mask.to(x.dtype).unsqueeze(-1)
+        pooled = (x * body_mask_f).sum(dim=1) / body_mask_f.sum(dim=1).clamp(min=1)
+
+        # Cast pooled to classifier dtype (bf16 encoder + fp32 classifier is common).
+        logits = self.classifier(pooled.to(self.classifier.weight.dtype))
+
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(logits, labels)
+
+        if not return_dict:
+            return (loss, logits) if loss is not None else (logits,)
+        return SequenceClassifierOutput(loss=loss, logits=logits)

tokenizer_config.json

@@ -0,0 +1,54 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_max_length": 1024,
+  "padding_side": "right",
+  "truncation_side": "right",
+  "pad_token": "[PAD]",
+  "unk_token": "[UNK]",
+  "cls_token": "[CLS]",
+  "sep_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "clean_up_tokenization_spaces": false,
+  "added_tokens_decoder": {
+    "2": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    },
+    "4": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    },
+    "5": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    },
+    "6": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    }
+  }
+}