---
language:
- az
license: apache-2.0
library_name: transformers
pipeline_tag: token-classification
tags:
- tokenizer
- wordpiece
- azerbaijani
- bert
- nlp
- transformers
---

# Azerbaijani WordPiece Tokenizer (`wp_2_uncased`)

This repository contains the uncased Azerbaijani WordPiece tokenizer used for AzNEOBERT pretraining.

## Overview

- Tokenizer type: WordPiece
- Vocabulary size: 64,000
- Casing: uncased
- Mean fertility: **1.727**
- Backend: Hugging Face `tokenizers` + `transformers`

The tokenizer was selected after evaluating six tokenizer variants on Azerbaijani corpora using fertility as the primary selection criterion.

## Training Data

The tokenizer was trained on approximately:

- ~100 GB Azerbaijani text
- 95.7B characters
- 10 corpus collections

The corpus includes diverse web, news, encyclopedic, and general-domain Azerbaijani text.

## Tokenization Efficiency

| Tokenizer | Mean Fertility |
|---|---:|
| mBERT | 2.846 |
| XLM-R | 2.167 |
| HPLT az-BERT | 2.068 |
| **wp_2_uncased** | **1.727** |

Lower fertility indicates more efficient tokenization.

Evaluation was performed on 47,934 Azerbaijani documents.

## Tokenizer Variants Evaluated

Six tokenizer variants were compared:

| Family | Variants |
|---|---|
| WordPiece | cased / uncased |
| SentencePiece Unigram | cased / uncased |
| SentencePiece BPE | cased / uncased |

The final selected tokenizer was the uncased WordPiece variant (`wp_2_uncased`).

## Usage

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained(
    "raufibishov/az-wordpiece-tokenizer"
)

text = "müqavilələrindən"

print(tok.tokenize(text))
```

Example output:

```python
['müqavilələrin', '##dən']
```

## Special Tokens

| Token | Purpose |
|---|---|
| `[UNK]` | Unknown token |
| `[CLS]` | Classification token |
| `[SEP]` | Separator token |
| `[PAD]` | Padding token |
| `[MASK]` | Masked language modeling token |

## Compatibility

This tokenizer supports:

- `AutoTokenizer`
- `PreTrainedTokenizerFast`
- `BertTokenizerFast`

## Citation

If you use this tokenizer in research, please cite the associated AzNEOBERT work.

## License

Apache-2.0