---
language: th
tags:
- thai
- tokenizer
- nlp
- text-processing
license: mit
---

# ZombitX64 Thai Tokenizer

A simple Thai language tokenizer that properly handles newlines and Thai text segmentation.

## Features

- **Newline Preservation**: Correctly handles and preserves newlines in tokenized text
- **Thai Character Support**: Recognizes and processes Thai Unicode characters
- **Hugging Face Compatible**: Works with transformers library
- **Simple API**: Easy to use tokenize and detokenize methods

## Usage

```python
from transformers import AutoTokenizer

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("ZombitX64/zombitx64-thaitokenizer")

# Tokenize text
text = "สวัสดีครับ\nนี่คือตัวอย่าง"
tokens = tokenizer.tokenize(text)
print(tokens)

# Encode to IDs
token_ids = tokenizer.encode(text)
print(token_ids)

# Decode back
decoded = tokenizer.decode(token_ids)
print(decoded)
```

## Model Details

- **Model Type**: Thai Tokenizer
- **Language**: Thai (th)
- **Vocab Size**: 112
- **Max Length**: 512

## Training Data

This tokenizer was trained on basic Thai character sets and common patterns.

## Limitations

- Basic Thai word segmentation (can be improved with pythainlp)
- Simple vocabulary (expandable for specific use cases)

## Contact

For questions or issues, please visit the [GitHub repository](https://github.com/ZombitX64/ZombitX64-Thaitokenizer).