README.md

---

language: th
tags:
- thai
- tokenizer
- nlp
- text-processing
license: mit
---


# ZombitX64 Thai Tokenizer

A simple Thai language tokenizer that properly handles newlines and Thai text segmentation.

## Features

- **Newline Preservation**: Correctly handles and preserves newlines in tokenized text
- **Thai Character Support**: Recognizes and processes Thai Unicode characters
- **Hugging Face Compatible**: Works with transformers library
- **Simple API**: Easy to use tokenize and detokenize methods

## Usage

```python

from transformers import AutoTokenizer



# Load the tokenizer

tokenizer = AutoTokenizer.from_pretrained("ZombitX64/zombitx64-thaitokenizer")



# Tokenize text

text = "สวัสดีครับ\nนี่คือตัวอย่าง"

tokens = tokenizer.tokenize(text)

print(tokens)



# Encode to IDs

token_ids = tokenizer.encode(text)

print(token_ids)



# Decode back

decoded = tokenizer.decode(token_ids)

print(decoded)

```

## Model Details

- **Model Type**: Thai Tokenizer
- **Language**: Thai (th)
- **Vocab Size**: 112
- **Max Length**: 512

## Training Data

This tokenizer was trained on basic Thai character sets and common patterns.

## Limitations

- Basic Thai word segmentation (can be improved with pythainlp)
- Simple vocabulary (expandable for specific use cases)

## Contact

For questions or issues, please visit the [GitHub repository](https://github.com/ZombitX64/ZombitX64-Thaitokenizer).