---
title: Tokenizer Playground
emoji: 🔤
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.2.0
app_file: app.py
pinned: false
---

# Tokenizer Playground

Inspect how text is tokenized with any Hugging Face tokenizer. Use it locally or on [Hugging Face Spaces](https://huggingface.co/spaces).

## Features

- **Input text** — Main textarea to type or paste text.
- **Tokenization** — See each token with its ID in the form `token (id)`.
- **Stats** — Character count, token count, chars per token.
- **Tokenizer config** — EOS, PAD, BOS, UNK, SEP, CLS tokens and IDs; `model_max_length`, `vocab_size`.
- **Chat template** — Raw template string and an example with `apply_chat_template`.

## Usage

1. Enter a **model / tokenizer ID** (e.g. `gpt2`, `meta-llama/Llama-2-7b-hf`, `Qwen/Qwen2.5-0.5B`).
2. Optionally click **Load tokenizer & refresh config** to load the tokenizer and see config + chat template.
3. Type or paste text in **Input text** and click **Tokenize** to see tokens, IDs, and stats.

## Running locally

No PyTorch/Torch is required — the app uses only the tokenizer and config, so installs stay small.

```bash
pip install -r requirements.txt
python app.py
```

## Deploying on Hugging Face Spaces

Push this repo to a Space with **Gradio** SDK. The `README` frontmatter above configures the Space; `app.py` and `requirements.txt` are used automatically.