---
library_name: transformers
pipeline_tag: text-generation
tags:
- causal-lm
- trust-remote-code
- sentencepiece
---

# Pinyin-Code Causal LM

This repository contains a custom Transformers causal language model.
External evaluation repositories should load it with
`trust_remote_code=True` and use the `causal` backend.

## Dependencies

Install the runtime dependencies before loading the model:

```bash
pip install torch transformers safetensors sentencepiece pypinyin jieba
```

`sentencepiece` is required for the SentencePiece tokenizer. `pypinyin` is required
for raw Mandarin-to-pinyin preprocessing. `jieba` is required when
`use_jieba` is true; this export was created with `use_jieba=true`.

## Loading

```python
from transformers import (
    AutoConfig,
    AutoModel,
    AutoModelForCausalLM,
    AutoModelForSequenceClassification,
    AutoTokenizer,
)

model_path = "PATH_OR_REPO_ID"

config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
base_model = AutoModel.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
classifier = AutoModelForSequenceClassification.from_pretrained(
    model_path,
    trust_remote_code=True,
    num_labels=3,
)
```

## Evaluation

Configure external evaluators with:

- model path: this local folder or Hugging Face repo ID
- backend: `causal`
- trust remote code: enabled

The tokenizer accepts raw text through standard calls such as
`tokenizer(text)`, `tokenizer(text, add_special_tokens=False)`, and
`tokenizer(texts, padding=True, truncation=True, return_tensors="pt")`.
It also accepts `return_offsets_mapping=True` for compatibility with
completion-ranking evaluators that need suffix masks. The model supports
`output_hidden_states=True` for representation extraction tasks.

This export sets `patch_pathlib_utf8_open=true` in `config.json`.
When loaded with `trust_remote_code=True`, the config installs a narrow
Windows compatibility shim so later text-mode `Path.open("r")` calls
without an explicit encoding default to UTF-8. Set
`PINYIN_CODE_DISABLE_UTF8_OPEN_PATCH=1` before loading the model to
disable that shim.

Export metadata:

- tokenizer_kind: `sentencepiece`
- transliteration: `pinyin-code`
- use_jieba: `true`