Image-Text-to-Text
Transformers
Safetensors
monkeyocrv2
text-generation
conversational
custom_code
Eval Results
Instructions to use zenosai/MonkeyOCRv2-B-Parsing with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use zenosai/MonkeyOCRv2-B-Parsing with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="zenosai/MonkeyOCRv2-B-Parsing", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("zenosai/MonkeyOCRv2-B-Parsing", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use zenosai/MonkeyOCRv2-B-Parsing with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "zenosai/MonkeyOCRv2-B-Parsing" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zenosai/MonkeyOCRv2-B-Parsing", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/zenosai/MonkeyOCRv2-B-Parsing
- SGLang
How to use zenosai/MonkeyOCRv2-B-Parsing with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "zenosai/MonkeyOCRv2-B-Parsing" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zenosai/MonkeyOCRv2-B-Parsing", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "zenosai/MonkeyOCRv2-B-Parsing" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zenosai/MonkeyOCRv2-B-Parsing", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use zenosai/MonkeyOCRv2-B-Parsing with Docker Model Runner:
docker model run hf.co/zenosai/MonkeyOCRv2-B-Parsing
Update README.md
Browse files
README.md
CHANGED
|
@@ -5,7 +5,6 @@ license: apache-2.0
|
|
| 5 |
pipeline_tag: image-text-to-text
|
| 6 |
library_name: transformers
|
| 7 |
---
|
| 8 |
-
|
| 9 |
<div align="center" xmlns="http://www.w3.org/1999/html">
|
| 10 |
<h2>
|
| 11 |
<b>MonkeyOCRv2: A Visual-Text Foundation Model for Document AI</b>
|
|
@@ -18,15 +17,39 @@ library_name: transformers
|
|
| 18 |
[](https://github.com/Yuliang-Liu/MonkeyOCRv2/issues?q=is%3Aissue+is%3Aclosed)
|
| 19 |
[](http://vlrlabmonkey.xyz:8891/)
|
| 20 |
|
| 21 |
-
<img src="https://
|
| 22 |
</div>
|
| 23 |
|
| 24 |
## News
|
|
|
|
| 25 |
* `2026.07.11` 🚀 We release [MonkeyOCRv2](https://github.com/Yuliang-Liu/MonkeyOCRv2), including MonkeyOCRv2 vision encoder, MonkeyOCRv2-Parsing for multilingual document parsing, MonkeyOCRv2-Und for efficient document understanding.
|
| 26 |
|
| 27 |
-
##
|
| 28 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
## Model Zoo
|
| 32 |
|
|
@@ -198,7 +221,6 @@ MonkeyOCRv2 is a text-centric visual foundation model that unifies fine-grained
|
|
| 198 |
</tbody>
|
| 199 |
</table>
|
| 200 |
|
| 201 |
-
|
| 202 |
## Quick Start
|
| 203 |
### Vision Encoder
|
| 204 |
#### 1. Install
|
|
@@ -330,7 +352,7 @@ python infer.py -h
|
|
| 330 |
Our model supports robust document parsing in real-world scenarios across 17 languages, including Simplified Chinese (ZH), Traditional Chinese (ZH-T), English (EN), Arabic (AR), German (DE), Spanish (ES), French (FR), Hindi (HI), Indonesian (ID), Italian (IT), Japanese (JP), Korean (KO), Dutch (NL), Portuguese (PT), Russian (RU), Thai (TH), and Vietnamese (VI).
|
| 331 |
|
| 332 |
<p align="center">
|
| 333 |
-
<img src="
|
| 334 |
</p>
|
| 335 |
|
| 336 |
## Evaluation Results
|
|
@@ -1191,7 +1213,10 @@ Our model supports robust document parsing in real-world scenarios across 17 lan
|
|
| 1191 |
</tbody>
|
| 1192 |
</table>
|
| 1193 |
|
| 1194 |
-
##
|
|
|
|
|
|
|
|
|
|
| 1195 |
|
| 1196 |
We adopt a multi-expert labeling pipeline to obtain reliable annotations for documents. The pipeline includes the following steps:
|
| 1197 |
1. **Structure Detection**
|
|
@@ -1209,6 +1234,26 @@ This multi-expert agreement strategy reduces model-specific annotation errors an
|
|
| 1209 |
- **PaddleOCR-VL**: https://github.com/PaddlePaddle/PaddleOCR
|
| 1210 |
- **Qwen3-VL**: https://github.com/QwenLM/Qwen3-VL
|
| 1211 |
- **Qwen3**: https://github.com/QwenLM/Qwen3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1212 |
|
| 1213 |
## Copyright
|
| 1214 |
-
We warmly welcome your feedback, suggestions, and contributions, which are essential to the continued development and improvement of our framework.
|
|
|
|
| 5 |
pipeline_tag: image-text-to-text
|
| 6 |
library_name: transformers
|
| 7 |
---
|
|
|
|
| 8 |
<div align="center" xmlns="http://www.w3.org/1999/html">
|
| 9 |
<h2>
|
| 10 |
<b>MonkeyOCRv2: A Visual-Text Foundation Model for Document AI</b>
|
|
|
|
| 17 |
[](https://github.com/Yuliang-Liu/MonkeyOCRv2/issues?q=is%3Aissue+is%3Aclosed)
|
| 18 |
[](http://vlrlabmonkey.xyz:8891/)
|
| 19 |
|
| 20 |
+
<img src="https://github.com/Yuliang-Liu/MonkeyOCRv2/blob/main/asserts/overview.png?raw=true" width="600"/>
|
| 21 |
</div>
|
| 22 |
|
| 23 |
## News
|
| 24 |
+
* `2026.07.21` 📦 We release [MonkeyDoc v2](https://modelscope.cn/datasets/zenosai/MonkeyDocv2), an open multilingual corpus for document-oriented pretraining. We hope it can serve as a shared data foundation for more transparent, reproducible, and fair comparisons in Document AI.
|
| 25 |
* `2026.07.11` 🚀 We release [MonkeyOCRv2](https://github.com/Yuliang-Liu/MonkeyOCRv2), including MonkeyOCRv2 vision encoder, MonkeyOCRv2-Parsing for multilingual document parsing, MonkeyOCRv2-Und for efficient document understanding.
|
| 26 |
|
| 27 |
+
## Use MonkeyOCRv2 as a Vision Backbone
|
| 28 |
+
|
| 29 |
+
MonkeyOCRv2 is released as a standalone, document-native vision encoder. It can be integrated into different OCR and document AI systems as a visual backbone.
|
| 30 |
+
|
| 31 |
+
The current release has been evaluated on document parsing, document understanding, text recognition, formula recognition, text detection, document tampering detection, and overlapping-text segmentation.
|
| 32 |
+
|
| 33 |
+
Beyond these evaluated tasks, the encoder may also be useful for text-rich scenarios such as scientific papers, historical documents, medical reports, charts and tables, and remote-sensing maps or reports. We welcome community exploration of these directions.
|
| 34 |
|
| 35 |
+
```python
|
| 36 |
+
from transformers import AutoModel
|
| 37 |
+
|
| 38 |
+
encoder = AutoModel.from_pretrained(
|
| 39 |
+
"zenosai/MonkeyOCRv2-B",
|
| 40 |
+
trust_remote_code=True,
|
| 41 |
+
dtype="auto",
|
| 42 |
+
device_map="auto",
|
| 43 |
+
)
|
| 44 |
+
```
|
| 45 |
+
See the Vision Encoder [Quick Start](https://github.com/Yuliang-Liu/MonkeyOCRv2#vision-encoder) for installation and feature-extraction examples. If you adapt MonkeyOCRv2 to a new task or domain, feel free to open an issue or pull request and share the results.
|
| 46 |
+
|
| 47 |
+
## MonkeyDoc v2
|
| 48 |
+
MonkeyDoc v2 is currently the largest document image pre-training image-text pair dataset, comprising 113 million document images across 17 languages. The open-sourcing of MonkeyDoc v2 is still underway. So far, we have released all 52 million synthetic samples and 35 million real-world samples derived from FinePDF. Additional real-world data from other sources is currently being organized and will be released progressively. You can download the full datset as follows:
|
| 49 |
+
```bash
|
| 50 |
+
pip install modelscope
|
| 51 |
+
modelscope download --dataset zenosai/MonkeyDocv2 --local_dir ./MonkeyDocv2
|
| 52 |
+
```
|
| 53 |
|
| 54 |
## Model Zoo
|
| 55 |
|
|
|
|
| 221 |
</tbody>
|
| 222 |
</table>
|
| 223 |
|
|
|
|
| 224 |
## Quick Start
|
| 225 |
### Vision Encoder
|
| 226 |
#### 1. Install
|
|
|
|
| 352 |
Our model supports robust document parsing in real-world scenarios across 17 languages, including Simplified Chinese (ZH), Traditional Chinese (ZH-T), English (EN), Arabic (AR), German (DE), Spanish (ES), French (FR), Hindi (HI), Indonesian (ID), Italian (IT), Japanese (JP), Korean (KO), Dutch (NL), Portuguese (PT), Russian (RU), Thai (TH), and Vietnamese (VI).
|
| 353 |
|
| 354 |
<p align="center">
|
| 355 |
+
<img src="asserts/Visualization.gif?raw=true" width="600"/>
|
| 356 |
</p>
|
| 357 |
|
| 358 |
## Evaluation Results
|
|
|
|
| 1213 |
</tbody>
|
| 1214 |
</table>
|
| 1215 |
|
| 1216 |
+
## MonkeyDoc v2
|
| 1217 |
+
MonkeyDoc v2 is currently the largest document image pre-training image-text pair dataset, comprising 113 million document images across 17 languages. The open-sourcing of MonkeyDoc v2 is still underway. So far, we have released all 52 million synthetic samples and 41 million real-world samples derived from FinePDF. Additional real-world data from other sources is currently being organized and will be released progressively.
|
| 1218 |
+
|
| 1219 |
+
### Expert Model Labeling Toolchain
|
| 1220 |
|
| 1221 |
We adopt a multi-expert labeling pipeline to obtain reliable annotations for documents. The pipeline includes the following steps:
|
| 1222 |
1. **Structure Detection**
|
|
|
|
| 1234 |
- **PaddleOCR-VL**: https://github.com/PaddlePaddle/PaddleOCR
|
| 1235 |
- **Qwen3-VL**: https://github.com/QwenLM/Qwen3-VL
|
| 1236 |
- **Qwen3**: https://github.com/QwenLM/Qwen3
|
| 1237 |
+
- **MonkeyOCR**: https://github.com/Yuliang-Liu/MonkeyOCR
|
| 1238 |
+
- **MDPBench**: https://github.com/Yuliang-Liu/MultimodalOCR
|
| 1239 |
+
- **MonkeyDoc**: https://modelscope.cn/datasets/zenosai/MonkeyDoc
|
| 1240 |
+
- **FinePDF**: https://huggingface.co/datasets/HuggingFaceFW/finepdfs
|
| 1241 |
+
|
| 1242 |
+
## Citation
|
| 1243 |
+
If you use any part of this release — the MonkeyOCRv2 encoders, MonkeyOCRv2-Parsing,
|
| 1244 |
+
MonkeyOCRv2-Und, the MDPBench benchmark, or the MonkeyDoc v2 dataset — please cite:
|
| 1245 |
+
|
| 1246 |
+
```bibtex
|
| 1247 |
+
@article{liu2026monkeyocrv2,
|
| 1248 |
+
title = {MonkeyOCRv2: A Visual-Text Foundation Model for Document AI},
|
| 1249 |
+
author = {Liu, Yuliang and Li, Zhang and Zhang, Ziyang and Zhang, Shuo and
|
| 1250 |
+
Liu, Qiang and Song, Jiajun and Guo, Zidun and Wang, Xinhan and
|
| 1251 |
+
Zheng, Handong and Liu, Yang and Luo, Dongliang and Ma, Zhiyin and
|
| 1252 |
+
Zhang, Jiarui and Bai, Xiang},
|
| 1253 |
+
journal = {arXiv preprint arXiv:2607.11562},
|
| 1254 |
+
year = {2026}
|
| 1255 |
+
}
|
| 1256 |
+
```
|
| 1257 |
|
| 1258 |
## Copyright
|
| 1259 |
+
We warmly welcome your feedback, suggestions, and contributions, which are essential to the continued development and improvement of our framework. The models are are released under the [Apache License 2.0](LICENSE) and are free for both research and commercial use. For any questions, please contact us at xbai@hust.edu.cn or ylliu@hust.edu.cn.
|