hunyuanocr-1.5-poneglyph-bbox

HunyuanOCR-1.5 fine-tuned for One Piece manga bubble text plus bounding boxes

This model reads a full manga page and emits HunyuanOCR spotting JSON:

[{"box":[x1,y1,x2,y2],"text":"Text content"}]

Coordinates are normalized to [0, 1000] on the resized page image.

---

Why HunyuanOCR For BBox

The upstream HunyuanOCR-1.5 card documents a lightweight OCR-specialized VLM, official transformers inference, vLLM AR/DFlash serving, and llama.cpp GGUF deployment. This fine-tune uses the same image-text-to-text family and teaches the generated text stream to match the existing Poneglyph bbox contract.

---

Benchmark: Hunyuan vs LightOn BBox Poneglyph

Metric	HunyuanOCR-1.5 fine-tuned	LightOn bbox Poneglyph	Winner
CER	2.77%	pending	pending
WER	5.26%	pending	pending
Mean IoU	91.75%	pending	pending
Median IoU	93.83%	pending	pending
F1 @ IoU=0.5	98.28%	pending	pending
Precision @ 0.5	98.46%	pending	pending
Recall @ 0.5	98.51%	pending	pending
Detection Rate	98.93%	pending	pending
Combined Score	0.973	pending	pending
Avg Inference	5.91s/page	pending	pending

---

Hunyuan Fine-Tuned Snapshot

Metric	Score
CER	2.77%
WER	5.26%
Mean IoU	91.75%
Median IoU	93.83%
F1 @ IoU=0.3	98.41%
F1 @ IoU=0.5	98.28%
F1 @ IoU=0.75	95.21%
Detection Rate	98.93%
Combined Score	0.973
Avg Inference	5.91s/page

Combined score:

0.35 * TextScore + 0.35 * F1@0.5 + 0.15 * Recall@0.5 + 0.10 * MeanIoU + 0.05 * CountAccuracy

---

Dataset

Source data comes from the Poneglyph Supabase bulles table, filtered to validated annotations, grouped at page level, and split by id_page to prevent page leakage.

Split	Pages	Bubbles
train	683	6147
val	147	1354
test	147	1390

Preprocessing:

• Full page image resized to 1540px longest side.
• JPEG quality 95.
• Bubble boxes normalized to [0, 1000].
• Target order follows the stored manga reading order.
• Target text uses HunyuanOCR-1.5's native spotting JSON schema.

---

How To Use

pip install torch pillow "transformers>=5.13.0" accelerate

import json
import torch
from PIL import Image
from transformers import AutoProcessor

MODEL_ID = "Remidesbois/hunyuanocr-1.5-poneglyph-bbox"
PROMPT = '检测并识别图中所有的文字行，请按日式漫画从右到左、从上到下的阅读顺序进行识别。输出格式为 JSON 数组，每个元素必须包含："box": [xmin, ymin, xmax, ymax]（坐标需归一化到 [0, 1000] 范围内）；"text": "识别出的文字内容"。注意：请直接输出 JSON 数组，不要包含任何多余的描述性文字。'

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True, use_fast=False)
try:
    from transformers import HunYuanVLForConditionalGeneration as ModelClass
except Exception:
    from transformers import AutoModelForImageTextToText as ModelClass

model = ModelClass.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="eager",
).eval()

image = Image.open("page.jpg").convert("RGB")
image.thumbnail((1540, 1540), Image.Resampling.LANCZOS)

messages = [
    {"role": "system", "content": ""},
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "page.jpg"},
            {"type": "text", "text": PROMPT},
        ],
    }
]

prompt = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)
inputs = processor(text=[prompt], images=[image], return_tensors="pt")
inputs = {
    k: v.to(model.device, dtype=torch.bfloat16) if v.is_floating_point() else v.to(model.device)
    for k, v in inputs.items()
}

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=False,
        repetition_penalty=1.0,
    )

generated = output_ids[0, inputs["input_ids"].shape[1]:]
text = processor.decode(
    generated,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
).strip()
print(text)

bubbles = [
    {"text": item["text"], "bbox": item["box"]}
    for item in json.loads(text)
]

GGUF and local desktop runtime

The repository also ships:

• gguf/hyocr-bbox-f16.gguf: language/decoder target.
• gguf/mmproj-hyocr-bbox-f16.gguf: vision projector.
• gguf/hyocr-bbox-dflash-bf16.gguf: DFlash multi-token draft.
• runtime/: official Windows CUDA llama.cpp binaries for stable AR inference.

Poneglyph Desktop downloads these files with the model, starts llama-server on demand, and uses its OpenAI-compatible endpoint. DFlash is enabled when the Hunyuan-specific DFlash fork executable is available; if that experimental server fails to start, the app automatically restarts the stable AR runtime.

The upstream DFlash draft was not retrained with this domain adapter. Its decoding remains lossless because target verification decides accepted tokens, but the acceptance rate and speedup must be measured separately from OCR quality. vLLM DFlash additionally requires its CUDA 13 nightly environment.

---

Training

The training package used for this model lives in:

docker_scripts/finetune_hunyuan_ocr_bbox

Pipeline:

python run_pipeline.py --dry-run --check-remote
python run_pipeline.py

The run exports the dataset, fine-tunes HunyuanOCR-1.5 with LoRA/DoRA, benchmarks the held-out test split, prepares DFlash/MTP artifacts, converts base/mmproj GGUF files for llama.cpp, benchmarks Remidesbois/LightonOCR-2-1b-poneglyph-bbox on the same pages, writes this README, and uploads the final merged model when HF_TOKEN is available.

---

Limitations

• Domain-specific: trained for One Piece manga pages.
• Text language: French annotations.
• Output is a generated text contract, so malformed lines are possible and should be parsed defensively.
• The model returns normalized bbox coordinates, not pixel coordinates.
• The LightOn comparison is only valid when both models are evaluated on the same exported test split.

---

Base Model

Fine-tuned from tencent/HunyuanOCR. The base model uses the official HunyuanOCR-1.5 image-text-to-text architecture.

---

Fine-tuned by Remidesbois.