File size: 3,712 Bytes
a32f3de | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | ---
license: mit
library_name: coreai
pipeline_tag: image-to-text
tags:
- core-ai
- on-device
- apple
- ocr
- document-understanding
- glm
base_model: zai-org/GLM-OCR
---
> **Mirror** of [`mlboydaisuke/GLM-OCR-CoreAI`](https://huggingface.co/mlboydaisuke/GLM-OCR-CoreAI) — the canonical repo ([CoreAI Model Zoo](https://github.com/john-rocky/coreai-model-zoo)). Updates land there first.
# GLM-OCR → Core AI
On-device document OCR, running entirely on Apple's **Core AI** (Neural Engine / GPU).
A port of [`zai-org/GLM-OCR`](https://huggingface.co/zai-org/GLM-OCR) (0.9B, **MIT**) — a small,
SOTA-quality document recognizer (OmniDocBench v1.5 **94.62**, #1 with its layout pipeline).
Prompt it with `Text Recognition:` / `Table Recognition:` / `Formula Recognition:` and get back
plain text (reading order), HTML tables (`<table>…`), or LaTeX. zh / en / fr / es / ru / de / **ja** / ko.
GLM-OCR is a small OCR variant of **GLM-4.V** (`Glm4v`): a CogViT vision tower + a 16-layer GLM text
decoder with sectioned 3D M-RoPE. This port reuses the shipped Qwen3-VL vision idiom and GLM text
decode — no R-SWA, no MoE, no MLA.
## Bundles
| dir | what | precision | size |
|---|---|---|---|
| `vision/` | CogViT encoder → `image_embeds [N, 1536]` | fp16 | 829 MB |
| `decoder/` | GLM text decoder, S=1 pipelined, M-RoPE + image injection | int8hu (body int8 per-block-32 + untied head absmax) | 764 MB |
| `tokenizer/` | `tokenizer.json` etc. | — | — |
The decoder rides three static graph inputs — `image_embeds [682,1536]` f16, `rope_shift_start [1]`,
`rope_shift_amount [1]` — so the vision tower runs once, its output is injected at the image
placeholder positions (`V + slot`, row-major over the merged grid), and the text decodes on top.
`N` (visual-token count) is fixed at export by the chosen input resolution (here 682 = a 22×31 merged
grid); resize the page to that grid host-side.
## Verified (M4 Max, GPU, Core AI pipelined engine)
- **End-to-end real generation on the engine: 40/40 tokens identical to the fp32 HF reference** — a
synthetic document read verbatim (*"Quarterly Report / On-device inference shipped across all
product lines this quarter…"*), **~375 tok/s** decode.
- Torch ladder vs HF: decoder logits cos **1.000020**, vision `image_embeds` cos **1.000061**,
full-VLM argmax **694/694**.
- Engine gate: vision `image_embeds` cos **0.9998**; decoder argmax exact over the sampled positions.
- int8hu vs fp16: **7 / 694** argmax flips, all at visual-token positions (0 in the text region), the
generation-driving position exact — i.e. the OCR text is preserved.
## Run it
The decoder is a standard Core AI pipelined LLM bundle with three multimodal static inputs. Drive it
with the pipelined engine (S=1, `COREAI_CHUNK_THRESHOLD=1`; feed the prompt with the image
placeholders rewritten to `V+slot`, bind `image_embeds` from the vision tower, set
`rope_shift_start = img_start + N`, `rope_shift_amount = N − max(gh, gw)`). The full conversion recipe
and the host contract (with the exact static-input values) are in the
[Core AI model zoo](https://github.com/john-rocky/coreai-model-zoo) —
`conversion/export_glm_ocr_pipelined.py`, `zoo/glm-ocr.md`, `knowledge/glm-ocr-port.md`.
## Scope / honesty
- This is the **recognition** model: per-prompt text / table / formula. Whole-page auto-structuring
(the 94.62 full-pipeline number) additionally needs a layout detector (PP-DocLayoutV3) that is not
part of this port.
- int4 is not shipped (weight-only int4 without QAT risks a quality cliff on a 0.9B model).
## License
**MIT** (inherited from `zai-org/GLM-OCR`). *Community port — not affiliated with Apple or Z.ai.*
|