--- library_name: transformers license: openrail license_link: https://huggingface.co/datalab-to/chandra-ocr-2/blob/main/LICENSE base_model: datalab-to/chandra-ocr-2 base_model_relation: quantized pipeline_tag: image-text-to-text tags: - ocr - pdf - markdown - layout - vllm - compressed-tensors - llm-compressor - quantization - fp8 - fp8-dynamic - hopper - ada - blackwell --- # chandra-ocr-2 — FP8 Dynamic FP8 dynamic-activation quantization of [`datalab-to/chandra-ocr-2`](https://huggingface.co/datalab-to/chandra-ocr-2) produced with [`llm-compressor`](https://github.com/vllm-project/llm-compressor) and packed as [compressed-tensors](https://github.com/neuralmagic/compressed-tensors) for native vLLM inference. > **The "works almost everywhere modern" quant.** FP8 runs natively on > Ada (RTX 4090 / L40S), Hopper (H100), and Blackwell. ~5434 ms/page > sequential = 0.18 pages/s, **2.3× over bf16**. Pick this when you > don't have a Blackwell GPU, or when the runner issues one batched > request per document. For the original model description, intended uses, accuracy benchmarks (olmOCR-bench, 90-language) and license terms, see the upstream card: . --- ## Quantization recipe ```yaml # recipe.yaml (shipped in this repo) default_stage: default_modifiers: QuantizationModifier: targets: [Linear] ignore: - 're:.*lm_head' - 're:visual.*' # keep ViT vision tower bf16 - 're:model.visual.*' - 're:.*mlp.gate$' - 're:.*embed_tokens$' - 're:.*shared_expert_gate$' - 're:.*mlp\.shared_expert$' - 're:.*linear_attn.*' scheme: FP8_DYNAMIC ``` - Weights: **FP8 E4M3** (per-channel static scales) - Activations: **FP8 dynamic** (per-token scales computed at runtime, no calibration needed) - Vision tower, `lm_head`, MoE gates and `linear_attn.*` kept in bf16. Because activations are dynamic, this quant requires **no calibration dataset** — accuracy ≈ upstream bf16 within OCR task noise. ## Hardware requirements | GPU family | Compute capability | FP8 tensor cores | Recommended? | |---|:---:|:---:|:---:| | Blackwell (RTX PRO 6000, B100/B200, RTX 5090) | sm_100+ | ✅ Native | ✅ | | Hopper (H100/H200) | sm_90 | ✅ Native | ✅ | | Ada (RTX 4090, L40S) | sm_89 | ✅ Native | ✅ | | Ampere (A100/3090) | sm_80/86 | Software fallback (bf16 compute) | ⚠️ no speedup | | Turing & older | ≤ sm_75 | ❌ | ❌ | vLLM ≥ **0.17** (works with the current OpenAI image). On Ada this is **the only Chandra-2 quant that actually accelerates inference** — NVFP4 variants have no FP4 tensor cores on Ada/Hopper. ## Benchmark (vs. other Chandra-2 quants) Test bed: RTX PRO 6000 Blackwell Max-Q (96 GB), 14-page Vietnamese financial-statement PDF, vLLM 0.19.1, `max-num-seqs=128`, `max-num-batched-tokens=32768`, kv-cache=fp8. | Build | Sequential per-doc | Concurrent per-page | Best ms/page | vs bf16 | |-----------------|-------------------:|--------------------:|-------------:|--------:| | bf16 baseline | 12724 ms | 12642 ms | 12642 | 1.0× | | **FP8_DYNAMIC** | **5434 ms** | 9525 ms | **5434** | **2.3×** | | NVFP4A16 | 12280 ms | 5058 ms | 5058 | 2.5× | | NVFP4 (W4A4) | 10092 ms | 5794 ms | 5794 | 2.2× | Take-away: FP8_DYNAMIC is fastest under **sequential per-document** batching (one big request, KV cache fully utilised). For page-level concurrent fan-out on Blackwell, switch to NVFP4A16. ## Usage ### vLLM (OpenAI-compatible server) — recommended ```bash vllm serve dangvansam/chandra-ocr-2-FP8-dynamic \ --served-model-name chandra \ --max-model-len 16384 \ --max-num-seqs 64 \ --max-num-batched-tokens 16384 \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.90 \ --enable-prefix-caching \ --enable-chunked-prefill \ --trust-remote-code \ --mm-processor-kwargs '{"min_pixels": 3136, "max_pixels": 6291456}' ``` ```python from openai import OpenAI import base64, pathlib client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") img_b64 = base64.b64encode(pathlib.Path("page.png").read_bytes()).decode() resp = client.chat.completions.create( model="chandra", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}}, {"type": "text", "text": ""}, ], }], max_tokens=12000, temperature=0.0, ) print(resp.choices[0].message.content) ``` ### HuggingFace Transformers The vision tower stays in bf16, so the upstream snippet works unchanged — just swap the repo id to `dangvansam/chandra-ocr-2-FP8-dynamic`. See the [upstream card](https://huggingface.co/datalab-to/chandra-ocr-2#with-huggingface-transformers). ## When to pick which Chandra-2 quant | Workload | Pick | |---|---| | Ada (RTX 4090, L40S) or Hopper (H100) GPU | **FP8_DYNAMIC (this repo)** | | Single sequential request per doc on any modern GPU | **FP8_DYNAMIC (this repo)** | | Page-concurrent fan-out on Blackwell | NVFP4A16 | | Max compression, accuracy not critical | NVFP4 (W4A4) | | Reference accuracy / older hardware | upstream bf16 | ## Files - `model.safetensors` — FP8-packed weights (~13 GB) - `config.json`, `processor_config.json`, `preprocessor_config.json`, `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`, `generation_config.json` — copied from upstream - `recipe.yaml` — exact llm-compressor recipe used ## License & attribution Inherits the upstream **OpenRAIL-M** license from `datalab-to/chandra-ocr-2`. Free for research, personal use, and startups <$2M; **not** for use competing with Datalab's hosted API. For broader commercial use see [Datalab pricing](https://www.datalab.to/pricing). This is an **unofficial community quant**. No additional weights or data were added — only a numerical re-encoding of the upstream model. All credit for the model itself goes to Datalab. ## Citation ```bibtex @misc{chandra_ocr_2, author = {Datalab}, title = {Chandra OCR 2}, year = {2026}, url = {https://huggingface.co/datalab-to/chandra-ocr-2} } ```