Instructions to use SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF",
	filename="PaddleOCR-VL-1.6-Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M

Use Docker

docker model run hf.co/SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M

Ollama
How to use SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF with Ollama:
```
ollama run hf.co/SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M
```

Unsloth Studio

How to use SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF with Docker Model Runner:
```
docker model run hf.co/SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M
```

Lemonade

How to use SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.PaddleOCR-VL-1.6-GGUF-Q4_K_M

List all available models

lemonade list

PaddleOCR-VL-1.6 — GGUF quants for llama.cpp

GGUF quantizations of PaddlePaddle/PaddleOCR-VL (the 1.6 release) — a ~0.9 B vision-language OCR model (NaViT encoder + ERNIE-4.5-0.3B) — for running document OCR locally with llama.cpp.

These are community quants. The original model and weights are © PaddlePaddle, released under Apache-2.0; this repo redistributes derivative (quantized) copies under the same license, with attribution. No model architecture or behavior was changed — only weight precision.

Quick start

👉 Use PaddleOCR-VL-1.6-Q4_K_M.gguf (the recommended file) + the required mmproj.

# 1. download the recommended quant + the vision projector (required)
huggingface-cli download SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF \
  PaddleOCR-VL-1.6-Q4_K_M.gguf PaddleOCR-VL-1.6-mmproj.gguf --local-dir .

# 2. serve it (8 GB GPU → ~1.6 GB VRAM, 8-way parallel)
llama-server -m PaddleOCR-VL-1.6-Q4_K_M.gguf \
  --mmproj PaddleOCR-VL-1.6-mmproj.gguf -c 8192 -np 8 -ngl 99 --port 8080

Then POST a page image with the text prompt OCR: (OpenAI vision format) to http://localhost:8080/v1/chat/completions. Prefer higher-precision quants? Use Q5_K_M / Q6_K / Q8_0 instead — same mmproj. See Validation and the ⚠️ repetition-loop note before production use.

Files

File	Quant	Size	Notes
`PaddleOCR-VL-1.6-Q4_K_M.gguf`	Q4_K_M	286 MB	recommended — fastest; reproduces F16's values on well-formed pages (see validation)
`PaddleOCR-VL-1.6-Q5_K_M.gguf`	Q5_K_M	326 MB	slightly higher precision
`PaddleOCR-VL-1.6-Q6_K.gguf`	Q6_K	367 MB	near-F16
`PaddleOCR-VL-1.6-Q8_0.gguf`	Q8_0	475 MB	highest-precision quant
`PaddleOCR-VL-1.6-mmproj.gguf`	F16	841 MB	vision projector (required for all of the above)

The F16 reference is not re-hosted here (it's identical to upstream) — get it from the official GGUF repo. It's used below only as the comparison baseline.

Validation & findings

Why not perplexity / KL-divergence?

The usual quant metric (KL-divergence or perplexity vs F16 on a text corpus) does not apply to this model. PaddleOCR-VL is a vision OCR model — given plain text without an image, it's out-of-distribution: measured F16 perplexity on academic text was ~160 (a real text-LM is ~5–30), and in that near-random regime the comparison is meaningless (Q8_0 even scored a lower perplexity than F16). So we validated on the actual OCR task instead.

Method (apple-to-apple)

All five models (F16 + four quants) OCR'd the same four pages (title/authors, dense body text, equation+table, references) with identical, deterministic settings (-np 1, temperature 0). At -np 1 the model is fully deterministic — F16-vs-F16 = 100% identical — so any quant divergence is real, not run-to-run noise.

Results (vs F16)

Quant	Size	bpw	Table values correct	Text similarity to F16
F16	892 MB	16.0	reference	—
Q8_0	475 MB	8.5	✅ all	73%
Q6_K	367 MB	6.6	✅ all	68%
Q5_K_M	326 MB	5.8	✅ all	70%
Q4_K_M	286 MB	5.1	✅ all	66%

Interpretation (the important part)

Every quant extracts the same values as F16. Each win-rate / conflict-rate / fraction on the table page (51.3%, 46.3%, 37 / 80, 82.5%, …) is reproduced correctly by all of Q4–Q8 — verified directly, not assumed.
The ~30% text divergence is cosmetic, not error. It's reading-order, whitespace, and where each transcription starts/ends — and it's non-monotonic in precision (Q4 ≈ Q8). If it were graded quantization loss, Q8 would track F16 most closely; it doesn't. That signature means the divergence is generation-path sensitivity, not accuracy loss.
Conclusion: no quant in Q4–Q8 shows OCR value-level degradation vs F16. Q4_K_M is the recommended default (smallest, fastest, values intact). Q5–Q8 are offered for headroom but showed no measurable accuracy advantage on this task.

⚠️ Known failure mode — repetition loops. On one of the four test pages (a highly repetitive template layout), the bare model degenerated into a runaway loop — repeating a line hundreds of times. Critically, this happened to F16 and several quants alike (F16 was the worst; Q8 didn't loop; Q4 did) — there is no precision ordering, so it is a model/inference limitation, not a quantization defect, and it occurred even with repeat_penalty 1.15. See Limitations for mitigations. The "values correct" finding above applies to well-behaved pages; repetition-prone layouts can break for any precision including F16.

Honest limitations

Sanity-grade, not a benchmark: 4 pages, one document, English, accuracy checked against F16 + targeted value spot-checks, not against human ground-truth labels (e.g. OmniDocBench). It establishes "Q4 ≈ F16 on values for well-formed pages" — it is not a multilingual or large-corpus accuracy claim, and (see warning above) it does not mean every page transcribes cleanly. For rigorous accuracy numbers, see the upstream model's OmniDocBench results.

Performance (21-page paper, RTX A4000 8 GB, llama.cpp, 8-way parallel)

Config	Time	VRAM
F16, sequential	118 s	~1.9 GB
F16, 8-way parallel	57 s	~2.1 GB
Q4_K_M, 8-way parallel	34 s	~1.6 GB

Q4 is ~1.7× faster than F16 and lighter, because decode (the bottleneck, ~89% of runtime) reads the weights for every generated token — 4-bit weights = 1/3 the memory traffic.

Usage (llama.cpp)

llama-server -m PaddleOCR-VL-1.6-Q4_K_M.gguf \
  --mmproj PaddleOCR-VL-1.6-mmproj.gguf \
  -c 8192 -np 8 -ngl 99 --port 8080

Then send a page image with the prompt OCR: (OpenAI vision format). On an 8 GB GPU this runs at ~1.6 GB VRAM and 8-way parallel.

Tips

This is a specialized OCR model, not a general chat VLM. Use the trained prompt OCR:; custom instructions ("convert to markdown…") make it worse.
Repetition loops are a real failure mode (affects full-precision F16 too — not a quantization issue): on highly repetitive layouts the model can run away repeating a line. "repeat_penalty": 1.15 is not always enough. Stronger mitigations: raise repeat_penalty (≈1.3) and/or enable DRY sampling (--dry-multiplier 0.8), cap max_tokens, and detect runaway output and retry/flag the page. For repetition-heavy documents, prefer the official PaddleOCR-VL pipeline (its layout stage avoids feeding whole repetitive pages to the VLM) or a model with a built-in no-repeat-ngram processor.
Output is flat text + LaTeX (the bare VLM). The structured layout (HTML tables, headings, figure crops) comes from PaddleOCR's separate layout stage, which is not in the GGUF — for fully-structured Markdown use the official PaddleOCR-VL pipeline.

Quantized with

llama-quantize (llama.cpp), static k-quants from the F16 GGUF. Imatrix was intentionally not used: at Q4–Q8 the values are preserved on the OCR task (see validation), and sub-4-bit IQ quants aren't useful for a 286 MB model whose constraint is decode speed, not size.

Test conditions

All quantization, validation, and benchmarks were run on a single machine. Numbers (speed/VRAM) are specific to this setup and will differ on yours.


GPU	NVIDIA RTX A4000 Laptop, 8 GB, compute 8.6 (Ampere), driver 573.91
CPU / RAM / OS	Intel i7-11850H · 64 GB · Windows 11 (build 26200)
Runtime	llama.cpp (turboquant fork, build `4595fff`, MSVC 19.44) — GGUFs also run on any mainline llama.cpp with PaddleOCR-VL support
Base model	PaddlePaddle/PaddleOCR-VL 1.6, F16 GGUF + F16 mmproj
Quantizer	`llama-quantize` from the same build, F16 → k-quant
Validation	4 pages of an English arXiv paper (title, body, equation+table, references); `-np 1`, `temp 0`, `-c 8192`, `-ngl 99`, prompt `OCR:`, `repeat_penalty 1.15`; deterministic F16 reference
Perf benchmark	21-page paper; `-np 8`, `-c 32768`, page render 200 DPI, 8 concurrent requests
Date	2026-06-25

License & attribution

Apache-2.0, inherited from the base model PaddlePaddle/PaddleOCR-VL. See that repo for the original LICENSE/NOTICE. "PaddleOCR" is a trademark of its owners; this is an unofficial community quantization and is not endorsed by PaddlePaddle.

Disclaimer

These are unofficial, community-produced quantizations, provided "as is" without warranty of any kind, express or implied, including fitness for a particular purpose. Use at your own risk.

Not affiliated with or endorsed by PaddlePaddle/Baidu. For the authoritative model, weights, and benchmarks, use the official repository.
Quantization is lossy. The validation here is sanity-grade (4 pages, one English document, checked against F16 + targeted value spot-checks — not against human ground-truth labels or a standard benchmark like OmniDocBench). It does not guarantee accuracy on your documents, languages, or layouts.
Do not rely on OCR output for safety-, legal-, medical-, or financial-critical decisions without independent human verification.
Benchmark numbers (speed, VRAM, timings) are specific to the hardware/software in Test conditions and will vary on other systems.
You are responsible for complying with the Apache-2.0 license terms and any applicable laws when using or redistributing these files.

Downloads last month: 513

GGUF

Model size

0.5B params

Architecture

paddleocr

Hardware compatibility

4-bit

5-bit

6-bit

8-bit

Model tree for SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF

Base model

baidu/ERNIE-4.5-0.3B-Paddle

Finetuned

PaddlePaddle/PaddleOCR-VL

Quantized

(6)

this model