Refactor application implementation

.agents/references/nuextract3/2026-07-20/README.md

@@ -0,0 +1,5 @@
+# nuextract3
+
+- Topic: `nuextract3`
+- Capture Date: `2026-07-20`
+- Scope: TODO

.agents/references/nuextract3/2026-07-20/REFERENCE_INDEX.md

@@ -0,0 +1,22 @@
+# Reference Index
+
+- Bundle Path: `S:\Spaces\Data-Extraction\NuMarkApp\.agents\references\nuextract3\2026-07-20`
+- Mirrored References: `1`
+- Fixtures: `1`
+
+- `README.md`: bundle scope
+- `SOURCE_URLS.md`: verified sources
+- `REFERENCE_SUMMARY.md`: findings and version notes
+- `fixtures/`: user-provided artifacts
+- `<domain>/`: mirrored markdown references by domain
+
+## References
+
+### huggingface.co
+
+- `huggingface.co/numind/NuExtract3.md`: `https://huggingface.co/numind/NuExtract3`
+
+## Fixtures
+
+- `fixtures/user-drop-001.txt`
+

.agents/references/nuextract3/2026-07-20/REFERENCE_SUMMARY.md

@@ -0,0 +1,20 @@
+# NuExtract3 Reference Summary
+
+- Reference date: 2026-07-20
+- Official model: `numind/NuExtract3`
+- Library: Transformers
+- Base model: Qwen/Qwen3.5-4B
+- License: Apache-2.0
+
+NuExtract3 is a 4B vision-language reasoning model for structured extraction and document-to-Markdown conversion. It accepts text, images, or both.
+
+For direct Transformers inference, the official README loads `AutoProcessor` and `AutoModelForImageTextToText`, then passes `template`, `mode`, and `enable_thinking` through `processor.apply_chat_template`. Non-thinking structured extraction uses `enable_thinking=False`; the documented starting temperature is 0.2.
+
+Runtime-specific note: this project uses Transformers 5.14.1, Python 3.12, BF16, automatic device placement, and FlashAttention 2.
+
+- Reference Date: `2026-07-20`
+- Official Version: TODO
+- Key Findings:
+  - TODO
+- Compatibility Notes:
+  - TODO

.agents/references/nuextract3/2026-07-20/SOURCE_URLS.md

@@ -0,0 +1,10 @@
+# Source URLs
+
+- Bundle: `2026-07-20`
+- Generated: `2026-07-20T17:29:26.022745+00:00`
+- Mirrored References: `1`
+
+## huggingface.co
+
+- `https://huggingface.co/numind/NuExtract3` -> `huggingface.co/numind/NuExtract3.md`
+

.agents/references/nuextract3/2026-07-20/fixtures/user-drop-001.txt

@@ -0,0 +1,942 @@
+---
+license: apache-2.0
+license_link: https://huggingface.co/numind/NuExtract3/blob/main/LICENSE
+library_name: transformers
+pipeline_tag: image-to-text
+tags:
+- image-text-to-text
+- transformers
+- safetensors
+- qwen3_5
+- vision-language
+- vlm
+- document-understanding
+- structured-extraction
+- information-extraction
+- ocr
+- document-to-markdown
+- markdown
+- rag
+- reasoning
+- multilingual
+- conversational
+base_model:
+- Qwen/Qwen3.5-4B
+model_name: NuExtract3
+---
+
+<p align="center">
+    <a href="https://nuextract.ai/">
+        <img src="header.svg" width="900px"/>
+    </a>
+</p>
+
+
+<p align="center">
+        🖥️ <a href="https://nuextract.ai/">API / Platform</a>&nbsp;&nbsp; | &nbsp;&nbsp;
+        📑 <a href="https://numind.ai/blog">Blog</a>&nbsp;&nbsp; | &nbsp;&nbsp;
+        🗣️ <a href="https://discord.gg/3tsEtJNCDe">Discord</a>&nbsp;&nbsp; | &nbsp;&nbsp;
+        🛠️ <a href="https://github.com/numindai/nuextract">GitHub</a>
+</p>
+
+**NuExtract3** is a unified **4B** vision-language reasoning model for document understanding.
+
+It combines strong **structured information extraction** with high-quality **image-to-Markdown** conversion, making it suitable for extraction pipelines, OCR, and RAG preprocessing for all types of documents such as scans, receipts, forms, invoices, contracts or tables.
+
+Try it out in [the 🤗 space!](https://huggingface.co/spaces/numind/NuExtract-3-4B)
+
+## Overview
+
+- **Structured extraction**: input (text/images) + JSON template + instructions --> JSON output
+- **Markdown conversion**: input (text/images) --> Markdown
+- **Multimodal inputs**: text, images, or text + images.
+- **Multilingual** documents.
+- **Reasoning** and non-reasoning inference modes.
+- **Template generation** for structured extraction from natural language or input document.
+
+# Benchmark results
+
+## Structured Extraction
+
+We benchmarked NuExtract on NuMind's internal structured benchmark, measuring model's performances on ~600 documents of diverse types including invoices, movie posters or floor plans. These documents and their ground-truth cover diverse use-cases testing model visual understanding, OCR, reasoning and ability to handle long input and output contexts.
+We plan to open-source this benchmark in the coming weeks, along with a extensive leaderboard including most popular open-weight and closed-sourced APIs and a Python library allowing to easily measure model performances on structured extraction.
+
+<img src="st.svg" width="1000"/>
+
+To measure a pair of predicted and ground-truth JSONs, we represent both as trees which we align based on node names, compute metric scores for aligned leaves and report the average of these scores. `string` and `verbatim-string` leaves are evaluated with indel distance (i.e. Levenshtein without replacement), while all others are evaluated with exact-match.
+Models were evaluated using vllm, with a temperature of 0.25 and a maximum of 65000 output token (for both thinking and answer), which largely exceeds 22000 which is the number of tokens of the largest ground truth output.
+
+<figure>
+
+|Model name          |Average score|Num. failed⁽¹⁾|Avg. num tokens thinking|Avg. num tokens answer|
+|--------------------|-------------|-----------|------------------------|----------------------|
+|NuExtract3.4_4B-RL  |**0.651 ± 0.019**|27     |2036                    |1856                  |
+|gemma-4-E4B-it      |0.538 ± 0.023|31         |3005                    |1287                  |
+|Qwen3.5-9B          |0.479 ± 0.030|170        |22409                   |1257                  |
+|Qwen3.5-4B          |0.417 ± 0.031|229        |27177                   |1201                  |
+|GLM-4.6V-Flash      |0.435 ± 0.026|153        |2989                    |1357                  |
+|Nemotron-3-Nano-Omni|0.387 ± 0.028|204        |25827                   |522                   |
+|Ministral-3-3B      |0.240 ± 0.022|344        |27586                   |362                   |
+
+<figcaption>
+<small>
+(1) number of model outputs that were not JSON deserializable, either directly or by removing leading and trailing backticks.<br>
+95% confidence intervals computed using a nonparametric bootstrap over scores distributions.
+</small>
+</figcaption>
+</figure>
+
+The benchmark include samples containing multiple images resulting in large input context, and some with ground-truth containing large numbers of items to extract resulting in large outputs. We found that the reasoning of small models significantly negatively impact their performances. The reason is that many models ended up falling in repetition loops, hitting the output tokens limit and resulting in failed requests.
+
+## Document to Markdown
+
+NuExtract can also convert document images into clean Markdown. Output will be Markdown for text (headers etc), HTML for tables, LaTeX for math and ```<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/> ```
+
+Modern, format-agnostic benchmarks for complex document understanding are limited, so we explored a new evaluation approach.
+We selected 100 documents with challenging layouts and tables, asked each model to convert them into a structured representation, then used Gemini 3 Flash to compare model outputs against the source document and choose the most accurate result.
+The rankings aligned with human votes, suggesting this is a promising method for evaluating document-to-Markdown capabilities. More details will be shared in an upcoming technical report.
+Here are some results:
+
+<img src="ocr_preferences.svg" width="1000"/>
+
+### Using "Markdown-to-structured"
+
+To add other evaluate references, we used our structured extraction benchmark to evaluate models in a two-step fashion: convert the benchmark inputs to Markdown, then use Qwen3.6 27B to perform the structured extraction task on them. Intuitively, it allows to evaluate how models achieve to keep the input document content and layout: good models will allow the "structured extractor" model to perform better scores.
+
+<img src="md2st.svg" width="1000"/>
+
+
+# Using NuExtract
+
+## Structured extraction
+
+Structured extraction takes as inputs:
+
+1. An input document, which can be text, image, or both;
+2. A JSON template describing the information to extract;
+3. (Optional) Instructions, allowing to specify expected output formats or values, to provide with the `instructions` chat template kwarg;
+4. (Optional) In-Context Learning (ICL) examples.
+
+### Input JSON template
+
+NuExtract uses a input JSON template whose structure is identical to the output JSON. Its leaf values are specify the **types** of the output JSON leaves. For examples:
+
+```json
+{
+  "invoice_number": "verbatim-string",
+  "invoice_date": "date",
+  "total_amount": "number",
+  "currency": "currency",
+  "line_items": [
+    {
+      "description": "verbatim-string",
+      "item_type": ["electronics", "clothing", "vehicle", "furniture", "other"],
+      "quantity": "integer",
+      "unit_price": "number",
+      "total": "number"
+    }
+  ]
+}
+```
+
+Supported template types include:
+
+- `verbatim-string`: extract text exactly as it appears in the document;
+- `string`: generic string field, allowing abstraction or light paraphrasing;
+- `integer`: whole number;
+- `number`: integer or decimal number;
+- `date-time`: ISO-8601 date, time or date-time;
+- Other specific types such as `data`, `time`, `country`, `currency`, `email` and so on.
+[**For more details, read the complete types specifications and examples**](TYPES.md)
+
+Template constructors:
+
+- Arrays, for example `["string"]`;
+- Enums, for example `["yes", "no", "maybe"]`;
+- Multi-enums (multiple possible values), for example `[["A", "B", "C"]]`.
+
+If the model does not find relevant information for a field, it returns `null` or `[]`.
+
+### Converting JSON schema / Pydantic models to NuExtract template
+
+Our Python SDK (`pip install numind`) offers a method to convert JSON schemas to NuExtract templates:
+
+```Python
+from typing import Literal
+
+from pydantic import Field, BaseModel
+from numind.nuextract_utils import convert_json_schema_to_nuextract_template
+
+
+class HotelBooking(BaseModel):
+    city: str
+    check_in_date: str = Field(description="date")
+    check_out_date: str = Field(description="date")
+    number_of_guests: int
+    room_type: Literal["single", "double", "suite"]
+
+
+template, dropped_branches = convert_json_schema_to_nuextract_template(
+    HotelBooking.model_json_schema()
+)
+
+# {'check_in_date': 'date', 'check_out_date': 'date', 'city': 'string', 'number_of_guests': 'integer', 'room_type': ['single', 'double', 'suite']}
+```
+
+## Document-to-Markdown
+
+NuExtract can also convert document images into clean Markdown. Output will be markdown for text (headers etc), html for tables, latex for mat and ```<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/> ```
+
+Markdown example:
+
+```markdown
+<figure data-type="image" data-id="img_1">
+  <img src="img_1.png" alt="Logo of Mobilier 2000 with contact information: Tél.: (418) 275-4232, 1654, boul. Marcotte, Roberval (Qc) G8H 2P2"/>
+</figure>
+
+# COMMANDE
+**NUMÉRO 72259**
+
+1
+
+**Vendu à**
+TREMBLAY ERIC
+ERIC TREMBLAY
+348 BOUL. DE L'ANSE
+ROBERVAL
+G8H 1Y9
+
+**Livré à**
+TREMBLAY ERIC
+ERIC TREMBLAY
+348 BOUL. DE L'ANSE
+ROBERVAL
+G8H 1Y9
+
+<table>
+  <thead>
+    <tr>
+      <th># CLIENT</th>
+      <th>EXPÉDITEUR</th>
+      <th>TERME DE CRÉDIT</th>
+      <th>DATE</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>2753133</td>
+      <td>Notre camion</td>
+      <td>à la livraison</td>
+      <td>22/06/2023</td>
+    </tr>
+  </tbody>
+</table>
+
+<table>
+  <thead>
+    <tr>
+      <th>NOM DU VENDEUR</th>
+      <th>VOTRE ÉCONOMIE !</th>
+      <th># COMMANDE</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>Éric</td>
+      <td>0.00</td>
+      <td></td>
+    </tr>
+  </tbody>
+</table>
+```
+
+---
+
+## Reasoning and non-reasoning modes
+
+NuExtract supports both reasoning and non-reasoning inference.
+
+### Non-thinking mode
+
+Use this for fast and deterministic extraction or Markdown conversion.
+
+```python
+enable_thinking = False
+temperature = 0.2
+```
+
+### Thinking mode
+
+Use this for difficult documents, complex layouts, ambiguous fields, or cases where the document structure requires additional reasoning.
+
+```python
+enable_thinking = True
+temperature = 0.6
+```
+
+For production extraction workloads, we recommend starting with **non-reasoning mode** and enabling reasoning only for difficult examples.
+
+
+---
+
+## vLLM deployment
+
+NuExtract can be served with vLLM using an OpenAI-compatible API.
+
+```bash
+vllm serve numind/NuExtract3 \
+  --trust-remote-code \
+  --limit-mm-per-prompt '{"image": 99, "video": 0}' \
+  --chat-template-content-format openai \
+  --generation-config vllm \
+  --max-model-len 131072 \
+  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+
+
+### Multi Token Prediction
+<details>
+The deployment commands above enable Multi Token Prediction (MTP) through vLLM speculative decoding:
+
+```bash
+--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+
+MTP can improve decoding throughput without changing the OpenAI-compatible request payload. You can tune `num_speculative_tokens` for your hardware and workload, or remove `--speculative-config` if your vLLM version or environment does not support this speculative decoding method.
+
+If you encounter memory issues, reduce the maximum model length and the maximum number of images:
+
+```bash
+vllm serve numind/NuExtract-3 \
+  --trust-remote-code \
+  --limit-mm-per-prompt '{"image": 6, "video": 0}' \
+  --chat-template-content-format openai \
+  --generation-config vllm \
+  --max-model-len 16384 \
+  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+</details>
+
+## vLLM inference: structured extraction: text
+```python
+import json
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "currency": ["USD", "EUR", "GBP", "JPY", "Other"],
+    "items": [
+        {
+            "name": "verbatim-string",
+            "price": "number"
+        }
+    ]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "Yesterday I bought apples and coffee at Trader Joe's for a total of $12.40."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template),
+            "instructions": "Specify the time for the `date` entry only if it is present, otherwise only output the date component.",
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```json
+{
+  "store": "Trader Joe's",
+  "date": null,
+  "total": 12.40,
+  "currency": "USD",
+  "items": [
+    {
+      "name": "apples",
+      "price": null
+    },
+    {
+      "name": "coffee",
+      "price": null
+    }
+  ]
+}
+```
+
+---
+
+## vLLM inference: structured extraction: image
+
+```python
+import json
+import base64
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def encode_image(image_path):
+    with open(image_path, "rb") as image_file:
+        return base64.b64encode(image_file.read()).decode("utf-8")
+
+image_base64 = encode_image("receipt.png")
+data_url = f"data:image/png;base64,{image_base64}"
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "payment_method": "verbatim-string"
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```json
+{
+  "store": "Trader Joe's",
+  "date": "2025-04-12",
+  "total": 42.85,
+  "payment_method": "Visa"
+}
+```
+
+### Multiple page PDF
+<details>
+You can render a PDF to one PNG image per page with PyMuPDF, then pass the images to vLLM in page order.
+
+```python
+import base64
+import json
+
+import fitz  # pip install pymupdf
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def pdf_to_png_data_urls(pdf_path, dpi=170):
+    data_urls = []
+
+    with fitz.open(pdf_path) as doc:
+        for page in doc:
+            pix = page.get_pixmap(dpi=dpi, alpha=False)
+            png_bytes = pix.tobytes("png")
+            png_base64 = base64.b64encode(png_bytes).decode("utf-8")
+            data_urls.append(f"data:image/png;base64,{png_base64}")
+
+    return data_urls
+
+data_urls = pdf_to_png_data_urls("invoice.pdf", dpi=170)
+
+template = {
+    "invoice_number": "verbatim-string",
+    "invoice_date": "date",
+    "total": "number",
+    "currency": "currency",
+    "line_items": [
+        {
+            "description": "verbatim-string",
+            "quantity": "number",
+            "unit_price": "number",
+            "total": "number"
+        }
+    ]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+                for data_url in data_urls
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+</details>
+
+
+
+## vLLM inference: document-to-Markdown
+
+For Markdown OCR, use `mode="markdown"` or `mode="content"` without a template.
+
+```python
+import base64
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def encode_image(image_path):
+    with open(image_path, "rb") as image_file:
+        return base64.b64encode(image_file.read()).decode("utf-8")
+
+image_base64 = encode_image("document.png")
+data_url = f"data:image/png;base64,{image_base64}"
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=1,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "markdown",
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+---
+
+## vLLM inference: reasoning mode
+<details>
+Reasoning can be enabled for harder structured extraction or Markdown tasks.
+
+```python
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=1,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "markdown",
+            "enable_thinking": True
+        }
+    }
+)
+
+result = response.choices[0].message.content
+
+if "</think>" in result:
+    reasoning, answer = [part.strip() for part in result.split("</think>")]
+else:
+    reasoning, answer = None, result
+
+print(answer)
+```
+</details>
+
+
+## In-context examples for extraction
+<details>
+NuExtract supports in-context examples for structured extraction.
+
+Examples are especially useful when the desired formatting is ambiguous or when the schema requires task-specific conventions. Examples can be provided by using `developer` messages, for which all items of the contents except the last one are the input, and the last one is the expected output.
+
+```python
+import json
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+template = {
+    "names": ["string"]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "developer",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "Stephen is the manager at Susan's store.",
+                },
+                {
+                    "type": "text",
+                    "text": "{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}",
+                }
+            ],
+        },
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "John went to the restaurant with Mary. James went to the cinema."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```json
+{
+  "names": ["-JOHN-", "-MARY-", "-JAMES-"]
+}
+```
+</details>
+
+
+## vLLM inference: template generation
+
+NuExtract can generate an extraction template from a natural language description.
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "I want to extract the key details from a rental contract."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "template-generation"
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```json
+{
+  "contract_title": "verbatim-string",
+  "landlord": "verbatim-string",
+  "tenant": "verbatim-string",
+  "property_address": "verbatim-string",
+  "start_date": "date-time",
+  "end_date": "date-time",
+  "monthly_rent": "number",
+  "currency": "verbatim-string",
+  "deposit": "number",
+  "signatories": ["verbatim-string"]
+}
+```
+
+## Curl examples
+<details>
+
+The following examples assume that vLLM is running locally on port 8000. They use `jq` to build valid JSON request bodies without manually escaping the image data or template string.
+
+### Single image structured extraction
+
+```bash
+API_KEY="EMPTY"
+IMAGE_BASE64_FILE=$(mktemp)
+REQUEST_BODY_FILE=$(mktemp)
+
+base64 < receipt.png | tr -d '\n' > "$IMAGE_BASE64_FILE"
+
+TEMPLATE=$(cat <<'JSON'
+{
+  "store": "verbatim-string",
+  "date": "date-time",
+  "total": "number",
+  "payment_method": "verbatim-string"
+}
+JSON
+)
+
+jq -n \
+  --rawfile image_base64 "$IMAGE_BASE64_FILE" \
+  --arg template "$TEMPLATE" \
+  '{
+    model: "numind/NuExtract3",
+    temperature: 0.6,
+    messages: [
+      {
+        role: "user",
+        content: [
+          {
+            type: "image_url",
+            image_url: {url: ("data:image/png;base64," + $image_base64)}
+          }
+        ]
+      }
+    ],
+    chat_template_kwargs: {
+      template: $template,
+      enable_thinking: false
+    }
+  }' > "$REQUEST_BODY_FILE"
+
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $API_KEY" \
+  --data-binary "@$REQUEST_BODY_FILE"
+
+rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"
+```
+
+### Single image content extraction
+
+```bash
+API_KEY="EMPTY"
+IMAGE_BASE64_FILE=$(mktemp)
+REQUEST_BODY_FILE=$(mktemp)
+
+base64 < document.png | tr -d '\n' > "$IMAGE_BASE64_FILE"
+
+jq -n \
+  --rawfile image_base64 "$IMAGE_BASE64_FILE" \
+  '{
+    model: "numind/NuExtract3",
+    temperature: 0.6,
+    messages: [
+      {
+        role: "user",
+        content: [
+          {
+            type: "image_url",
+            image_url: {url: ("data:image/png;base64," + $image_base64)}
+          }
+        ]
+      }
+    ],
+    chat_template_kwargs: {
+      mode: "content",
+      enable_thinking: false
+    }
+  }' > "$REQUEST_BODY_FILE"
+
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $API_KEY" \
+  --data-binary "@$REQUEST_BODY_FILE"
+
+rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"
+```
+</details>
+
+
+## Transformers example
+<details>
+You can also run NuExtract directly with `transformers`. The same `template`, `mode`, and `enable_thinking` options are passed to `processor.apply_chat_template`.
+
+```python
+import json
+
+import torch
+from PIL import Image
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+model_id = "numind/NuExtract3"
+
+processor = AutoProcessor.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+).eval()
+
+def run_nuextract(messages, **chat_template_kwargs):
+    inputs = processor.apply_chat_template(
+        messages,
+        add_generation_prompt=True,
+        tokenize=True,
+        return_dict=True,
+        return_tensors="pt",
+        **chat_template_kwargs,
+    ).to(model.device)
+
+    with torch.inference_mode():
+        generated_ids = model.generate(
+            **inputs,
+            max_new_tokens=4096,
+            do_sample=False,
+        )
+
+    generated_ids = generated_ids[:, inputs.input_ids.shape[1]:]
+    return processor.batch_decode(
+        generated_ids,
+        skip_special_tokens=True,
+        clean_up_tokenization_spaces=False,
+    )[0].strip()
+
+# Single image structured extraction
+receipt_image = Image.open("receipt.png").convert("RGB")
+receipt_messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": receipt_image,
+            }
+        ],
+    }
+]
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "payment_method": "verbatim-string"
+}
+
+structured_output = run_nuextract(
+    receipt_messages,
+    template=json.dumps(template, indent=4),
+    enable_thinking=False,
+)
+print(structured_output)
+
+# Single image content extraction
+document_image = Image.open("document.png").convert("RGB")
+document_messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": document_image,
+            }
+        ],
+    }
+]
+
+content_output = run_nuextract(
+    document_messages,
+    mode="content",
+    enable_thinking=False,
+)
+print(content_output)
+```
+</details>
+
+Special thanks to the Lambda.ai team for the compute that made this project a success.
+
+## Citation
+
+If you use NuExtract, please cite NuMind and link to the model page.
+
+```bibtex
+@misc{nuextract3,
+  title = {NuExtract3},
+  author = {NuMind},
+  year = {2026},
+  url = {https://nuextract.ai/}
+}
+```

.agents/references/nuextract3/2026-07-20/huggingface.co/numind/NuExtract3.md

@@ -0,0 +1,1010 @@
+<!--
+source: https://huggingface.co/numind/NuExtract3
+retrieved: 2026-07-20T17:29:25.403047+00:00
+final_url: https://huggingface.co/numind/NuExtract3
+content_type: text/html
+-->
+
+# numind/NuExtract3 · Hugging Face
+
+[Hugging Face](https://huggingface.co/)
+
+# 
+
+[https://huggingface.co/numind](https://huggingface.co/numind)
+
+[numind](https://huggingface.co/numind)
+
+/
+
+[NuExtract3](https://huggingface.co/numind/NuExtract3)
+
+like 295
+
+Follow
+
+NuMind 649
+
+[Image-to-Text](https://huggingface.co/models?pipeline_tag=image-to-text)
+
+[Transformers](https://huggingface.co/models?library=transformers)
+
+[Safetensors](https://huggingface.co/models?library=safetensors)
+
+[qwen3_5](https://huggingface.co/models?other=qwen3_5)
+
+[image-text-to-text](https://huggingface.co/models?other=image-text-to-text)
+
+[vision-language](https://huggingface.co/models?other=vision-language)
+
+[vlm](https://huggingface.co/models?other=vlm)
+
+[document-understanding](https://huggingface.co/models?other=document-understanding)
+
+[structured-extraction](https://huggingface.co/models?other=structured-extraction)
+
+[information-extraction](https://huggingface.co/models?other=information-extraction)
+
+[ocr](https://huggingface.co/models?other=ocr)
+
+[document-to-markdown](https://huggingface.co/models?other=document-to-markdown)
+
+[markdown](https://huggingface.co/models?other=markdown)
+
+[rag](https://huggingface.co/models?other=rag)
+
+[reasoning](https://huggingface.co/models?other=reasoning)
+
+[multilingual](https://huggingface.co/models?other=multilingual)
+
+[conversational](https://huggingface.co/models?other=conversational)
+
+[Eval Results](https://huggingface.co/models?other=eval-results)
+
+License: apache-2.0
+
+[Model card](https://huggingface.co/numind/NuExtract3)
+
+[Files Files and versions xet](https://huggingface.co/numind/NuExtract3/tree/main)
+
+[Community 2](https://huggingface.co/numind/NuExtract3/discussions)
+
+ Deploy
+
+ Copy to bucket new
+
+ Use this model
+
+[https://nuextract.ai/](https://nuextract.ai/)
+
+ 🖥️ [API / Platform](https://nuextract.ai/)  | 📑 [Blog](https://numind.ai/blog)  | 🗣️ [Discord](https://discord.gg/3tsEtJNCDe)  | 🛠️ [GitHub](https://github.com/numindai/nuextract)
+
+NuExtract3  is a unified 4B  vision-language reasoning model for document understanding.
+
+It combines strong structured information extraction  with high-quality image-to-Markdown  conversion, making it suitable for extraction pipelines, OCR, and RAG preprocessing for all types of documents such as scans, receipts, forms, invoices, contracts or tables.
+
+Try it out in [the 🤗 space!](https://huggingface.co/spaces/numind/NuExtract-3-4B)
+
+## [https://huggingface.co/numind/NuExtract3#overview](https://huggingface.co/numind/NuExtract3#overview)  Overview 
+
+- Structured extraction : input (text/images) + JSON template + instructions --> JSON output
+
+- Markdown conversion : input (text/images) --> Markdown
+
+- Multimodal inputs : text, images, or text + images.
+
+- Multilingual  documents.
+
+- Reasoning  and non-reasoning inference modes.
+
+- Template generation  for structured extraction from natural language or input document.
+
+# [https://huggingface.co/numind/NuExtract3#benchmark-results](https://huggingface.co/numind/NuExtract3#benchmark-results)  Benchmark results 
+
+## [https://huggingface.co/numind/NuExtract3#structured-extraction](https://huggingface.co/numind/NuExtract3#structured-extraction)  Structured Extraction 
+
+We benchmarked NuExtract on NuMind's internal structured benchmark, measuring model's performances on ~600 documents of diverse types including invoices, movie posters or floor plans. These documents and their ground-truth cover diverse use-cases testing model visual understanding, OCR, reasoning and ability to handle long input and output contexts. We plan to open-source this benchmark in the coming weeks, along with a extensive leaderboard including most popular open-weight and closed-sourced APIs and a Python library allowing to easily measure model performances on structured extraction.
+
+To measure a pair of predicted and ground-truth JSONs, we represent both as trees which we align based on node names, compute metric scores for aligned leaves and report the average of these scores. `string` and `verbatim-string` leaves are evaluated with indel distance (i.e. Levenshtein without replacement), while all others are evaluated with exact-match. Models were evaluated using vllm, with a temperature of 0.25 and a maximum of 65000 output token (for both thinking and answer), which largely exceeds 22000 which is the number of tokens of the largest ground truth output.
+
+Model name Average score Num. failed⁽¹⁾ Avg. num tokens thinking Avg. num tokens answer NuExtract3.4_4B-RL 0.651 ± 0.019 27 2036 1856 gemma-4-E4B-it 0.538 ± 0.023 31 3005 1287 Qwen3.5-9B 0.479 ± 0.030 170 22409 1257 Qwen3.5-4B 0.417 ± 0.031 229 27177 1201 GLM-4.6V-Flash 0.435 ± 0.026 153 2989 1357 Nemotron-3-Nano-Omni 0.387 ± 0.028 204 25827 522 Ministral-3-3B 0.240 ± 0.022 344 27586 362
+
+ (1) number of model outputs that were not JSON deserializable, either directly or by removing leading and trailing backticks.
+ 95% confidence intervals computed using a nonparametric bootstrap over scores distributions. 
+
+The benchmark include samples containing multiple images resulting in large input context, and some with ground-truth containing large numbers of items to extract resulting in large outputs. We found that the reasoning of small models significantly negatively impact their performances. The reason is that many models ended up falling in repetition loops, hitting the output tokens limit and resulting in failed requests.
+
+## [https://huggingface.co/numind/NuExtract3#document-to-markdown](https://huggingface.co/numind/NuExtract3#document-to-markdown)  Document to Markdown 
+
+NuExtract can also convert document images into clean Markdown. Output will be Markdown for text (headers etc), HTML for tables, LaTeX for math and `<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/> `
+
+Modern, format-agnostic benchmarks for complex document understanding are limited, so we explored a new evaluation approach. We selected 100 documents with challenging layouts and tables, asked each model to convert them into a structured representation, then used Gemini 3 Flash to compare model outputs against the source document and choose the most accurate result. The rankings aligned with human votes, suggesting this is a promising method for evaluating document-to-Markdown capabilities. More details will be shared in an upcoming technical report. Here are some results:
+
+### [https://huggingface.co/numind/NuExtract3#using-markdown-to-structured](https://huggingface.co/numind/NuExtract3#using-markdown-to-structured)  Using "Markdown-to-structured" 
+
+To add other evaluate references, we used our structured extraction benchmark to evaluate models in a two-step fashion: convert the benchmark inputs to Markdown, then use Qwen3.6 27B to perform the structured extraction task on them. Intuitively, it allows to evaluate how models achieve to keep the input document content and layout: good models will allow the "structured extractor" model to perform better scores.
+
+# [https://huggingface.co/numind/NuExtract3#using-nuextract](https://huggingface.co/numind/NuExtract3#using-nuextract)  Using NuExtract 
+
+## [https://huggingface.co/numind/NuExtract3#structured-extraction-1](https://huggingface.co/numind/NuExtract3#structured-extraction-1)  Structured extraction 
+
+Structured extraction takes as inputs:
+
+1. An input document, which can be text, image, or both;
+
+2. A JSON template describing the information to extract;
+
+3. (Optional) Instructions, allowing to specify expected output formats or values, to provide with the `instructions` chat template kwarg;
+
+4. (Optional) In-Context Learning (ICL) examples.
+
+### [https://huggingface.co/numind/NuExtract3#input-json-template](https://huggingface.co/numind/NuExtract3#input-json-template)  Input JSON template 
+
+NuExtract uses a input JSON template whose structure is identical to the output JSON. Its leaf values are specify the types  of the output JSON leaves. For examples:
+
+```
+{
+  "invoice_number": "verbatim-string",
+  "invoice_date": "date",
+  "total_amount": "number",
+  "currency": "currency",
+  "line_items": [
+    {
+      "description": "verbatim-string",
+      "item_type": ["electronics", "clothing", "vehicle", "furniture", "other"],
+      "quantity": "integer",
+      "unit_price": "number",
+      "total": "number"
+    }
+  ]
+}
+```
+
+Supported template types include:
+
+- `verbatim-string`: extract text exactly as it appears in the document;
+
+- `string`: generic string field, allowing abstraction or light paraphrasing;
+
+- `integer`: whole number;
+
+- `number`: integer or decimal number;
+
+- `date-time`: ISO-8601 date, time or date-time;
+
+- Other specific types such as `data`, `time`, `country`, `currency`, `email` and so on. [For more details, read the complete types specifications and examples](https://huggingface.co/numind/NuExtract3/blob/main/TYPES.md)
+
+Template constructors:
+
+- Arrays, for example `["string"]`;
+
+- Enums, for example `["yes", "no", "maybe"]`;
+
+- Multi-enums (multiple possible values), for example `[["A", "B", "C"]]`.
+
+If the model does not find relevant information for a field, it returns `null` or `[]`.
+
+### [https://huggingface.co/numind/NuExtract3#converting-json-schema--pydantic-models-to-nuextract-template](https://huggingface.co/numind/NuExtract3#converting-json-schema--pydantic-models-to-nuextract-template)  Converting JSON schema / Pydantic models to NuExtract template 
+
+Our Python SDK (`pip install numind`) offers a method to convert JSON schemas to NuExtract templates:
+
+```
+from typing import Literal
+
+from pydantic import Field, BaseModel
+from numind.nuextract_utils import convert_json_schema_to_nuextract_template
+
+class HotelBooking(BaseModel):
+    city: str
+    check_in_date: str = Field(description="date")
+    check_out_date: str = Field(description="date")
+    number_of_guests: int
+    room_type: Literal["single", "double", "suite"]
+
+template, dropped_branches = convert_json_schema_to_nuextract_template(
+    HotelBooking.model_json_schema()
+)
+
+# {'check_in_date': 'date', 'check_out_date': 'date', 'city': 'string', 'number_of_guests': 'integer', 'room_type': ['single', 'double', 'suite']}
+```
+
+## [https://huggingface.co/numind/NuExtract3#document-to-markdown-1](https://huggingface.co/numind/NuExtract3#document-to-markdown-1)  Document-to-Markdown 
+
+NuExtract can also convert document images into clean Markdown. Output will be markdown for text (headers etc), html for tables, latex for mat and `<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/> `
+
+Markdown example:
+
+```
+<figure data-type="image" data-id="img_1">
+  <img src="/numind/NuExtract3/resolve/main/img_1.png" alt="Logo of Mobilier 2000 with contact information: Tél.: (418) 275-4232, 1654, boul. Marcotte, Roberval (Qc) G8H 2P2"/>
+</figure>
+
+# COMMANDE
+**NUMÉRO 72259**
+
+1
+
+**Vendu à**
+TREMBLAY ERIC
+ERIC TREMBLAY
+348 BOUL. DE L'ANSE
+ROBERVAL
+G8H 1Y9
+
+**Livré à**
+TREMBLAY ERIC
+ERIC TREMBLAY
+348 BOUL. DE L'ANSE
+ROBERVAL
+G8H 1Y9
+
+<table>
+  <thead>
+    <tr>
+      <th># CLIENT</th>
+      <th>EXPÉDITEUR</th>
+      <th>TERME DE CRÉDIT</th>
+      <th>DATE</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>2753133</td>
+      <td>Notre camion</td>
+      <td>à la livraison</td>
+      <td>22/06/2023</td>
+    </tr>
+  </tbody>
+</table>
+
+<table>
+  <thead>
+    <tr>
+      <th>NOM DU VENDEUR</th>
+      <th>VOTRE ÉCONOMIE !</th>
+      <th># COMMANDE</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>Éric</td>
+      <td>0.00</td>
+      <td></td>
+    </tr>
+  </tbody>
+</table>
+```
+
+## [https://huggingface.co/numind/NuExtract3#reasoning-and-non-reasoning-modes](https://huggingface.co/numind/NuExtract3#reasoning-and-non-reasoning-modes)  Reasoning and non-reasoning modes 
+
+NuExtract supports both reasoning and non-reasoning inference.
+
+### [https://huggingface.co/numind/NuExtract3#non-thinking-mode](https://huggingface.co/numind/NuExtract3#non-thinking-mode)  Non-thinking mode 
+
+Use this for fast and deterministic extraction or Markdown conversion.
+
+```
+enable_thinking = False
+temperature = 0.2
+```
+
+### [https://huggingface.co/numind/NuExtract3#thinking-mode](https://huggingface.co/numind/NuExtract3#thinking-mode)  Thinking mode 
+
+Use this for difficult documents, complex layouts, ambiguous fields, or cases where the document structure requires additional reasoning.
+
+```
+enable_thinking = True
+temperature = 0.6
+```
+
+For production extraction workloads, we recommend starting with non-reasoning mode  and enabling reasoning only for difficult examples.
+
+## [https://huggingface.co/numind/NuExtract3#vllm-deployment](https://huggingface.co/numind/NuExtract3#vllm-deployment)  vLLM deployment 
+
+NuExtract can be served with vLLM using an OpenAI-compatible API.
+
+```
+vllm serve numind/NuExtract3 \
+  --trust-remote-code \
+  --limit-mm-per-prompt '{"image": 99, "video": 0}' \
+  --chat-template-content-format openai \
+  --generation-config vllm \
+  --max-model-len 131072 \
+  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+
+### [https://huggingface.co/numind/NuExtract3#multi-token-prediction](https://huggingface.co/numind/NuExtract3#multi-token-prediction)  Multi Token Prediction 
+
+ The deployment commands above enable Multi Token Prediction (MTP) through vLLM speculative decoding: 
+
+```
+--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+
+MTP can improve decoding throughput without changing the OpenAI-compatible request payload. You can tune `num_speculative_tokens` for your hardware and workload, or remove `--speculative-config` if your vLLM version or environment does not support this speculative decoding method.
+
+If you encounter memory issues, reduce the maximum model length and the maximum number of images:
+
+```
+vllm serve numind/NuExtract-3 \
+  --trust-remote-code \
+  --limit-mm-per-prompt '{"image": 6, "video": 0}' \
+  --chat-template-content-format openai \
+  --generation-config vllm \
+  --max-model-len 16384 \
+  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+
+## [https://huggingface.co/numind/NuExtract3#vllm-inference-structured-extraction-text](https://huggingface.co/numind/NuExtract3#vllm-inference-structured-extraction-text)  vLLM inference: structured extraction: text 
+
+```
+import json
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "currency": ["USD", "EUR", "GBP", "JPY", "Other"],
+    "items": [
+        {
+            "name": "verbatim-string",
+            "price": "number"
+        }
+    ]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "Yesterday I bought apples and coffee at Trader Joe's for a total of $12.40."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template),
+            "instructions": "Specify the time for the `date` entry only if it is present, otherwise only output the date component.",
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```
+{
+  "store": "Trader Joe's",
+  "date": null,
+  "total": 12.40,
+  "currency": "USD",
+  "items": [
+    {
+      "name": "apples",
+      "price": null
+    },
+    {
+      "name": "coffee",
+      "price": null
+    }
+  ]
+}
+```
+
+## [https://huggingface.co/numind/NuExtract3#vllm-inference-structured-extraction-image](https://huggingface.co/numind/NuExtract3#vllm-inference-structured-extraction-image)  vLLM inference: structured extraction: image 
+
+```
+import json
+import base64
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def encode_image(image_path):
+    with open(image_path, "rb") as image_file:
+        return base64.b64encode(image_file.read()).decode("utf-8")
+
+image_base64 = encode_image("receipt.png")
+data_url = f"data:image/png;base64,{image_base64}"
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "payment_method": "verbatim-string"
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```
+{
+  "store": "Trader Joe's",
+  "date": "2025-04-12",
+  "total": 42.85,
+  "payment_method": "Visa"
+}
+```
+
+### [https://huggingface.co/numind/NuExtract3#multiple-page-pdf](https://huggingface.co/numind/NuExtract3#multiple-page-pdf)  Multiple page PDF 
+
+ You can render a PDF to one PNG image per page with PyMuPDF, then pass the images to vLLM in page order. 
+
+```
+import base64
+import json
+
+import fitz  # pip install pymupdf
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def pdf_to_png_data_urls(pdf_path, dpi=170):
+    data_urls = []
+
+    with fitz.open(pdf_path) as doc:
+        for page in doc:
+            pix = page.get_pixmap(dpi=dpi, alpha=False)
+            png_bytes = pix.tobytes("png")
+            png_base64 = base64.b64encode(png_bytes).decode("utf-8")
+            data_urls.append(f"data:image/png;base64,{png_base64}")
+
+    return data_urls
+
+data_urls = pdf_to_png_data_urls("invoice.pdf", dpi=170)
+
+template = {
+    "invoice_number": "verbatim-string",
+    "invoice_date": "date",
+    "total": "number",
+    "currency": "currency",
+    "line_items": [
+        {
+            "description": "verbatim-string",
+            "quantity": "number",
+            "unit_price": "number",
+            "total": "number"
+        }
+    ]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+                for data_url in data_urls
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+## [https://huggingface.co/numind/NuExtract3#vllm-inference-document-to-markdown](https://huggingface.co/numind/NuExtract3#vllm-inference-document-to-markdown)  vLLM inference: document-to-Markdown 
+
+For Markdown OCR, use `mode="markdown"` or `mode="content"` without a template.
+
+```
+import base64
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def encode_image(image_path):
+    with open(image_path, "rb") as image_file:
+        return base64.b64encode(image_file.read()).decode("utf-8")
+
+image_base64 = encode_image("document.png")
+data_url = f"data:image/png;base64,{image_base64}"
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=1,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "markdown",
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+## [https://huggingface.co/numind/NuExtract3#vllm-inference-reasoning-mode](https://huggingface.co/numind/NuExtract3#vllm-inference-reasoning-mode)  vLLM inference: reasoning mode 
+
+ Reasoning can be enabled for harder structured extraction or Markdown tasks. 
+
+```
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=1,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "markdown",
+            "enable_thinking": True
+        }
+    }
+)
+
+result = response.choices[0].message.content
+
+if "</think>" in result:
+    reasoning, answer = [part.strip() for part in result.split("</think>")]
+else:
+    reasoning, answer = None, result
+
+print(answer)
+```
+
+## [https://huggingface.co/numind/NuExtract3#in-context-examples-for-extraction](https://huggingface.co/numind/NuExtract3#in-context-examples-for-extraction)  In-context examples for extraction 
+
+ NuExtract supports in-context examples for structured extraction. 
+
+Examples are especially useful when the desired formatting is ambiguous or when the schema requires task-specific conventions. Examples can be provided by using `developer` messages, for which all items of the contents except the last one are the input, and the last one is the expected output.
+
+```
+import json
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+template = {
+    "names": ["string"]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "developer",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "Stephen is the manager at Susan's store.",
+                },
+                {
+                    "type": "text",
+                    "text": "{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}",
+                }
+            ],
+        },
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "John went to the restaurant with Mary. James went to the cinema."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```
+{
+  "names": ["-JOHN-", "-MARY-", "-JAMES-"]
+}
+```
+
+## [https://huggingface.co/numind/NuExtract3#vllm-inference-template-generation](https://huggingface.co/numind/NuExtract3#vllm-inference-template-generation)  vLLM inference: template generation 
+
+NuExtract can generate an extraction template from a natural language description.
+
+```
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "I want to extract the key details from a rental contract."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "template-generation"
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```
+{
+  "contract_title": "verbatim-string",
+  "landlord": "verbatim-string",
+  "tenant": "verbatim-string",
+  "property_address": "verbatim-string",
+  "start_date": "date-time",
+  "end_date": "date-time",
+  "monthly_rent": "number",
+  "currency": "verbatim-string",
+  "deposit": "number",
+  "signatories": ["verbatim-string"]
+}
+```
+
+## [https://huggingface.co/numind/NuExtract3#curl-examples](https://huggingface.co/numind/NuExtract3#curl-examples)  Curl examples 
+
+The following examples assume that vLLM is running locally on port 8000. They use `jq` to build valid JSON request bodies without manually escaping the image data or template string.
+
+### [https://huggingface.co/numind/NuExtract3#single-image-structured-extraction](https://huggingface.co/numind/NuExtract3#single-image-structured-extraction)  Single image structured extraction 
+
+```
+API_KEY="EMPTY"
+IMAGE_BASE64_FILE=$(mktemp)
+REQUEST_BODY_FILE=$(mktemp)
+
+base64 < receipt.png | tr -d '\n' > "$IMAGE_BASE64_FILE"
+
+TEMPLATE=$(cat <<'JSON'
+{
+  "store": "verbatim-string",
+  "date": "date-time",
+  "total": "number",
+  "payment_method": "verbatim-string"
+}
+JSON
+)
+
+jq -n \
+  --rawfile image_base64 "$IMAGE_BASE64_FILE" \
+  --arg template "$TEMPLATE" \
+  '{
+    model: "numind/NuExtract3",
+    temperature: 0.6,
+    messages: [
+      {
+        role: "user",
+        content: [
+          {
+            type: "image_url",
+            image_url: {url: ("data:image/png;base64," + $image_base64)}
+          }
+        ]
+      }
+    ],
+    chat_template_kwargs: {
+      template: $template,
+      enable_thinking: false
+    }
+  }' > "$REQUEST_BODY_FILE"
+
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $API_KEY" \
+  --data-binary "@$REQUEST_BODY_FILE"
+
+rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"
+```
+
+### [https://huggingface.co/numind/NuExtract3#single-image-content-extraction](https://huggingface.co/numind/NuExtract3#single-image-content-extraction)  Single image content extraction 
+
+```
+API_KEY="EMPTY"
+IMAGE_BASE64_FILE=$(mktemp)
+REQUEST_BODY_FILE=$(mktemp)
+
+base64 < document.png | tr -d '\n' > "$IMAGE_BASE64_FILE"
+
+jq -n \
+  --rawfile image_base64 "$IMAGE_BASE64_FILE" \
+  '{
+    model: "numind/NuExtract3",
+    temperature: 0.6,
+    messages: [
+      {
+        role: "user",
+        content: [
+          {
+            type: "image_url",
+            image_url: {url: ("data:image/png;base64," + $image_base64)}
+          }
+        ]
+      }
+    ],
+    chat_template_kwargs: {
+      mode: "content",
+      enable_thinking: false
+    }
+  }' > "$REQUEST_BODY_FILE"
+
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $API_KEY" \
+  --data-binary "@$REQUEST_BODY_FILE"
+
+rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"
+```
+
+## [https://huggingface.co/numind/NuExtract3#transformers-example](https://huggingface.co/numind/NuExtract3#transformers-example)  Transformers example 
+
+ You can also run NuExtract directly with `transformers`. The same `template`, `mode`, and `enable_thinking` options are passed to `processor.apply_chat_template`. 
+
+```
+import json
+
+import torch
+from PIL import Image
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+model_id = "numind/NuExtract3"
+
+processor = AutoProcessor.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+).eval()
+
+def run_nuextract(messages, **chat_template_kwargs):
+    inputs = processor.apply_chat_template(
+        messages,
+        add_generation_prompt=True,
+        tokenize=True,
+        return_dict=True,
+        return_tensors="pt",
+        **chat_template_kwargs,
+    ).to(model.device)
+
+    with torch.inference_mode():
+        generated_ids = model.generate(
+            **inputs,
+            max_new_tokens=4096,
+            do_sample=False,
+        )
+
+    generated_ids = generated_ids[:, inputs.input_ids.shape[1]:]
+    return processor.batch_decode(
+        generated_ids,
+        skip_special_tokens=True,
+        clean_up_tokenization_spaces=False,
+    )[0].strip()
+
+# Single image structured extraction
+receipt_image = Image.open("receipt.png").convert("RGB")
+receipt_messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": receipt_image,
+            }
+        ],
+    }
+]
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "payment_method": "verbatim-string"
+}
+
+structured_output = run_nuextract(
+    receipt_messages,
+    template=json.dumps(template, indent=4),
+    enable_thinking=False,
+)
+print(structured_output)
+
+# Single image content extraction
+document_image = Image.open("document.png").convert("RGB")
+document_messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": document_image,
+            }
+        ],
+    }
+]
+
+content_output = run_nuextract(
+    document_messages,
+    mode="content",
+    enable_thinking=False,
+)
+print(content_output)
+```
+
+Special thanks to the Lambda.ai team for the compute that made this project a success.
+
+## [https://huggingface.co/numind/NuExtract3#citation](https://huggingface.co/numind/NuExtract3#citation)  Citation 
+
+If you use NuExtract, please cite NuMind and link to the model page.
+
+```
+@misc{nuextract3,
+  title = {NuExtract3},
+  author = {NuMind},
+  year = {2026},
+  url = {https://nuextract.ai/}
+}
+```
+
+Downloads last month 1,236,855
+
+Safetensors[https://huggingface.co/docs/safetensors](https://huggingface.co/docs/safetensors)
+
+Model size
+
+5B params
+
+Tensor type
+
+BF16 
+
+·
+
+Chat template
+
+Files info
+
+Inference Providers[NEW](https://huggingface.co/docs/inference-providers)
+
+[Image-to-Text](https://huggingface.co/tasks/image-to-text)
+
+This model isn't deployed by any Inference Provider.[🙋 3 Ask for provider support](https://huggingface.co/spaces/huggingface/InferenceSupport/discussions/10487)
+
+##  Model tree for numind/NuExtract3[https://huggingface.co/docs/hub/model-cards#specifying-a-base-model](https://huggingface.co/docs/hub/model-cards#specifying-a-base-model)
+
+Base model
+
+[Qwen/Qwen3.5-4B-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base)
+
+ Finetuned
+
+[Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B)
+
+ Finetuned
+
+([392](https://huggingface.co/models?other=base_model:finetune:Qwen/Qwen3.5-4B) )
+
+this model
+
+Finetunes
+
+[4 models](https://huggingface.co/models?other=base_model:finetune:numind/NuExtract3)
+
+Quantizations
+
+[11 models](https://huggingface.co/models?other=base_model:quantized:numind/NuExtract3)
+
+##  Spaces using numind/NuExtract3 9
+
+##  Collection including numind/NuExtract3
+
+#### 
+
+[NuExtract3  Collection 12 items • Updated May 26 •  15](https://huggingface.co/collections/numind/nuextract3)
+
+##  Evaluation results [https://huggingface.co/docs/hub/eval-results](https://huggingface.co/docs/hub/eval-results)
+
+- [allenai/olmOCR-bench](https://huggingface.co/datasets/allenai/olmOCR-bench) · Old Scans[View evaluation results](https://huggingface.co/numind/NuExtract3/blob/main/.eval_results/olmocrbench.yaml)
+
+[source](https://github.com/davanstrien/ocr-bench/blob/99f7550c/experiments/olmocr-bench-oldscans/BENCHMARKING.md)[leaderboard](https://huggingface.co/datasets/allenai/olmOCR-bench?eval_result=numind/NuExtract3&leaderboard_task_id=old_scans)
+
+37.8 *

.agents/references/nuextract3/2026-07-20/huggingface.co/numind/NuExtract3/blob/main/README.md

@@ -0,0 +1,1001 @@
+<!--
+source: https://huggingface.co/numind/NuExtract3/blob/main/README.md
+retrieved: 2026-07-20T17:29:25.828402+00:00
+final_url: https://huggingface.co/numind/NuExtract3/blob/main/README.md
+content_type: text/html
+-->
+
+# README.md · numind/NuExtract3 at main
+
+[Hugging Face](https://huggingface.co/)
+
+# 
+
+[https://huggingface.co/numind](https://huggingface.co/numind)
+
+[numind](https://huggingface.co/numind)
+
+/
+
+[NuExtract3](https://huggingface.co/numind/NuExtract3)
+
+like 295
+
+Follow
+
+NuMind 649
+
+[Image-to-Text](https://huggingface.co/models?pipeline_tag=image-to-text)
+
+[Transformers](https://huggingface.co/models?library=transformers)
+
+[Safetensors](https://huggingface.co/models?library=safetensors)
+
+[qwen3_5](https://huggingface.co/models?other=qwen3_5)
+
+[image-text-to-text](https://huggingface.co/models?other=image-text-to-text)
+
+[vision-language](https://huggingface.co/models?other=vision-language)
+
+[vlm](https://huggingface.co/models?other=vlm)
+
+[document-understanding](https://huggingface.co/models?other=document-understanding)
+
+[structured-extraction](https://huggingface.co/models?other=structured-extraction)
+
+[information-extraction](https://huggingface.co/models?other=information-extraction)
+
+[ocr](https://huggingface.co/models?other=ocr)
+
+[document-to-markdown](https://huggingface.co/models?other=document-to-markdown)
+
+[markdown](https://huggingface.co/models?other=markdown)
+
+[rag](https://huggingface.co/models?other=rag)
+
+[reasoning](https://huggingface.co/models?other=reasoning)
+
+[multilingual](https://huggingface.co/models?other=multilingual)
+
+[conversational](https://huggingface.co/models?other=conversational)
+
+[Eval Results](https://huggingface.co/models?other=eval-results)
+
+License: apache-2.0
+
+[Model card](https://huggingface.co/numind/NuExtract3)
+
+[Files Files and versions xet](https://huggingface.co/numind/NuExtract3/tree/main)
+
+[Community 2](https://huggingface.co/numind/NuExtract3/discussions)
+
+ Deploy
+
+ Copy to bucket new
+
+ Use this model
+
+ main
+
+[NuExtract3](https://huggingface.co/numind/NuExtract3/tree/main) / README.md
+
+[NathanFradet](https://huggingface.co/NathanFradet)
+
+Update README.md
+
+[acaf70e](https://huggingface.co/numind/NuExtract3/commit/acaf70ecff9c3dbbfcbae651b82b66a0d8dbd0c6) verified about 2 months ago
+
+[preview](https://huggingface.co/numind/NuExtract3/blob/main/README.md)[code](https://huggingface.co/numind/NuExtract3/blob/main/README.md?code=true)
+
+|
+
+[Raw](https://huggingface.co/numind/NuExtract3/raw/main/README.md)
+
+ Download with hf CLI
+
+Copy download link
+
+[History](https://huggingface.co/numind/NuExtract3/commits/main/README.md)[Blame](https://huggingface.co/numind/NuExtract3/blame/main/README.md)[Contribute](https://huggingface.co/numind/NuExtract3/edit/main/README.md)[Delete](https://huggingface.co/numind/NuExtract3/delete/main/README.md)
+
+Safe
+
+ 25.5 kB
+
+[metadata](https://huggingface.co/docs/hub/model-cards#model-card-metadata)
+
+```
+license: apache-2.0
+license_link: https://huggingface.co/numind/NuExtract3/blob/main/LICENSE
+library_name: transformers
+pipeline_tag: image-to-text
+tags:
+  - image-text-to-text
+  - transformers
+  - safetensors
+  - qwen3_5
+  - vision-language
+  - vlm
+  - document-understanding
+  - structured-extraction
+  - information-extraction
+  - ocr
+  - document-to-markdown
+  - markdown
+  - rag
+  - reasoning
+  - multilingual
+  - conversational
+base_model:
+  - Qwen/Qwen3.5-4B
+model_name: NuExtract3
+```
+
+[https://nuextract.ai/](https://nuextract.ai/)
+
+ 🖥️ [API / Platform](https://nuextract.ai/)  | 📑 [Blog](https://numind.ai/blog)  | 🗣️ [Discord](https://discord.gg/3tsEtJNCDe)  | 🛠️ [GitHub](https://github.com/numindai/nuextract)
+
+NuExtract3  is a unified 4B  vision-language reasoning model for document understanding.
+
+It combines strong structured information extraction  with high-quality image-to-Markdown  conversion, making it suitable for extraction pipelines, OCR, and RAG preprocessing for all types of documents such as scans, receipts, forms, invoices, contracts or tables.
+
+Try it out in [the 🤗 space!](https://huggingface.co/spaces/numind/NuExtract-3-4B)
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#overview](https://huggingface.co/numind/NuExtract3/blob/main/README.md#overview)  Overview 
+
+- Structured extraction : input (text/images) + JSON template + instructions --> JSON output
+
+- Markdown conversion : input (text/images) --> Markdown
+
+- Multimodal inputs : text, images, or text + images.
+
+- Multilingual  documents.
+
+- Reasoning  and non-reasoning inference modes.
+
+- Template generation  for structured extraction from natural language or input document.
+
+# [https://huggingface.co/numind/NuExtract3/blob/main/README.md#benchmark-results](https://huggingface.co/numind/NuExtract3/blob/main/README.md#benchmark-results)  Benchmark results 
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#structured-extraction](https://huggingface.co/numind/NuExtract3/blob/main/README.md#structured-extraction)  Structured Extraction 
+
+We benchmarked NuExtract on NuMind's internal structured benchmark, measuring model's performances on ~600 documents of diverse types including invoices, movie posters or floor plans. These documents and their ground-truth cover diverse use-cases testing model visual understanding, OCR, reasoning and ability to handle long input and output contexts. We plan to open-source this benchmark in the coming weeks, along with a extensive leaderboard including most popular open-weight and closed-sourced APIs and a Python library allowing to easily measure model performances on structured extraction.
+
+To measure a pair of predicted and ground-truth JSONs, we represent both as trees which we align based on node names, compute metric scores for aligned leaves and report the average of these scores. `string` and `verbatim-string` leaves are evaluated with indel distance (i.e. Levenshtein without replacement), while all others are evaluated with exact-match. Models were evaluated using vllm, with a temperature of 0.25 and a maximum of 65000 output token (for both thinking and answer), which largely exceeds 22000 which is the number of tokens of the largest ground truth output.
+
+Model name Average score Num. failed⁽¹⁾ Avg. num tokens thinking Avg. num tokens answer NuExtract3.4_4B-RL 0.651 ± 0.019 27 2036 1856 gemma-4-E4B-it 0.538 ± 0.023 31 3005 1287 Qwen3.5-9B 0.479 ± 0.030 170 22409 1257 Qwen3.5-4B 0.417 ± 0.031 229 27177 1201 GLM-4.6V-Flash 0.435 ± 0.026 153 2989 1357 Nemotron-3-Nano-Omni 0.387 ± 0.028 204 25827 522 Ministral-3-3B 0.240 ± 0.022 344 27586 362
+
+ (1) number of model outputs that were not JSON deserializable, either directly or by removing leading and trailing backticks.
+ 95% confidence intervals computed using a nonparametric bootstrap over scores distributions. 
+
+The benchmark include samples containing multiple images resulting in large input context, and some with ground-truth containing large numbers of items to extract resulting in large outputs. We found that the reasoning of small models significantly negatively impact their performances. The reason is that many models ended up falling in repetition loops, hitting the output tokens limit and resulting in failed requests.
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#document-to-markdown](https://huggingface.co/numind/NuExtract3/blob/main/README.md#document-to-markdown)  Document to Markdown 
+
+NuExtract can also convert document images into clean Markdown. Output will be Markdown for text (headers etc), HTML for tables, LaTeX for math and `<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/> `
+
+Modern, format-agnostic benchmarks for complex document understanding are limited, so we explored a new evaluation approach. We selected 100 documents with challenging layouts and tables, asked each model to convert them into a structured representation, then used Gemini 3 Flash to compare model outputs against the source document and choose the most accurate result. The rankings aligned with human votes, suggesting this is a promising method for evaluating document-to-Markdown capabilities. More details will be shared in an upcoming technical report. Here are some results:
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#using-markdown-to-structured](https://huggingface.co/numind/NuExtract3/blob/main/README.md#using-markdown-to-structured)  Using "Markdown-to-structured" 
+
+To add other evaluate references, we used our structured extraction benchmark to evaluate models in a two-step fashion: convert the benchmark inputs to Markdown, then use Qwen3.6 27B to perform the structured extraction task on them. Intuitively, it allows to evaluate how models achieve to keep the input document content and layout: good models will allow the "structured extractor" model to perform better scores.
+
+# [https://huggingface.co/numind/NuExtract3/blob/main/README.md#using-nuextract](https://huggingface.co/numind/NuExtract3/blob/main/README.md#using-nuextract)  Using NuExtract 
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#structured-extraction-1](https://huggingface.co/numind/NuExtract3/blob/main/README.md#structured-extraction-1)  Structured extraction 
+
+Structured extraction takes as inputs:
+
+1. An input document, which can be text, image, or both;
+
+2. A JSON template describing the information to extract;
+
+3. (Optional) Instructions, allowing to specify expected output formats or values, to provide with the `instructions` chat template kwarg;
+
+4. (Optional) In-Context Learning (ICL) examples.
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#input-json-template](https://huggingface.co/numind/NuExtract3/blob/main/README.md#input-json-template)  Input JSON template 
+
+NuExtract uses a input JSON template whose structure is identical to the output JSON. Its leaf values are specify the types  of the output JSON leaves. For examples:
+
+```
+{
+  "invoice_number": "verbatim-string",
+  "invoice_date": "date",
+  "total_amount": "number",
+  "currency": "currency",
+  "line_items": [
+    {
+      "description": "verbatim-string",
+      "item_type": ["electronics", "clothing", "vehicle", "furniture", "other"],
+      "quantity": "integer",
+      "unit_price": "number",
+      "total": "number"
+    }
+  ]
+}
+```
+
+Supported template types include:
+
+- `verbatim-string`: extract text exactly as it appears in the document;
+
+- `string`: generic string field, allowing abstraction or light paraphrasing;
+
+- `integer`: whole number;
+
+- `number`: integer or decimal number;
+
+- `date-time`: ISO-8601 date, time or date-time;
+
+- Other specific types such as `data`, `time`, `country`, `currency`, `email` and so on. [For more details, read the complete types specifications and examples](https://huggingface.co/numind/NuExtract3/blob/main/TYPES.md)
+
+Template constructors:
+
+- Arrays, for example `["string"]`;
+
+- Enums, for example `["yes", "no", "maybe"]`;
+
+- Multi-enums (multiple possible values), for example `[["A", "B", "C"]]`.
+
+If the model does not find relevant information for a field, it returns `null` or `[]`.
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#converting-json-schema--pydantic-models-to-nuextract-template](https://huggingface.co/numind/NuExtract3/blob/main/README.md#converting-json-schema--pydantic-models-to-nuextract-template)  Converting JSON schema / Pydantic models to NuExtract template 
+
+Our Python SDK (`pip install numind`) offers a method to convert JSON schemas to NuExtract templates:
+
+```
+from typing import Literal
+
+from pydantic import Field, BaseModel
+from numind.nuextract_utils import convert_json_schema_to_nuextract_template
+
+class HotelBooking(BaseModel):
+    city: str
+    check_in_date: str = Field(description="date")
+    check_out_date: str = Field(description="date")
+    number_of_guests: int
+    room_type: Literal["single", "double", "suite"]
+
+template, dropped_branches = convert_json_schema_to_nuextract_template(
+    HotelBooking.model_json_schema()
+)
+
+# {'check_in_date': 'date', 'check_out_date': 'date', 'city': 'string', 'number_of_guests': 'integer', 'room_type': ['single', 'double', 'suite']}
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#document-to-markdown-1](https://huggingface.co/numind/NuExtract3/blob/main/README.md#document-to-markdown-1)  Document-to-Markdown 
+
+NuExtract can also convert document images into clean Markdown. Output will be markdown for text (headers etc), html for tables, latex for mat and `<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/> `
+
+Markdown example:
+
+```
+<figure data-type="image" data-id="img_1">
+  <img src="/numind/NuExtract3/resolve/main/img_1.png" alt="Logo of Mobilier 2000 with contact information: Tél.: (418) 275-4232, 1654, boul. Marcotte, Roberval (Qc) G8H 2P2"/>
+</figure>
+
+# COMMANDE
+**NUMÉRO 72259**
+
+1
+
+**Vendu à**
+TREMBLAY ERIC
+ERIC TREMBLAY
+348 BOUL. DE L'ANSE
+ROBERVAL
+G8H 1Y9
+
+**Livré à**
+TREMBLAY ERIC
+ERIC TREMBLAY
+348 BOUL. DE L'ANSE
+ROBERVAL
+G8H 1Y9
+
+<table>
+  <thead>
+    <tr>
+      <th># CLIENT</th>
+      <th>EXPÉDITEUR</th>
+      <th>TERME DE CRÉDIT</th>
+      <th>DATE</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>2753133</td>
+      <td>Notre camion</td>
+      <td>à la livraison</td>
+      <td>22/06/2023</td>
+    </tr>
+  </tbody>
+</table>
+
+<table>
+  <thead>
+    <tr>
+      <th>NOM DU VENDEUR</th>
+      <th>VOTRE ÉCONOMIE !</th>
+      <th># COMMANDE</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>Éric</td>
+      <td>0.00</td>
+      <td></td>
+    </tr>
+  </tbody>
+</table>
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#reasoning-and-non-reasoning-modes](https://huggingface.co/numind/NuExtract3/blob/main/README.md#reasoning-and-non-reasoning-modes)  Reasoning and non-reasoning modes 
+
+NuExtract supports both reasoning and non-reasoning inference.
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#non-thinking-mode](https://huggingface.co/numind/NuExtract3/blob/main/README.md#non-thinking-mode)  Non-thinking mode 
+
+Use this for fast and deterministic extraction or Markdown conversion.
+
+```
+enable_thinking = False
+temperature = 0.2
+```
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#thinking-mode](https://huggingface.co/numind/NuExtract3/blob/main/README.md#thinking-mode)  Thinking mode 
+
+Use this for difficult documents, complex layouts, ambiguous fields, or cases where the document structure requires additional reasoning.
+
+```
+enable_thinking = True
+temperature = 0.6
+```
+
+For production extraction workloads, we recommend starting with non-reasoning mode  and enabling reasoning only for difficult examples.
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-deployment](https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-deployment)  vLLM deployment 
+
+NuExtract can be served with vLLM using an OpenAI-compatible API.
+
+```
+vllm serve numind/NuExtract3 \
+  --trust-remote-code \
+  --limit-mm-per-prompt '{"image": 99, "video": 0}' \
+  --chat-template-content-format openai \
+  --generation-config vllm \
+  --max-model-len 131072 \
+  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#multi-token-prediction](https://huggingface.co/numind/NuExtract3/blob/main/README.md#multi-token-prediction)  Multi Token Prediction 
+
+ The deployment commands above enable Multi Token Prediction (MTP) through vLLM speculative decoding: 
+
+```
+--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+
+MTP can improve decoding throughput without changing the OpenAI-compatible request payload. You can tune `num_speculative_tokens` for your hardware and workload, or remove `--speculative-config` if your vLLM version or environment does not support this speculative decoding method.
+
+If you encounter memory issues, reduce the maximum model length and the maximum number of images:
+
+```
+vllm serve numind/NuExtract-3 \
+  --trust-remote-code \
+  --limit-mm-per-prompt '{"image": 6, "video": 0}' \
+  --chat-template-content-format openai \
+  --generation-config vllm \
+  --max-model-len 16384 \
+  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-structured-extraction-text](https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-structured-extraction-text)  vLLM inference: structured extraction: text 
+
+```
+import json
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "currency": ["USD", "EUR", "GBP", "JPY", "Other"],
+    "items": [
+        {
+            "name": "verbatim-string",
+            "price": "number"
+        }
+    ]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "Yesterday I bought apples and coffee at Trader Joe's for a total of $12.40."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template),
+            "instructions": "Specify the time for the `date` entry only if it is present, otherwise only output the date component.",
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```
+{
+  "store": "Trader Joe's",
+  "date": null,
+  "total": 12.40,
+  "currency": "USD",
+  "items": [
+    {
+      "name": "apples",
+      "price": null
+    },
+    {
+      "name": "coffee",
+      "price": null
+    }
+  ]
+}
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-structured-extraction-image](https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-structured-extraction-image)  vLLM inference: structured extraction: image 
+
+```
+import json
+import base64
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def encode_image(image_path):
+    with open(image_path, "rb") as image_file:
+        return base64.b64encode(image_file.read()).decode("utf-8")
+
+image_base64 = encode_image("receipt.png")
+data_url = f"data:image/png;base64,{image_base64}"
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "payment_method": "verbatim-string"
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```
+{
+  "store": "Trader Joe's",
+  "date": "2025-04-12",
+  "total": 42.85,
+  "payment_method": "Visa"
+}
+```
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#multiple-page-pdf](https://huggingface.co/numind/NuExtract3/blob/main/README.md#multiple-page-pdf)  Multiple page PDF 
+
+ You can render a PDF to one PNG image per page with PyMuPDF, then pass the images to vLLM in page order. 
+
+```
+import base64
+import json
+
+import fitz  # pip install pymupdf
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def pdf_to_png_data_urls(pdf_path, dpi=170):
+    data_urls = []
+
+    with fitz.open(pdf_path) as doc:
+        for page in doc:
+            pix = page.get_pixmap(dpi=dpi, alpha=False)
+            png_bytes = pix.tobytes("png")
+            png_base64 = base64.b64encode(png_bytes).decode("utf-8")
+            data_urls.append(f"data:image/png;base64,{png_base64}")
+
+    return data_urls
+
+data_urls = pdf_to_png_data_urls("invoice.pdf", dpi=170)
+
+template = {
+    "invoice_number": "verbatim-string",
+    "invoice_date": "date",
+    "total": "number",
+    "currency": "currency",
+    "line_items": [
+        {
+            "description": "verbatim-string",
+            "quantity": "number",
+            "unit_price": "number",
+            "total": "number"
+        }
+    ]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+                for data_url in data_urls
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-document-to-markdown](https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-document-to-markdown)  vLLM inference: document-to-Markdown 
+
+For Markdown OCR, use `mode="markdown"` or `mode="content"` without a template.
+
+```
+import base64
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+def encode_image(image_path):
+    with open(image_path, "rb") as image_file:
+        return base64.b64encode(image_file.read()).decode("utf-8")
+
+image_base64 = encode_image("document.png")
+data_url = f"data:image/png;base64,{image_base64}"
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=1,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "markdown",
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-reasoning-mode](https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-reasoning-mode)  vLLM inference: reasoning mode 
+
+ Reasoning can be enabled for harder structured extraction or Markdown tasks. 
+
+```
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=1,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": data_url}
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "markdown",
+            "enable_thinking": True
+        }
+    }
+)
+
+result = response.choices[0].message.content
+
+if "</think>" in result:
+    reasoning, answer = [part.strip() for part in result.split("</think>")]
+else:
+    reasoning, answer = None, result
+
+print(answer)
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#in-context-examples-for-extraction](https://huggingface.co/numind/NuExtract3/blob/main/README.md#in-context-examples-for-extraction)  In-context examples for extraction 
+
+ NuExtract supports in-context examples for structured extraction. 
+
+Examples are especially useful when the desired formatting is ambiguous or when the schema requires task-specific conventions. Examples can be provided by using `developer` messages, for which all items of the contents except the last one are the input, and the last one is the expected output.
+
+```
+import json
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+template = {
+    "names": ["string"]
+}
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "developer",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "Stephen is the manager at Susan's store.",
+                },
+                {
+                    "type": "text",
+                    "text": "{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}",
+                }
+            ],
+        },
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "John went to the restaurant with Mary. James went to the cinema."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "template": json.dumps(template, indent=4),
+            "enable_thinking": False
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```
+{
+  "names": ["-JOHN-", "-MARY-", "-JAMES-"]
+}
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-template-generation](https://huggingface.co/numind/NuExtract3/blob/main/README.md#vllm-inference-template-generation)  vLLM inference: template generation 
+
+NuExtract can generate an extraction template from a natural language description.
+
+```
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+
+response = client.chat.completions.create(
+    model="numind/NuExtract3",
+    temperature=0.2,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "I want to extract the key details from a rental contract."
+                }
+            ],
+        }
+    ],
+    extra_body={
+        "chat_template_kwargs": {
+            "mode": "template-generation"
+        }
+    }
+)
+
+print(response.choices[0].message.content)
+```
+
+Example output:
+
+```
+{
+  "contract_title": "verbatim-string",
+  "landlord": "verbatim-string",
+  "tenant": "verbatim-string",
+  "property_address": "verbatim-string",
+  "start_date": "date-time",
+  "end_date": "date-time",
+  "monthly_rent": "number",
+  "currency": "verbatim-string",
+  "deposit": "number",
+  "signatories": ["verbatim-string"]
+}
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#curl-examples](https://huggingface.co/numind/NuExtract3/blob/main/README.md#curl-examples)  Curl examples 
+
+The following examples assume that vLLM is running locally on port 8000. They use `jq` to build valid JSON request bodies without manually escaping the image data or template string.
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#single-image-structured-extraction](https://huggingface.co/numind/NuExtract3/blob/main/README.md#single-image-structured-extraction)  Single image structured extraction 
+
+```
+API_KEY="EMPTY"
+IMAGE_BASE64_FILE=$(mktemp)
+REQUEST_BODY_FILE=$(mktemp)
+
+base64 < receipt.png | tr -d '\n' > "$IMAGE_BASE64_FILE"
+
+TEMPLATE=$(cat <<'JSON'
+{
+  "store": "verbatim-string",
+  "date": "date-time",
+  "total": "number",
+  "payment_method": "verbatim-string"
+}
+JSON
+)
+
+jq -n \
+  --rawfile image_base64 "$IMAGE_BASE64_FILE" \
+  --arg template "$TEMPLATE" \
+  '{
+    model: "numind/NuExtract3",
+    temperature: 0.6,
+    messages: [
+      {
+        role: "user",
+        content: [
+          {
+            type: "image_url",
+            image_url: {url: ("data:image/png;base64," + $image_base64)}
+          }
+        ]
+      }
+    ],
+    chat_template_kwargs: {
+      template: $template,
+      enable_thinking: false
+    }
+  }' > "$REQUEST_BODY_FILE"
+
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $API_KEY" \
+  --data-binary "@$REQUEST_BODY_FILE"
+
+rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"
+```
+
+### [https://huggingface.co/numind/NuExtract3/blob/main/README.md#single-image-content-extraction](https://huggingface.co/numind/NuExtract3/blob/main/README.md#single-image-content-extraction)  Single image content extraction 
+
+```
+API_KEY="EMPTY"
+IMAGE_BASE64_FILE=$(mktemp)
+REQUEST_BODY_FILE=$(mktemp)
+
+base64 < document.png | tr -d '\n' > "$IMAGE_BASE64_FILE"
+
+jq -n \
+  --rawfile image_base64 "$IMAGE_BASE64_FILE" \
+  '{
+    model: "numind/NuExtract3",
+    temperature: 0.6,
+    messages: [
+      {
+        role: "user",
+        content: [
+          {
+            type: "image_url",
+            image_url: {url: ("data:image/png;base64," + $image_base64)}
+          }
+        ]
+      }
+    ],
+    chat_template_kwargs: {
+      mode: "content",
+      enable_thinking: false
+    }
+  }' > "$REQUEST_BODY_FILE"
+
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $API_KEY" \
+  --data-binary "@$REQUEST_BODY_FILE"
+
+rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"
+```
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#transformers-example](https://huggingface.co/numind/NuExtract3/blob/main/README.md#transformers-example)  Transformers example 
+
+ You can also run NuExtract directly with `transformers`. The same `template`, `mode`, and `enable_thinking` options are passed to `processor.apply_chat_template`. 
+
+```
+import json
+
+import torch
+from PIL import Image
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+model_id = "numind/NuExtract3"
+
+processor = AutoProcessor.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+).eval()
+
+def run_nuextract(messages, **chat_template_kwargs):
+    inputs = processor.apply_chat_template(
+        messages,
+        add_generation_prompt=True,
+        tokenize=True,
+        return_dict=True,
+        return_tensors="pt",
+        **chat_template_kwargs,
+    ).to(model.device)
+
+    with torch.inference_mode():
+        generated_ids = model.generate(
+            **inputs,
+            max_new_tokens=4096,
+            do_sample=False,
+        )
+
+    generated_ids = generated_ids[:, inputs.input_ids.shape[1]:]
+    return processor.batch_decode(
+        generated_ids,
+        skip_special_tokens=True,
+        clean_up_tokenization_spaces=False,
+    )[0].strip()
+
+# Single image structured extraction
+receipt_image = Image.open("receipt.png").convert("RGB")
+receipt_messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": receipt_image,
+            }
+        ],
+    }
+]
+
+template = {
+    "store": "verbatim-string",
+    "date": "date-time",
+    "total": "number",
+    "payment_method": "verbatim-string"
+}
+
+structured_output = run_nuextract(
+    receipt_messages,
+    template=json.dumps(template, indent=4),
+    enable_thinking=False,
+)
+print(structured_output)
+
+# Single image content extraction
+document_image = Image.open("document.png").convert("RGB")
+document_messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": document_image,
+            }
+        ],
+    }
+]
+
+content_output = run_nuextract(
+    document_messages,
+    mode="content",
+    enable_thinking=False,
+)
+print(content_output)
+```
+
+Special thanks to the Lambda.ai team for the compute that made this project a success.
+
+## [https://huggingface.co/numind/NuExtract3/blob/main/README.md#citation](https://huggingface.co/numind/NuExtract3/blob/main/README.md#citation)  Citation 
+
+If you use NuExtract, please cite NuMind and link to the model page.
+
+```
+@misc{nuextract3,
+  title = {NuExtract3},
+  author = {NuMind},
+  year = {2026},
+  url = {https://nuextract.ai/}
+}
+```

AGENTS.md

@@ -0,0 +1,9 @@
+# Repository implementation policy
+
+- This repository has one supported execution path: the explicitly requested path.
+- Never add error handling, `try`/`except`, recovery logic, fallback values, fallback providers, compatibility shims, silent substitution, or degraded modes.
+- Never catch, suppress, translate, wrap, retry, or replace failures.
+- Never add validation branches whose purpose is to produce a custom error path.
+- Let failures surface naturally at their source.
+- Do not preserve legacy behavior unless the user explicitly requests it.
+- When a requested implementation cannot run as specified, report the blocker; do not create an alternate implementation.

app/config.py

@@ -1,2 +0,0 @@
-import os
-TRANSFORMERS_MODEL = os.environ.get("TRANSFORMERS_MODEL", "mradermacher/NuMarkdown-8B-Thinking-i1-GGUF")

app/nuextract.py

@@ -1,7 +1,71 @@
-from transformers import AutoModel
-from app.config import TRANSFORMERS_MODEL
-
-
-
-model = AutoModel.from_pretrained(TRANSFORMERS_MODEL, dtype="auto")
-
+from pathlib import Path
+
+import torch
+from PIL import Image
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+model_id = "numind/NuExtract3"
+
+processor = AutoProcessor.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+).eval()
+
+
+def run_nuextract(messages, **chat_template_kwargs):
+    inputs = processor.apply_chat_template(
+        messages,
+        add_generation_prompt=True,
+        tokenize=True,
+        return_dict=True,
+        return_tensors="pt",
+        **chat_template_kwargs,
+    ).to(model.device)
+
+    with torch.inference_mode():
+        generated_ids = model.generate(
+            **inputs,
+            max_new_tokens=4096,
+            do_sample=False,
+        )
+
+    generated_ids = generated_ids[:, inputs.input_ids.shape[1]:]
+    return processor.batch_decode(
+        generated_ids,
+        skip_special_tokens=True,
+        clean_up_tokenization_spaces=False,
+    )[0].strip()
+
+
+sample_path = (
+    Path(__file__).parents[1]
+    / "data"
+    / "samples"
+    / "Screenshot 2026-07-18 171416.png"
+)
+sample_image = Image.open(sample_path).convert("RGB")
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": sample_image,
+            }
+        ],
+    }
+]
+
+output = run_nuextract(
+    messages,
+    mode="content",
+    enable_thinking=False,
+)
+print(output)

notebooks/NuExtract3.ipynb

@@ -0,0 +1,66 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "38837f4a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "282526cb378b487a8b6743dcdbaaf51d",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "NuExtract3-Q4_K_M.gguf:   0%|          | 0.00/2.71G [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "ename": "ValueError",
+     "evalue": "Could not load model numind/NuExtract3-GGUF with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForImageTextToText'>, <class 'transformers.models.qwen3_5.modeling_qwen3_5.Qwen3_5ForConditionalGeneration'>). See the original errors:\n\nwhile loading with AutoModelForImageTextToText, an error is thrown:\nTraceback (most recent call last):\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py\", line 240, in load_model\n    model = model_class.from_pretrained(model, **kwargs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\models\\auto\\auto_factory.py\", line 336, in from_pretrained\n    config, kwargs = AutoConfig.from_pretrained(\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\models\\auto\\configuration_auto.py\", line 376, in from_pretrained\n    config_dict, unused_kwargs = PreTrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)\n                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\configuration_utils.py\", line 721, in get_config_dict\n    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\configuration_utils.py\", line 807, in _get_config_dict\n    config_dict = load_gguf_checkpoint(resolved_config_file, return_tensors=False)[\"config\"]\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\modeling_gguf_pytorch_utils.py\", line 590, in load_gguf_checkpoint\n    if is_gguf_available() and is_torch_available():\n       ^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\utils\\import_utils.py\", line 1207, in is_gguf_available\n    return is_available and version.parse(gguf_version) >= version.parse(min_version)\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\packaging\\version.py\", line 56, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\packaging\\version.py\", line 202, in __init__\n    raise InvalidVersion(f\"Invalid version: '{version}'\")\npackaging.version.InvalidVersion: Invalid version: 'N/A'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py\", line 256, in load_model\n    model = model_class.from_pretrained(model, **fp32_kwargs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\models\\auto\\auto_factory.py\", line 336, in from_pretrained\n    config, kwargs = AutoConfig.from_pretrained(\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\models\\auto\\configuration_auto.py\", line 376, in from_pretrained\n    config_dict, unused_kwargs = PreTrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)\n                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\configuration_utils.py\", line 721, in get_config_dict\n    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\configuration_utils.py\", line 807, in _get_config_dict\n    config_dict = load_gguf_checkpoint(resolved_config_file, return_tensors=False)[\"config\"]\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\modeling_gguf_pytorch_utils.py\", line 590, in load_gguf_checkpoint\n    if is_gguf_available() and is_torch_available():\n       ^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\utils\\import_utils.py\", line 1207, in is_gguf_available\n    return is_available and version.parse(gguf_version) >= version.parse(min_version)\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\packaging\\version.py\", line 56, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\packaging\\version.py\", line 202, in __init__\n    raise InvalidVersion(f\"Invalid version: '{version}'\")\npackaging.version.InvalidVersion: Invalid version: 'N/A'\n\nwhile loading with Qwen3_5ForConditionalGeneration, an error is thrown:\nTraceback (most recent call last):\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py\", line 240, in load_model\n    model = model_class.from_pretrained(model, **kwargs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\modeling_utils.py\", line 4281, in from_pretrained\n    raise ValueError(\"accelerate is required when loading a GGUF file `pip install accelerate`.\")\nValueError: accelerate is required when loading a GGUF file `pip install accelerate`.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py\", line 256, in load_model\n    model = model_class.from_pretrained(model, **fp32_kwargs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\modeling_utils.py\", line 4281, in from_pretrained\n    raise ValueError(\"accelerate is required when loading a GGUF file `pip install accelerate`.\")\nValueError: accelerate is required when loading a GGUF file `pip install accelerate`.\n\n\n",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
+      "\u001b[31mValueError\u001b[39m                                Traceback (most recent call last)",
+      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[3]\u001b[39m\u001b[32m, line 6\u001b[39m\n\u001b[32m      2\u001b[39m \u001b[38;5;28;01mfrom\u001b[39;00m transformers \u001b[38;5;28;01mimport\u001b[39;00m AutoConfig, AutoProcessor, pipeline\n\u001b[32m      3\u001b[39m \n\u001b[32m      4\u001b[39m config = AutoConfig.from_pretrained(\u001b[33m\"numind/NuExtract3\"\u001b[39m)\n\u001b[32m      5\u001b[39m processor = AutoProcessor.from_pretrained(\u001b[33m\"numind/NuExtract3\"\u001b[39m)\n\u001b[32m----> \u001b[39m\u001b[32m6\u001b[39m pipe = pipeline(\u001b[33m\"image-text-to-text\"\u001b[39m, model=\u001b[33m\"numind/NuExtract3-GGUF\"\u001b[39m, config=config, processor=processor, device_map=\u001b[33m\"auto\"\u001b[39m, model_kwargs={\u001b[33m\"gguf_file\"\u001b[39m: \u001b[33m\"NuExtract3-Q4_K_M.gguf\"\u001b[39m})\n\u001b[32m      7\u001b[39m image = str(next((Path.cwd() / \u001b[33m\"data\"\u001b[39m / \u001b[33m\"samples\"\u001b[39m).glob(\u001b[33m\"*.png\"\u001b[39m)))\n\u001b[32m      8\u001b[39m messages = [{\"role\": \"user\", \"content\": [\n\u001b[32m      9\u001b[39m     {\u001b[33m\"type\"\u001b[39m: \u001b[33m\"image\"\u001b[39m, \u001b[33m\"url\"\u001b[39m: image},\n",
+      "\u001b[36mFile \u001b[39m\u001b[32ms:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\__init__.py:1033\u001b[39m, in \u001b[36mpipeline\u001b[39m\u001b[34m(task, model, config, tokenizer, feature_extractor, image_processor, video_processor, processor, revision, use_fast, token, device, device_map, dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)\u001b[39m\n\u001b[32m   1031\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(model, \u001b[38;5;28mstr\u001b[39m):\n\u001b[32m   1032\u001b[39m     model_classes = targeted_task[\u001b[33m\"\u001b[39m\u001b[33mpt\u001b[39m\u001b[33m\"\u001b[39m]\n\u001b[32m-> \u001b[39m\u001b[32m1033\u001b[39m     model = \u001b[30;43mload_model\u001b[39;49m\u001b[30;43m(\u001b[39;49m\n\u001b[32m   1034\u001b[39m \u001b[30;43m        \u001b[39;49m\u001b[30;43madapter_path\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43;01mif\u001b[39;49;00m\u001b[30;43m \u001b[39;49m\u001b[30;43madapter_path\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43;01mis\u001b[39;49;00m\u001b[30;43m \u001b[39;49m\u001b[30;43;01mnot\u001b[39;49;00m\u001b[30;43m \u001b[39;49m\u001b[30;43;01mNone\u001b[39;49;00m\u001b[30;43m \u001b[39;49m\u001b[30;43;01melse\u001b[39;49;00m\u001b[30;43m \u001b[39;49m\u001b[30;43mmodel\u001b[39;49m\u001b[30;43m,\u001b[39;49m\n\u001b[32m   1035\u001b[39m \u001b[30;43m        \u001b[39;49m\u001b[30;43mmodel_classes\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mmodel_classes\u001b[39;49m\u001b[30;43m,\u001b[39;49m\n\u001b[32m   1036\u001b[39m \u001b[30;43m        \u001b[39;49m\u001b[30;43mconfig\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mconfig\u001b[39;49m\u001b[30;43m,\u001b[39;49m\n\u001b[32m   1037\u001b[39m \u001b[30;43m        \u001b[39;49m\u001b[30;43mtask\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mtask\u001b[39;49m\u001b[30;43m,\u001b[39;49m\n\u001b[32m   1038\u001b[39m \u001b[30;43m        \u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43mhub_kwargs\u001b[39;49m\u001b[30;43m,\u001b[39;49m\n\u001b[32m   1039\u001b[39m \u001b[30;43m        \u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43mmodel_kwargs\u001b[39;49m\u001b[30;43m,\u001b[39;49m\n\u001b[32m   1040\u001b[39m \u001b[30;43m    \u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m   1042\u001b[39m hub_kwargs[\u001b[33m\"\u001b[39m\u001b[33m_commit_hash\u001b[39m\u001b[33m\"\u001b[39m] = model.config._commit_hash\n\u001b[32m   1044\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m pipeline_class \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
+      "\u001b[36mFile \u001b[39m\u001b[32ms:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py:276\u001b[39m, in \u001b[36mload_model\u001b[39m\u001b[34m(model, config, model_classes, task, **model_kwargs)\u001b[39m\n\u001b[32m    274\u001b[39m         \u001b[38;5;28;01mfor\u001b[39;00m class_name, trace \u001b[38;5;129;01min\u001b[39;00m all_traceback.items():\n\u001b[32m    275\u001b[39m             error += \u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mwhile loading with \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mclass_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m, an error is thrown:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;132;01m{\u001b[39;00mtrace\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[33m\"\u001b[39m\n\u001b[32m--> \u001b[39m\u001b[32m276\u001b[39m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[32m    277\u001b[39m             \u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mCould not load model \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mmodel\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m with any of the following classes: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mclass_tuple\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m. See the original errors:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;132;01m{\u001b[39;00merror\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[33m\"\u001b[39m\n\u001b[32m    278\u001b[39m         )\n\u001b[32m    280\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m model\n",
+      "\u001b[31mValueError\u001b[39m: Could not load model numind/NuExtract3-GGUF with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForImageTextToText'>, <class 'transformers.models.qwen3_5.modeling_qwen3_5.Qwen3_5ForConditionalGeneration'>). See the original errors:\n\nwhile loading with AutoModelForImageTextToText, an error is thrown:\nTraceback (most recent call last):\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py\", line 240, in load_model\n    model = model_class.from_pretrained(model, **kwargs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\models\\auto\\auto_factory.py\", line 336, in from_pretrained\n    config, kwargs = AutoConfig.from_pretrained(\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\models\\auto\\configuration_auto.py\", line 376, in from_pretrained\n    config_dict, unused_kwargs = PreTrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)\n                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\configuration_utils.py\", line 721, in get_config_dict\n    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\configuration_utils.py\", line 807, in _get_config_dict\n    config_dict = load_gguf_checkpoint(resolved_config_file, return_tensors=False)[\"config\"]\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\modeling_gguf_pytorch_utils.py\", line 590, in load_gguf_checkpoint\n    if is_gguf_available() and is_torch_available():\n       ^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\utils\\import_utils.py\", line 1207, in is_gguf_available\n    return is_available and version.parse(gguf_version) >= version.parse(min_version)\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\packaging\\version.py\", line 56, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\packaging\\version.py\", line 202, in __init__\n    raise InvalidVersion(f\"Invalid version: '{version}'\")\npackaging.version.InvalidVersion: Invalid version: 'N/A'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py\", line 256, in load_model\n    model = model_class.from_pretrained(model, **fp32_kwargs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\models\\auto\\auto_factory.py\", line 336, in from_pretrained\n    config, kwargs = AutoConfig.from_pretrained(\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\models\\auto\\configuration_auto.py\", line 376, in from_pretrained\n    config_dict, unused_kwargs = PreTrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)\n                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\configuration_utils.py\", line 721, in get_config_dict\n    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\configuration_utils.py\", line 807, in _get_config_dict\n    config_dict = load_gguf_checkpoint(resolved_config_file, return_tensors=False)[\"config\"]\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\modeling_gguf_pytorch_utils.py\", line 590, in load_gguf_checkpoint\n    if is_gguf_available() and is_torch_available():\n       ^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\utils\\import_utils.py\", line 1207, in is_gguf_available\n    return is_available and version.parse(gguf_version) >= version.parse(min_version)\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\packaging\\version.py\", line 56, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\packaging\\version.py\", line 202, in __init__\n    raise InvalidVersion(f\"Invalid version: '{version}'\")\npackaging.version.InvalidVersion: Invalid version: 'N/A'\n\nwhile loading with Qwen3_5ForConditionalGeneration, an error is thrown:\nTraceback (most recent call last):\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py\", line 240, in load_model\n    model = model_class.from_pretrained(model, **kwargs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\modeling_utils.py\", line 4281, in from_pretrained\n    raise ValueError(\"accelerate is required when loading a GGUF file `pip install accelerate`.\")\nValueError: accelerate is required when loading a GGUF file `pip install accelerate`.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\pipelines\\base.py\", line 256, in load_model\n    model = model_class.from_pretrained(model, **fp32_kwargs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\transformers\\modeling_utils.py\", line 4281, in from_pretrained\n    raise ValueError(\"accelerate is required when loading a GGUF file `pip install accelerate`.\")\nValueError: accelerate is required when loading a GGUF file `pip install accelerate`.\n\n\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "from transformers import AutoConfig, AutoProcessor, pipeline\n",
+    "\n",
+    "config = AutoConfig.from_pretrained(\"numind/NuExtract3\")\n",
+    "processor = AutoProcessor.from_pretrained(\"numind/NuExtract3\")\n",
+    "pipe = pipeline(\"image-text-to-text\", model=\"numind/NuExtract3-GGUF\", config=config, processor=processor, device_map=\"auto\", model_kwargs={\"gguf_file\": \"NuExtract3-Q4_K_M.gguf\"})\n",
+    "image = str(next((Path.cwd() / \"data\" / \"samples\").glob(\"*.png\")))\n",
+    "messages = [{\"role\": \"user\", \"content\": [\n",
+    "    {\"type\": \"image\", \"url\": image},\n",
+    "    {\"type\": \"text\", \"text\": \"Extract all document content as structured JSON.\"},\n",
+    "]}]\n",
+    "pipe(text=messages, max_new_tokens=4096)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/nuextract-2.0_inference.ipynb

@@ -1,876 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "cbe8b1f6-ec66-4fbc-a9f8-774acd8a255a",
-   "metadata": {},
-   "source": [
-    "# NuExtract 2.0 Inference\n",
-    "\n",
-    "In this notebook we will provide examples of how to use the NuExtract 2.0 models for inference.\n",
-    "\n",
-    "First, let's load a model."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a561e38e-58e5-4f7e-80ee-eabc2d05e49a",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n"
-     ]
-    }
-   ],
-   "source": [
-    "import torch\n",
-    "from transformers import AutoModelForImageTextToText, AutoProcessor\n",
-    "model_name = \"numind/NuExtract-2.0-2B\"\n",
-    "# model_name = \"numind/NuExtract-2.0-8B\"\n",
-    "\n",
-    "processor = AutoProcessor.from_pretrained(model_name,\n",
-    "                                          trust_remote_code=True,\n",
-    "                                          padding_side='left',\n",
-    "                                          use_fast=True)\n",
-    "model = AutoModelForImageTextToText.from_pretrained(model_name,\n",
-    "                                               trust_remote_code=True,\n",
-    "                                               dtype=torch.bfloat16,\n",
-    "                                               attn_implementation=\"flash_attention_2\",\n",
-    "                                               device_map=\"auto\")\n",
-    "\n",
-    "# we choose greedy sampling here, which works well for most information extraction tasks\n",
-    "generation_config = {\n",
-    "    \"do_sample\": False,\n",
-    "    \"num_beams\": 1,\n",
-    "    \"max_new_tokens\": 2048,\n",
-    "    \"temperature\": None,\n",
-    "    \"top_p\": None,\n",
-    "    \"top_k\": None\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d739f582-16ad-4e52-b012-429757945a84",
-   "metadata": {},
-   "source": [
-    "## Preparing Model Inputs\n",
-    "\n",
-    "Before using the model, we also need to make sure our prompts are properly formatted to work with NuExtract. NuExtract expects all input information to come as a single user chat prompt, formatted as follows:\n",
-    "\n",
-    "```python\n",
-    "f\"\"\"\n",
-    "# Template:\n",
-    "{template}\n",
-    "# Context:\n",
-    "{text}\n",
-    "\"\"\"\n",
-    "```\n",
-    "and if in-context examples are provided:\n",
-    "```python\n",
-    "f\"\"\"\n",
-    "# Template:\n",
-    "{template}\n",
-    "# Examples\n",
-    "## Input:\n",
-    "{input1}\n",
-    "## Output:\n",
-    "{output1}\n",
-    "## Input:\n",
-    "{input2}\n",
-    "## Output:\n",
-    "{output2}\n",
-    "# Context:\n",
-    "{text}\n",
-    "\"\"\"\n",
-    "```\n",
-    "\n",
-    "If you are working with image inputs, you should use image placeholders for `text`, `input1`, etc. Later, we will inject tokens representing the actual image content in the location of these placeholders.\n",
-    "\n",
-    "The following function can make this formatting more convenient for us (the model can actually do this for us internally, but we will use this function initially to illustrate how things work). "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "58b017a4-4802-4afe-b2b4-8f034f572c24",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "def construct_messages(document, template, examples=None, image_placeholder=\"<|vision_start|><|image_pad|><|vision_end|>\"):\n",
-    "    \"\"\"\n",
-    "    Construct the individual NuExtract message texts, prior to chat template formatting.\n",
-    "    \"\"\"\n",
-    "    images = []\n",
-    "    # add few-shot examples if needed\n",
-    "    if examples is not None and len(examples) > 0:\n",
-    "        icl = \"# Examples:\\n\"\n",
-    "        for row in examples:\n",
-    "            example_input = row['input']\n",
-    "\n",
-    "            if not isinstance(row['input'], str):\n",
-    "                example_input = image_placeholder\n",
-    "                images.append(row['input'])\n",
-    "\n",
-    "            icl += f\"## Input:\\n{example_input}\\n## Output:\\n{row['output']}\\n\"\n",
-    "    else:\n",
-    "        icl = \"\"\n",
-    "\n",
-    "    # if input document is an image, set text to an image placeholder\n",
-    "    text = document\n",
-    "    if not isinstance(document, str):\n",
-    "        text = image_placeholder\n",
-    "        images.append(document)\n",
-    "    text = f\"\"\"# Template:\\n{template}\\n{icl}# Context:\\n{text}\"\"\"\n",
-    "\n",
-    "    messages = [\n",
-    "        {\n",
-    "            \"role\": \"system\",\n",
-    "            \"content\": \"You are NuExtract, an information extraction tool created by NuMind.\"\n",
-    "        },\n",
-    "        {\n",
-    "            \"role\": \"user\",\n",
-    "            \"content\": [{\"type\": \"text\", \"text\": text}] + images,\n",
-    "        }\n",
-    "    ]\n",
-    "    return messages"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e918d64e-bc31-4d68-8ef5-2d485b5e5745",
-   "metadata": {},
-   "source": [
-    "## Inference\n",
-    "### Basic Example\n",
-    "\n",
-    "Now we are ready to run the model!\n",
-    "\n",
-    "Let's start with a basic text-only example, where we want to extract peoples' names from a short text."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b1cad1ec-4753-4064-aca0-c89fec4b5bfd",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from qwen_vl_utils import process_vision_info\n",
-    "\n",
-    "template = \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\"\n",
-    "document = \"John went to the restaurant with Mary. James went to the cinema.\"\n",
-    "\n",
-    "# prepare the user message content\n",
-    "messages = construct_messages(document, template)\n",
-    "text = processor.apply_chat_template(\n",
-    "    messages, tokenize=False, add_generation_prompt=True,\n",
-    ")\n",
-    "\n",
-    "image_inputs = process_vision_info(messages)[0]\n",
-    "inputs = processor(\n",
-    "    text=[text],\n",
-    "    images=image_inputs,\n",
-    "    padding=True,\n",
-    "    return_tensors=\"pt\",\n",
-    ").to(\"cuda\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6c0e4f92-94dd-4b37-8772-591246f9b71a",
-   "metadata": {},
-   "source": [
-    "Our NuExtract message is now formatted in standard chat template formatting; the tokenized version (`inputs`) will be given directly to the model."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "42bd5add-b48a-44e4-9843-2f187e768678",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "<|im_start|>system\n",
-      "You are NuExtract, an information extraction tool created by NuMind.<|im_end|>\n",
-      "<|im_start|>user\n",
-      "# Template:\n",
-      "{\"names\": [\"verbatim-string\"]}\n",
-      "# Context:\n",
-      "John went to the restaurant with Mary. James went to the cinema.<|im_end|>\n",
-      "<|im_start|>assistant\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(text)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "df0979a7-ea46-4d25-a838-baeaeeca6ee5",
-   "metadata": {},
-   "source": [
-    "The other `image_inputs` are empty in this case because this is a text-only example."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e3493013-313f-4318-a813-d492732bb0c1",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "None\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(image_inputs)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0e300775-48a9-42fb-9c92-3e1ceb530101",
-   "metadata": {},
-   "source": [
-    "Now let's actually run the model."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "ed374bf3-1af1-4a3e-a9eb-9e2e7ca113b8",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "['{\"names\": [\"John\", \"Mary\", \"James\"]}']\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Inference: Generation of the output\n",
-    "generated_ids = model.generate(\n",
-    "    **inputs,\n",
-    "    **generation_config\n",
-    ")\n",
-    "generated_ids_trimmed = [\n",
-    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
-    "]\n",
-    "output_text = processor.batch_decode(\n",
-    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
-    ")\n",
-    "print(output_text)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "21d37d33-85ad-421c-a014-523ddbec5a2f",
-   "metadata": {},
-   "source": [
-    "Alternatively, you can directly provide the template and in-context examples to `.apply_chat_template()`, rather then manually preparing the prompt via `construct_messages()`. This is a bit more convenient and so we will use this from now on."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "27fecb60-b0eb-46bc-ba2d-ef6a32d04f0c",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "<|im_start|>system\n",
-      "You are NuExtract, an information extraction tool created by NuMind.<|im_end|>\n",
-      "<|im_start|>user\n",
-      "# Template:\n",
-      "{\"names\": [\"verbatim-string\"]}\n",
-      "# Context:\n",
-      "John went to the restaurant with Mary. James went to the cinema.<|im_end|>\n",
-      "<|im_start|>assistant\n",
-      "\n",
-      "['{\"names\": [\"John\", \"Mary\", \"James\"]}']\n"
-     ]
-    }
-   ],
-   "source": [
-    "template = \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\"\n",
-    "document = \"John went to the restaurant with Mary. James went to the cinema.\"\n",
-    "\n",
-    "# prepare the user message content\n",
-    "messages = [{\"role\": \"user\", \"content\": document}]\n",
-    "text = processor.tokenizer.apply_chat_template(\n",
-    "    messages,\n",
-    "    template=template, # template is specified here\n",
-    "    tokenize=False,\n",
-    "    add_generation_prompt=True,\n",
-    ")\n",
-    "\n",
-    "print(text)\n",
-    "\n",
-    "image_inputs = process_vision_info(messages)[0]\n",
-    "inputs = processor(\n",
-    "    text=[text],\n",
-    "    images=image_inputs,\n",
-    "    padding=True,\n",
-    "    return_tensors=\"pt\",\n",
-    ").to(\"cuda\")\n",
-    "\n",
-    "# Inference: Generation of the output\n",
-    "generated_ids = model.generate(\n",
-    "    **inputs,\n",
-    "    **generation_config\n",
-    ")\n",
-    "generated_ids_trimmed = [\n",
-    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
-    "]\n",
-    "output_text = processor.batch_decode(\n",
-    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
-    ")\n",
-    "\n",
-    "print(output_text)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c18002f2-447a-45c2-90f6-0ff5a63c5dc2",
-   "metadata": {},
-   "source": [
-    "### In-Context Examples\n",
-    "\n",
-    "Sometimes the model might not perform as well as we want because our task is challenging or involves some degree of ambiguity. Alternatively, we may want the model to follow some specific formatting, or just give it a bit more help. In cases like this it can be valuable to provide \"in-context examples\" to help NuExtract better understand the task.\n",
-    "\n",
-    "To do so, we can provide a list `examples` to `apply_chat_template()` (or `construct_messages()`) which contains dictionaries of input/output pairs. In the example below, we show to the model that we want the extracted names to be in captial letters with `-` on either side (for the sake of illustration)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "146baa04-2206-4eca-86bd-66f94924e293",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "template = \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\"\n",
-    "document = \"John went to the restaurant with Mary. James went to the cinema.\"\n",
-    "examples = [\n",
-    "    {\n",
-    "        \"input\": \"Stephen is the manager at Susan's store.\",\n",
-    "        \"output\": \"\"\"{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}\"\"\"\n",
-    "    }\n",
-    "]\n",
-    "\n",
-    "messages = [{\"role\": \"user\", \"content\": document}]\n",
-    "text = processor.tokenizer.apply_chat_template(\n",
-    "    messages,\n",
-    "    template=template,\n",
-    "    examples=examples, # examples provided here\n",
-    "    tokenize=False,\n",
-    "    add_generation_prompt=True,\n",
-    ")\n",
-    "\n",
-    "image_inputs = process_vision_info(messages)[0]\n",
-    "inputs = processor(\n",
-    "    text=[text],\n",
-    "    images=image_inputs,\n",
-    "    padding=True,\n",
-    "    return_tensors=\"pt\",\n",
-    ").to(\"cuda\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1d2561cc-a662-4898-b226-41d12394fdb4",
-   "metadata": {},
-   "source": [
-    "We can see below that the in-context example has now been included in the model prompt, specifically between the template and context components."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "79b10e44-5bb4-40eb-af74-e013897948ea",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "<|im_start|>system\n",
-      "You are NuExtract, an information extraction tool created by NuMind.<|im_end|>\n",
-      "<|im_start|>user\n",
-      "# Template:\n",
-      "{\"names\": [\"verbatim-string\"]}\n",
-      "# Examples:\n",
-      "## Input:\n",
-      "Stephen is the manager at Susan's store.\n",
-      "## Output:\n",
-      "{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}\n",
-      "# Context:\n",
-      "John went to the restaurant with Mary. James went to the cinema.<|im_end|>\n",
-      "<|im_start|>assistant\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(text)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "3f1364bf-2588-4935-a478-7a85b87866fc",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "['{\"names\": [\"-JOHN-\", \"-MARY-\", \"-JAMES-\"]}']\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Inference: Generation of the output\n",
-    "generated_ids = model.generate(\n",
-    "    **inputs,\n",
-    "    **generation_config\n",
-    ")\n",
-    "generated_ids_trimmed = [\n",
-    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
-    "]\n",
-    "output_text = processor.batch_decode(\n",
-    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
-    ")\n",
-    "print(output_text)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "bc0c3ecf-80ee-40b7-b240-12b5a9e10042",
-   "metadata": {},
-   "source": [
-    "To get even better performance, add multiple in-context examples to your input."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "99faf117-18b7-4b5e-8cd0-b6f8941db581",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "### Image Inputs\n",
-    "\n",
-    "If we want to give image inputs to NuExtract, instead of text, we simply provide a dictionary specifying the desired image file as the message content, instead of a string. E.g. `{\"type\": \"image\", \"image\": \"file://image.jpg\"}`.\n",
-    "\n",
-    "You can also specify an image URL (e.g. `{\"type\": \"image\", \"image\": \"http://path/to/your/image.jpg\"}`) or base64 encoding (e.g. `{\"type\": \"image\", \"image\": \"data:image;base64,/9j/...\"}`).\n",
-    "\n",
-    "First, we will need a modified version of `process_vision_info()` that handles image-based in-context examples as well as primary inputs."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "1aaf1912-4f50-483f-a9a0-f5f57caf66e0",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "def process_all_vision_info(messages, examples=None):\n",
-    "    \"\"\"\n",
-    "    Process vision information from both messages and in-context examples, supporting batch processing.\n",
-    "\n",
-    "    Args:\n",
-    "        messages: List of message dictionaries (single input) OR list of message lists (batch input)\n",
-    "        examples: Optional list of example dictionaries (single input) OR list of example lists (batch)\n",
-    "\n",
-    "    Returns:\n",
-    "        A flat list of all images in the correct order:\n",
-    "        - For single input: example images followed by message images\n",
-    "        - For batch input: interleaved as (item1 examples, item1 input, item2 examples, item2 input, etc.)\n",
-    "        - Returns None if no images were found\n",
-    "    \"\"\"\n",
-    "    from qwen_vl_utils import process_vision_info, fetch_image\n",
-    "\n",
-    "    # Helper function to extract images from examples\n",
-    "    def extract_example_images(example_item):\n",
-    "        if not example_item:\n",
-    "            return []\n",
-    "\n",
-    "        # Handle both list of examples and single example\n",
-    "        examples_to_process = example_item if isinstance(example_item, list) else [example_item]\n",
-    "        images = []\n",
-    "\n",
-    "        for example in examples_to_process:\n",
-    "            if isinstance(example.get('input'), dict) and example['input'].get('type') == 'image':\n",
-    "                images.append(fetch_image(example['input']))\n",
-    "\n",
-    "        return images\n",
-    "\n",
-    "    # Normalize inputs to always be batched format\n",
-    "    is_batch = messages and isinstance(messages[0], list)\n",
-    "    messages_batch = messages if is_batch else [messages]\n",
-    "    is_batch_examples = examples and isinstance(examples, list) and (isinstance(examples[0], list) or examples[0] is None)\n",
-    "    examples_batch = examples if is_batch_examples else ([examples] if examples is not None else None)\n",
-    "\n",
-    "    # Ensure examples batch matches messages batch if provided\n",
-    "    if examples and len(examples_batch) != len(messages_batch):\n",
-    "        if not is_batch and len(examples_batch) == 1:\n",
-    "            # Single example set for a single input is fine\n",
-    "            pass\n",
-    "        else:\n",
-    "            raise ValueError(\"Examples batch length must match messages batch length\")\n",
-    "\n",
-    "    # Process all inputs, maintaining correct order\n",
-    "    all_images = []\n",
-    "    for i, message_group in enumerate(messages_batch):\n",
-    "        # Get example images for this input\n",
-    "        if examples and i < len(examples_batch):\n",
-    "            input_example_images = extract_example_images(examples_batch[i])\n",
-    "            all_images.extend(input_example_images)\n",
-    "\n",
-    "        # Get message images for this input\n",
-    "        input_message_images = process_vision_info(message_group)[0] or []\n",
-    "        all_images.extend(input_message_images)\n",
-    "\n",
-    "    return all_images if all_images else None\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8db9904c-451d-4ff4-b776-5a557abc1437",
-   "metadata": {},
-   "source": [
-    "In the example below, we give an image of a receipt from Trader Joe's (`data/1.jpg`) and ask the model to extract the name of the store. We also provide an ICL example of a receipt from Walmart (`data/0.jpg`)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "fd11e855-594b-4fd7-af01-7695a65b75a9",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from pathlib import Path\n",
-    "\n",
-    "project_root = Path.cwd().parent if Path.cwd().name == \"notebooks\" else Path.cwd()\n",
-    "samples_dir = project_root / \"data\" / \"samples\"\n",
-    "sample_images = sorted(\n",
-    "    path for path in samples_dir.iterdir()\n",
-    "    if path.suffix.lower() in {\".jpg\", \".jpeg\", \".png\", \".webp\"}\n",
-    ")\n",
-    "if not sample_images:\n",
-    "    raise FileNotFoundError(f\"No supported images found in {samples_dir}\")\n",
-    "\n",
-    "# Pick a different entry from sample_images to use another file.\n",
-    "selected_image = sample_images[0]\n",
-    "input_image_uri = selected_image.resolve().as_uri()\n",
-    "print(f\"Using input image: {selected_image}\")\n",
-    "\n",
-    "template = \"\"\"{\"store\": \"verbatim-string\"}\"\"\"\n",
-    "document = {\"type\": \"image\", \"image\": input_image_uri}\n",
-    "examples = [\n",
-    "    {\n",
-    "        \"input\": {\"type\": \"image\", \"image\": input_image_uri},\n",
-    "        \"output\": \"\"\"{\"store\": \"WALMART\"}\"\"\"\n",
-    "    }\n",
-    "]\n",
-    "\n",
-    "messages = [{\"role\": \"user\", \"content\": [document]}]\n",
-    "text = processor.tokenizer.apply_chat_template(\n",
-    "    messages,\n",
-    "    template=template,\n",
-    "    examples=examples,\n",
-    "    tokenize=False,\n",
-    "    add_generation_prompt=True,\n",
-    ")\n",
-    "\n",
-    "image_inputs = process_all_vision_info(messages, examples)\n",
-    "inputs = processor(\n",
-    "    text=[text],\n",
-    "    images=image_inputs,\n",
-    "    padding=True,\n",
-    "    return_tensors=\"pt\",\n",
-    ").to(\"cuda\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f14a8e8a-a091-410a-a964-9d275b936e75",
-   "metadata": {},
-   "source": [
-    "Just like in the text-only case above, our in-context example has been included in the prompt before the main context."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "ba00c8d6-56e2-43fc-a646-61f693c6b51a",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "<|im_start|>system\n",
-      "You are NuExtract, an information extraction tool created by NuMind.<|im_end|>\n",
-      "<|im_start|>user\n",
-      "# Template:\n",
-      "{\"store\": \"verbatim-string\"}\n",
-      "# Examples:\n",
-      "## Input:\n",
-      "<|vision_start|><|image_pad|><|vision_end|>\n",
-      "## Output:\n",
-      "{\"store\": \"WALMART\"}\n",
-      "# Context:\n",
-      "<|vision_start|><|image_pad|><|vision_end|><|im_end|>\n",
-      "<|im_start|>assistant\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(text)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4820e6da-544b-431a-86fb-a6f9f0eb6bdd",
-   "metadata": {},
-   "source": [
-    "Now if we look at `image_inputs` we will see that it contains actual images. When we pass this along with `text` to `processor()` it automatically encodes the images and injects a tokenized representation into the image placeholders within `text`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "1aa9fba6-eb58-4401-abc2-79b7fcd205c0",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "[<PIL.Image.Image image mode=RGB size=588x896 at 0x7FDD295E5600>, <PIL.Image.Image image mode=RGB size=476x980 at 0x7FDD295E4460>]\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(image_inputs)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "fbed800c-519b-4ee1-90d9-7c27e383d904",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "['{\"store\": \"TRADER JOE\\'S\"}']\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Inference: Generation of the output\n",
-    "generated_ids = model.generate(\n",
-    "    **inputs,\n",
-    "    **generation_config\n",
-    ")\n",
-    "generated_ids_trimmed = [\n",
-    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
-    "]\n",
-    "output_text = processor.batch_decode(\n",
-    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
-    ")\n",
-    "print(output_text)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c6beb77f-2de5-4846-b244-2cda27f5f553",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "### Batched Inference\n",
-    "\n",
-    "Finally, we can run batched inference over a list of input examples, regardless of whether they contain text, images, and/or ICL examples."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "947b8090-236c-4821-b295-5a8fb5247943",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{\"store_name\": \"WAL-MART\"}\n",
-      "{\"store_name\": \"Walmart\"}\n",
-      "{\"names\": [\"John\", \"Mary\", \"James\"]}\n",
-      "{\"names\": [\"-JOHN-\", \"-MARY-\", \"-JAMES-\"]}\n"
-     ]
-    }
-   ],
-   "source": [
-    "inputs = [\n",
-    "    # image input with no ICL examples\n",
-    "    {\n",
-    "        \"document\": {\"type\": \"image\", \"image\": input_image_uri},\n",
-    "        \"template\": \"\"\"{\"store_name\": \"verbatim-string\"}\"\"\",\n",
-    "    },\n",
-    "    # image input with 1 ICL example\n",
-    "    {\n",
-    "        \"document\": {\"type\": \"image\", \"image\": input_image_uri},\n",
-    "        \"template\": \"\"\"{\"store_name\": \"verbatim-string\"}\"\"\",\n",
-    "        \"examples\": [\n",
-    "            {\n",
-    "                \"input\": {\"type\": \"image\", \"image\": input_image_uri},\n",
-    "                \"output\": \"\"\"{\"store_name\": \"Trader Joe's\"}\"\"\",\n",
-    "            }\n",
-    "        ],\n",
-    "    },\n",
-    "    # text input with no ICL examples\n",
-    "    {\n",
-    "        \"document\": {\"type\": \"text\", \"text\": \"John went to the restaurant with Mary. James went to the cinema.\"},\n",
-    "        \"template\": \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\",\n",
-    "    },\n",
-    "    # text input with ICL example\n",
-    "    {\n",
-    "        \"document\": {\"type\": \"text\", \"text\": \"John went to the restaurant with Mary. James went to the cinema.\"},\n",
-    "        \"template\": \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\",\n",
-    "        \"examples\": [\n",
-    "            {\n",
-    "                \"input\": \"Stephen is the manager at Susan's store.\",\n",
-    "                \"output\": \"\"\"{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}\"\"\"\n",
-    "            }\n",
-    "        ],\n",
-    "    },\n",
-    "]\n",
-    "\n",
-    "# messages should be a list of lists for batch processing\n",
-    "messages = [[{\"role\": \"user\", \"content\": [x['document']]}] for x in inputs]\n",
-    "\n",
-    "# apply chat template to each example individually\n",
-    "texts = [\n",
-    "    processor.tokenizer.apply_chat_template(\n",
-    "        messages[i],  # Now this is a list containing one message\n",
-    "        template=x['template'],\n",
-    "        examples=x.get('examples', None),\n",
-    "        tokenize=False,\n",
-    "        add_generation_prompt=True)\n",
-    "    for i, x in enumerate(inputs)\n",
-    "]\n",
-    "\n",
-    "image_inputs = process_all_vision_info(messages, [x.get('examples') for x in inputs])\n",
-    "inputs = processor(\n",
-    "    text=texts,\n",
-    "    images=image_inputs,\n",
-    "    padding=True,\n",
-    "    return_tensors=\"pt\",\n",
-    ").to(\"cuda\")\n",
-    "\n",
-    "# Batch Inference\n",
-    "generated_ids = model.generate(**inputs, **generation_config)\n",
-    "generated_ids_trimmed = [\n",
-    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
-    "]\n",
-    "output_texts = processor.batch_decode(\n",
-    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
-    ")\n",
-    "for y in output_texts:\n",
-    "    print(y)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "cf456dcc-52ce-495d-872f-a641c6f40f7d",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "environment": {
-   "kernel": "conda-base-py",
-   "name": "workbench-notebooks.m127",
-   "type": "gcloud",
-   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m127"
-  },
-  "kernelspec": {
-   "display_name": ".venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.12.10"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

pyproject.toml

@@ -5,12 +5,17 @@ description = "Add your description here"
 readme = "README.md"
 requires-python = ">=3.12,<3.13"
 dependencies = [
+    "accelerate>=1.14.0",
     "flash-attn",
+    "gguf>=0.19.0",
     "gradio",
     "huggingface-hub",
-    "qwen-vl-utils>=0.0.14",
-    "torch==2.9.0",
-    "transformers>=5.14.1",
+    "ipykernel>=7.3.0",
+    "ipython>=9.15.0",
+    "ipywidgets>=8.1.8",
+    "openai>=2.46.0",
+    "torch==2.9.1",
+    "transformers",
 ]
 
 [tool.uv.sources]

uv.lock

@@ -7,6 +7,24 @@ resolution-markers = [
     "sys_platform != 'emscripten' and sys_platform != 'win32'",
 ]
 
+[[package]]
+name = "accelerate"
+version = "1.14.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "huggingface-hub" },
+    { name = "numpy" },
+    { name = "packaging" },
+    { name = "psutil" },
+    { name = "pyyaml" },
+    { name = "safetensors" },
+    { name = "torch" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/8d/75/94cd5d389649578aca399e5aa822637eec18319a1dadc400ffe2f9a7493f/accelerate-1.14.0.tar.gz", hash = "sha256:41b9c4377a54e0b460a959b0defa1b736e4ca0a2373252d9a539964c2afe3c8d", size = 412167, upload-time = "2026-06-11T13:45:52.326Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/a8/db/253133d7e7cb40d3af384bb2f5c0b4a2b7fdcffbc95c688cc67a20a3c103/accelerate-1.14.0-py3-none-any.whl", hash = "sha256:e94390c2863b873be18f623f9df48a0d8fe5eff13ea7f1a00092b0a7904888c6", size = 389246, upload-time = "2026-06-11T13:45:50.477Z" },
+]
+
 [[package]]
 name = "annotated-doc"
 version = "0.0.4"
@@ -39,20 +57,21 @@ wheels = [
 ]
 
 [[package]]
-name = "av"
-version = "18.0.0"
+name = "appnope"
+version = "0.1.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/35/5d/752690df9ef5b76e169e68d6a129fa6d08a7100ca7f754c89495db3c6019/appnope-0.1.4.tar.gz", hash = "sha256:1de3860566df9caf38f01f86f65e0e13e379af54f9e4bee1e66b48f2efffd1ee", size = 4170, upload-time = "2024-02-06T09:43:11.258Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/81/29/5ecc3a15d5a33e31b26c11426c45c501e439cb865d0bff96315d86443b78/appnope-0.1.4-py2.py3-none-any.whl", hash = "sha256:502575ee11cd7a28c0205f379b525beefebab9d161b7c964670864014ed7213c", size = 4321, upload-time = "2024-02-06T09:43:09.663Z" },
+]
+
+[[package]]
+name = "asttokens"
+version = "3.0.2"
 source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/ae/a4/570a5a35c8638aba01e739925846c35fdd6b0756a15526766d0a4dd3b7df/av-18.0.0.tar.gz", hash = "sha256:4ef7e72c3d3a872584a1215173b16e0226811037f40dcdbf75992631098df1ba", size = 4340222, upload-time = "2026-07-02T06:37:58.907Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/25/1e/faf0f247f6f881b98fc4d6d07e14085cb89d13665084e6d6ac1dc2c03d0b/asttokens-3.0.2.tar.gz", hash = "sha256:3ecdbd8f2cc195f53ccada3a613538bb5f9ef6f6869129f13e03c30a677b8fe2", size = 63136, upload-time = "2026-07-12T03:31:49.084Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/18/4a/9e3463df030e063d757fa12f0f39be6541b45b06b5bad48c2ce361b924bf/av-18.0.0-cp311-abi3-macosx_11_0_x86_64.whl", hash = "sha256:149289d40e732a6e49c9530bc245b49d9964cfd1c8c9e06778703b7d5bba6b25", size = 22499354, upload-time = "2026-07-02T06:36:58.751Z" },
-    { url = "https://files.pythonhosted.org/packages/77/b3/2576a44b4f39c7462ced4c17fec04c756f7b0f3c5cb940d124173e417d6a/av-18.0.0-cp311-abi3-macosx_14_0_arm64.whl", hash = "sha256:35274c20d2ad3b4774fe632bcef2e34af79858ddf899352339cc3babbc13a484", size = 18175248, upload-time = "2026-07-02T06:37:01.741Z" },
-    { url = "https://files.pythonhosted.org/packages/84/74/6732f17b96dc23fd23b876b2805435855abdc8a3b397142be4e581165de8/av-18.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:4d683b7747a0ba9222b8a5f81e41db5f796e7f64473454ec4fe2548e083c2fa0", size = 33387843, upload-time = "2026-07-02T06:37:05.097Z" },
-    { url = "https://files.pythonhosted.org/packages/6d/b9/7708c43fed7ae28b4a1bad060b4221e3334cd827cec24f7165902a6ac1f4/av-18.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:ae56b40b6f8b067a8ad2dac664fbfbabac7f7a55b9a7bb031eb99289252bc017", size = 35536910, upload-time = "2026-07-02T06:37:08.806Z" },
-    { url = "https://files.pythonhosted.org/packages/5a/94/eba99691d184f6a395a242d54dc370e2fd2265e95bbc98e2963a0fdbdd6c/av-18.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:ea2e8ebbce521f21b55df9400e00d721623c9020ef158f5a188a96130be0743f", size = 38984619, upload-time = "2026-07-02T06:37:11.861Z" },
-    { url = "https://files.pythonhosted.org/packages/c9/cf/0d7aee07fe16aa9ffdf96043c14bed5485a52c0dea4259de87aa306ecab4/av-18.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:ef96dabb3e50dac249913145dff5424b302b257fd95dcb64be3c7b7a8aef16d1", size = 34451176, upload-time = "2026-07-02T06:37:15.154Z" },
-    { url = "https://files.pythonhosted.org/packages/76/92/810da80b12680d4c4fe235bd1b4003289be9213ac7f114b77b8ecf0e3b3e/av-18.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:0f65518a184613e41536f29e8758c8e3d8293e46bf5bef108f04f925bbfa3f44", size = 36619869, upload-time = "2026-07-02T06:37:18.495Z" },
-    { url = "https://files.pythonhosted.org/packages/11/85/0f121ff43dc5a70696676c98a8f1674e2fa787614c2abaacb15fa1a9bc99/av-18.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:aaf4d354d2beaa6651e4f92e54409a578bde64f79c0beef9a30b388d06f7c629", size = 27556236, upload-time = "2026-07-02T06:37:21.388Z" },
-    { url = "https://files.pythonhosted.org/packages/8b/f6/2509754d4d2356abc6fc0ea3d57c12ade29bac23a1fb7fc215a53ca518fb/av-18.0.0-cp311-abi3-win_arm64.whl", hash = "sha256:adac2b3833b6cb9bd6cb52664a522b94db453615b3675b1dbb26e13fe1c80da6", size = 20221133, upload-time = "2026-07-02T06:37:23.88Z" },
+    { url = "https://files.pythonhosted.org/packages/d4/2b/04b8a15f3a1c77bc79ddf5c73875327f34b4fa75982df2b76e45e402d364/asttokens-3.0.2-py3-none-any.whl", hash = "sha256:9da13157f5b28becde0bd374fc677dcd3c290614264eff096f167c469cd9f933", size = 28702, upload-time = "2026-07-12T03:31:47.542Z" },
 ]
 
 [[package]]
@@ -82,6 +101,29 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/ef/2f/c5464532e965badff2f4c4c1a3a83f5697f0d7c407ed0cda44aaa99bb451/certifi-2026.6.17-py3-none-any.whl", hash = "sha256:2227dcbaafe0d2f59279d1762ddddc37783ed4354594f194ffc31d20f41fc3db", size = 133289, upload-time = "2026-06-17T10:31:06.348Z" },
 ]
 
+[[package]]
+name = "cffi"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "pycparser", marker = "implementation_name != 'PyPy'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/57/5f/ff100cae70ebe9d8df1c01a00e510e45d9adb5c1fdda84791b199141de97/cffi-2.1.0.tar.gz", hash = "sha256:efc1cdd798b1aaf39b4610bba7aad28c9bea9b910f25c784ccf9ec1fa719d1f9", size = 531036, upload-time = "2026-07-06T21:34:30.382Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/1e/85/990925db5df586ec90beb97529c853497e7f85ba0234830447faf41c3057/cffi-2.1.0-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:df2b82571a1b30f58a87bf4e5a9e78d2b1eff6c6ce8fd3aa3757221f93f0863f", size = 184829, upload-time = "2026-07-06T21:32:44.324Z" },
+    { url = "https://files.pythonhosted.org/packages/4b/92/e7bb136ad6b5352603732cf907ef862ca103f20f2031c1735a46300c20c9/cffi-2.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:78474632761faa0fb96f30b1c928c84ebcf68713cbb80d15bab09dfe61640fde", size = 184728, upload-time = "2026-07-06T21:32:45.683Z" },
+    { url = "https://files.pythonhosted.org/packages/c3/c0/d1ec30ffb370f748f2fb54425972bfef9871e0132e82fb589c46b6676049/cffi-2.1.0-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:5972433ad71a9e46516584ef60a0fda12d9dc459938d1539c3ddecf9bdc1368d", size = 214815, upload-time = "2026-07-06T21:32:48.557Z" },
+    { url = "https://files.pythonhosted.org/packages/1b/dc/5620cf930688be01f2d673804291de757a934c90b946dbdc3d84130c2ea4/cffi-2.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b6422532152adf4e59b110cb2808cee7a033800952f5c036b4af047ee43199e7", size = 222429, upload-time = "2026-07-06T21:32:49.848Z" },
+    { url = "https://files.pythonhosted.org/packages/4b/a4/77b53abbf7a1e0beb9637edbef2a94d15f9c822f591e85d439ffd91519a6/cffi-2.1.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:46b1c8db8f6122420f32d02fffb924c2fe9bc772d228c7c711748fff56aabb2b", size = 210315, upload-time = "2026-07-06T21:32:51.221Z" },
+    { url = "https://files.pythonhosted.org/packages/58/0c/f528df19cc94b675087324d4760d9e6d5bfae97d6217aa4fac43de4f5fcc/cffi-2.1.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:d9fafc5aa2e2a39aaf7f8cc0c1f044a9b07fca12e558dca53a3cc5c654ad67a7", size = 208859, upload-time = "2026-07-06T21:32:52.512Z" },
+    { url = "https://files.pythonhosted.org/packages/62/f2/c9522a81c32132799a1972c39f5c5f8b4c8b9f00488a23feaa6c06f07741/cffi-2.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1e9f50d192a3e525b15a75ab5114e442d83d657b7ec29182a991bc9a88fd3a66", size = 221844, upload-time = "2026-07-06T21:32:53.704Z" },
+    { url = "https://files.pythonhosted.org/packages/6e/28/bd53988b9833e8f8ad539d26f4c07a6b3f6bcb1e9e02e7ca038250b3428d/cffi-2.1.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:98fff996e983a36d3aa2eca83af40c5821202e7e6f32d13ae94e3d2286f10cfe", size = 225287, upload-time = "2026-07-06T21:32:54.907Z" },
+    { url = "https://files.pythonhosted.org/packages/79/99/0d0fd37f055224085f42bbb2c022d002e17dde4a97972822327b07d84101/cffi-2.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:379de10ce1ba048b1448599d1b37b24caee16309d1ac98d3982fc997f768700b", size = 223681, upload-time = "2026-07-06T21:32:56.329Z" },
+    { url = "https://files.pythonhosted.org/packages/b0/80/c138990aa2a70b1a269f6e06348729836d733d6f970867943f61d367f8cc/cffi-2.1.0-cp312-cp312-win32.whl", hash = "sha256:9b8f0f26ca4e7513c534d351eca551947d053fac438f2a04ac96d882909b0d3a", size = 175269, upload-time = "2026-07-06T21:32:57.777Z" },
+    { url = "https://files.pythonhosted.org/packages/a8/eb/f636456ff21a83fc13c032b58cc5dde061691546ac79efa284b2989b7982/cffi-2.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:c97f080ea627e2863524c5af3836e2270b5f5dfff1f104392b959f8df0c5d384", size = 185881, upload-time = "2026-07-06T21:32:59.253Z" },
+    { url = "https://files.pythonhosted.org/packages/dd/2c/400ea43e721727dca8a65c4521390e9196757caba4a45643acb2b63271b8/cffi-2.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:6d194185eabd279f1c05ebe3504265ddfc5ad2b58d0714f7db9f01da592e9eb6", size = 180088, upload-time = "2026-07-06T21:33:02.278Z" },
+]
+
 [[package]]
 name = "charset-normalizer"
 version = "3.4.9"
@@ -125,6 +167,46 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
 ]
 
+[[package]]
+name = "comm"
+version = "0.2.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4c/13/7d740c5849255756bc17888787313b61fd38a0a8304fc4f073dfc46122aa/comm-0.2.3.tar.gz", hash = "sha256:2dc8048c10962d55d7ad693be1e7045d891b7ce8d999c97963a5e3e99c055971", size = 6319, upload-time = "2025-07-25T14:02:04.452Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/60/97/891a0971e1e4a8c5d2b20bbe0e524dc04548d2307fee33cdeba148fd4fc7/comm-0.2.3-py3-none-any.whl", hash = "sha256:c615d91d75f7f04f095b30d1c1711babd43bdc6419c1be9886a85f2f4e489417", size = 7294, upload-time = "2025-07-25T14:02:02.896Z" },
+]
+
+[[package]]
+name = "debugpy"
+version = "1.8.21"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f2/aa/12037145b7a56eaa5b29b41872f7a21b538e807e13f32c4d3c46e59be084/debugpy-1.8.21.tar.gz", hash = "sha256:a3c53278e84c94e11bd87c53970ec391d1a67396c8b22609fcac576520e611a6", size = 1697577, upload-time = "2026-06-01T19:30:35.156Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/a2/df/bf625547431a9cadc9f4cbfeda38866e2b17f6aed147b625377e87834449/debugpy-1.8.21-cp312-cp312-macosx_15_0_universal2.whl", hash = "sha256:9f96713896f39c3dff0ee841f47320c3f2983d33c341e009361bb0ebc79adc4e", size = 2483609, upload-time = "2026-06-01T19:30:50.794Z" },
+    { url = "https://files.pythonhosted.org/packages/bf/09/59324b903599031ff9faaec1758292409f6561a0ec2492fe4b703327705a/debugpy-1.8.21-cp312-cp312-manylinux_2_34_x86_64.whl", hash = "sha256:c193d474f0a211191f2b4449d2d06157c689013035bd952f3b617e0ef422b176", size = 3968900, upload-time = "2026-06-01T19:30:52.341Z" },
+    { url = "https://files.pythonhosted.org/packages/14/cd/27f65b805d7fe005c44e1a36b9183ecdfbcdbf9d3e721a5115d461ecc7ee/debugpy-1.8.21-cp312-cp312-win32.whl", hash = "sha256:4743373c1cac7f9e74a1b9915bf1dbe0e900eca657ffb170ae07ac8363205ae9", size = 5336340, upload-time = "2026-06-01T19:30:54.047Z" },
+    { url = "https://files.pythonhosted.org/packages/77/1d/c84e30c0c674184948b66f076ab271c01d940618a2824c23cd035a27bc20/debugpy-1.8.21-cp312-cp312-win_amd64.whl", hash = "sha256:bd7ba9dd3daa7c2f942c6ca8d4695a16bf9ac16b63615261c7982bc74f7ed20c", size = 5374751, upload-time = "2026-06-01T19:30:55.891Z" },
+    { url = "https://files.pythonhosted.org/packages/95/51/67e7cf11a53e40694f720457d5b3a1cdaaa3d5a9a633e482f225456b93ff/debugpy-1.8.21-py2.py3-none-any.whl", hash = "sha256:b1e37d333663c8851516a47364ef473da127f9caebe4417e6df6f5825a7e9a92", size = 5352888, upload-time = "2026-06-01T19:31:25.186Z" },
+]
+
+[[package]]
+name = "decorator"
+version = "5.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/60/8b/32f9823da46cde7df2087faa08cd98d01b908f8dcab982cdba9c84e85355/decorator-5.3.1.tar.gz", hash = "sha256:4cbcdd55a6efadb9dbea26b858f4fb3264567b52d69ca0d25b721b553f60ea82", size = 58084, upload-time = "2026-05-18T06:03:28.057Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/05/7f/798705f5296a58ca505d600456748d1be48078eac8a7050d8a98bc9edb89/decorator-5.3.1-py3-none-any.whl", hash = "sha256:f47fe6fdbd2edd623ecfe36875d37aba411624e2670dd395dddae1358689bb3c", size = 10365, upload-time = "2026-05-18T06:03:26.517Z" },
+]
+
+[[package]]
+name = "distro"
+version = "1.9.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fc/f8/98eea607f65de6527f8a2e8885fc8015d3e6f5775df186e443e0964a11c3/distro-1.9.0.tar.gz", hash = "sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed", size = 60722, upload-time = "2023-12-24T09:54:32.31Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" },
+]
+
 [[package]]
 name = "einops"
 version = "0.8.2"
@@ -134,6 +216,15 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl", hash = "sha256:54058201ac7087911181bfec4af6091bb59380360f069276601256a76af08193", size = 65638, upload-time = "2026-01-26T04:13:18.546Z" },
 ]
 
+[[package]]
+name = "executing"
+version = "2.2.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cc/28/c14e053b6762b1044f34a13aab6859bbf40456d37d23aa286ac24cfd9a5d/executing-2.2.1.tar.gz", hash = "sha256:3632cc370565f6648cc328b32435bd120a1e4ebb20c77e3fdde9a13cd1e533c4", size = 1129488, upload-time = "2025-09-01T09:48:10.866Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/c1/ea/53f2148663b321f21b5a606bd5f191517cf40b7072c0497d3c92c4a13b1e/executing-2.2.1-py2.py3-none-any.whl", hash = "sha256:760643d3452b4d777d295bb167ccc74c64a81df23fb5e08eff250c425a4b2017", size = 28317, upload-time = "2025-09-01T09:48:08.5Z" },
+]
+
 [[package]]
 name = "fastapi"
 version = "0.139.2"
@@ -186,6 +277,21 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/d5/0c/043d5e551459da400957a1395e0febbf771446ff34291afcbe3d8be2a279/fsspec-2026.4.0-py3-none-any.whl", hash = "sha256:11ef7bb35dab8a394fde6e608221d5cf3e8499401c249bebaeaad760a1a8dec2", size = 203402, upload-time = "2026-04-29T20:42:36.842Z" },
 ]
 
+[[package]]
+name = "gguf"
+version = "0.19.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "numpy" },
+    { name = "pyyaml" },
+    { name = "requests" },
+    { name = "tqdm" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/48/ae/17f1308ae45cd7b08ebb521747d5b23f4efc4d172038a4e228dd5106c3ff/gguf-0.19.0.tar.gz", hash = "sha256:dbadcd6cc7ccd44256f2229fe7c2dff5e8aa5cf0612ab987fd2b1a57e428923f", size = 111220, upload-time = "2026-05-06T13:04:03.667Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/b3/bb/d71d6da82763528c2c2ed6b59a9d6142c6595545a4c448e2085d155e88c2/gguf-0.19.0-py3-none-any.whl", hash = "sha256:70bcd10edfe697fb2dad6e40af2234b9d8ece9a41a99761405121ebda1c3c1cd", size = 118475, upload-time = "2026-05-06T13:04:02.588Z" },
+]
+
 [[package]]
 name = "gradio"
 version = "6.20.0"
@@ -344,6 +450,92 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl", hash = "sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2", size = 61538, upload-time = "2022-09-14T00:24:23.22Z" },
 ]
 
+[[package]]
+name = "ipykernel"
+version = "7.3.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "appnope", marker = "sys_platform == 'darwin'" },
+    { name = "comm" },
+    { name = "debugpy" },
+    { name = "ipython" },
+    { name = "jupyter-client" },
+    { name = "jupyter-core" },
+    { name = "matplotlib-inline" },
+    { name = "nest-asyncio2" },
+    { name = "packaging" },
+    { name = "psutil" },
+    { name = "pyzmq" },
+    { name = "tornado" },
+    { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3d/c4/e4a38f579de4225a561305666f7541cdabb30075def2aa1ac17bd73c1fb5/ipykernel-7.3.0.tar.gz", hash = "sha256:9acaaaf97d16355166e4085afe9d225bfbdf2b7ef520f9df3be8f2b248275e09", size = 184899, upload-time = "2026-06-10T08:41:25.481Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/3d/02/77b271f5dc58bfbc0b577c877b2365d1ffea2afe66a80c13f2312820348c/ipykernel-7.3.0-py3-none-any.whl", hash = "sha256:897eb64da762549ef610698fca5e9675195ec6ac8ec7f19d81ce1ca20c876057", size = 120583, upload-time = "2026-06-10T08:41:23.648Z" },
+]
+
+[[package]]
+name = "ipython"
+version = "9.15.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "colorama", marker = "sys_platform == 'win32'" },
+    { name = "decorator" },
+    { name = "ipython-pygments-lexers" },
+    { name = "jedi" },
+    { name = "matplotlib-inline" },
+    { name = "pexpect", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+    { name = "prompt-toolkit" },
+    { name = "psutil", marker = "sys_platform != 'cygwin' and sys_platform != 'emscripten'" },
+    { name = "pygments" },
+    { name = "stack-data" },
+    { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/53/59/165d3b4d75cc34add3122c4417ecb229085140ac573103c223cd01dde96f/ipython-9.15.0.tar.gz", hash = "sha256:da2819ce2aa83135257df830660b1176d986c3d2876db24df01974fa955b2756", size = 4442580, upload-time = "2026-06-26T11:03:35.913Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/40/3a/948263ca3b9d65bb2b1b0c521b3a49fad5d59ada58724bd87d2bd5ff3f36/ipython-9.15.0-py3-none-any.whl", hash = "sha256:515ad9c3cdf0c932a5a9f6245419e8aba706b7bd03c3e1d3a1c83d9351d6aa6e", size = 630895, upload-time = "2026-06-26T11:03:33.809Z" },
+]
+
+[[package]]
+name = "ipython-pygments-lexers"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ef/4c/5dd1d8af08107f88c7f741ead7a40854b8ac24ddf9ae850afbcf698aa552/ipython_pygments_lexers-1.1.1.tar.gz", hash = "sha256:09c0138009e56b6854f9535736f4171d855c8c08a563a0dcd8022f78355c7e81", size = 8393, upload-time = "2025-01-17T11:24:34.505Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/d9/33/1f075bf72b0b747cb3288d011319aaf64083cf2efef8354174e3ed4540e2/ipython_pygments_lexers-1.1.1-py3-none-any.whl", hash = "sha256:a9462224a505ade19a605f71f8fa63c2048833ce50abc86768a0d81d876dc81c", size = 8074, upload-time = "2025-01-17T11:24:33.271Z" },
+]
+
+[[package]]
+name = "ipywidgets"
+version = "8.1.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "comm" },
+    { name = "ipython" },
+    { name = "jupyterlab-widgets" },
+    { name = "traitlets" },
+    { name = "widgetsnbextension" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/4c/ae/c5ce1edc1afe042eadb445e95b0671b03cee61895264357956e61c0d2ac0/ipywidgets-8.1.8.tar.gz", hash = "sha256:61f969306b95f85fba6b6986b7fe45d73124d1d9e3023a8068710d47a22ea668", size = 116739, upload-time = "2025-11-01T21:18:12.393Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/56/6d/0d9848617b9f753b87f214f1c682592f7ca42de085f564352f10f0843026/ipywidgets-8.1.8-py3-none-any.whl", hash = "sha256:ecaca67aed704a338f88f67b1181b58f821ab5dc89c1f0f5ef99db43c1c2921e", size = 139808, upload-time = "2025-11-01T21:18:10.956Z" },
+]
+
+[[package]]
+name = "jedi"
+version = "0.20.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "parso" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/46/b7/a3635f6a2d7cf5b5dd98064fc1d5fbbafcb25477bcea204a3a92145d158b/jedi-0.20.0.tar.gz", hash = "sha256:c3f4ccbd276696f4b19c54618d4fb18f9fc24b0aef02acf704b23f487daa1011", size = 3119416, upload-time = "2026-05-01T23:38:47.814Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/9a/93/242e2eab5fe682ffcb8b0084bde703a41d51e17ee0f3a31ff0d9d813620a/jedi-0.20.0-py2.py3-none-any.whl", hash = "sha256:7bdd9c2634f56713299976f4cbd59cb3fa92165cc5e05ea811fb253480728b67", size = 4884812, upload-time = "2026-05-01T23:38:43.919Z" },
+]
+
 [[package]]
 name = "jinja2"
 version = "3.1.6"
@@ -356,6 +548,71 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" },
 ]
 
+[[package]]
+name = "jiter"
+version = "0.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1d/1f/10936e16d8860c70698a1aa939a46aa0224813b782bce4e000e637da0b2d/jiter-0.16.0.tar.gz", hash = "sha256:7b24c3492c5f4f84a37946ad9cf504910cf6a782d6a4e0689b6673c5894b4a1c", size = 176431, upload-time = "2026-06-29T13:05:13.657Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/83/2b/52ace16ed031354f0539749a49e4bf33797d82bea5137910835fa4b09793/jiter-0.16.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:67c3bc1760f8c99d805dcab4e644027142a53b1d5d861f18780ebdbd5d40b72a", size = 306943, upload-time = "2026-06-29T13:03:14.035Z" },
+    { url = "https://files.pythonhosted.org/packages/94/2e/34957c2c1b661c252ba9bcc60ae0bddc27e0f7202c6073326a13c5390eec/jiter-0.16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:5af7780e4a26bd7d0d989592bf9ef12ebf806b74ab709223ecca37c749872ea9", size = 307779, upload-time = "2026-06-29T13:03:15.418Z" },
+    { url = "https://files.pythonhosted.org/packages/88/6c/59bd309cab4460c54cf1079f3eb7fe7af6a4c895c5c957a53378693bad2b/jiter-0.16.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d5bf78d0e05e45cfdd66558893938d59afe3d1b1a824a202039b20e607d25a72", size = 335826, upload-time = "2026-06-29T13:03:17.11Z" },
+    { url = "https://files.pythonhosted.org/packages/3b/8c/f5ef7b65f0df47afa16596969defb281ebb86e96df346d62be6fd853d620/jiter-0.16.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f4444a83f946605990c98f625cdd3d2725bfb818158760c5748c653170a20e0e", size = 362573, upload-time = "2026-06-29T13:03:18.781Z" },
+    { url = "https://files.pythonhosted.org/packages/2b/0b/ace4354da061ee38844a0c27dc2c21eecd27aea119e8da324bea987522d0/jiter-0.16.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3a23f0e4f957e1be65752d2dfac9a5a06b1917af8dc85deb639c3b9d02e31290", size = 457979, upload-time = "2026-06-29T13:03:20.293Z" },
+    { url = "https://files.pythonhosted.org/packages/55/40/c0253d3772eb9dcd8e6606ee9b2d53ec8e5b814589c47f140aa585f21eaa/jiter-0.16.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c22a488f7b9218e245a0025a9ba6b100e2e54700831cf4cf16833a27fba3ad01", size = 372302, upload-time = "2026-06-29T13:03:21.739Z" },
+    { url = "https://files.pythonhosted.org/packages/a8/d2/4839422241aa12860ce597b20068727094ba0bc480723c74924ca5bad483/jiter-0.16.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:46add52f4ad47a08bfb1219f3e673da972191489a33016edefdb5ea55bfa8c48", size = 343805, upload-time = "2026-06-29T13:03:23.384Z" },
+    { url = "https://files.pythonhosted.org/packages/e2/59/e196888a05befdda7dbe299b722d56f2f6eec65402bc34c0a3306d595feb/jiter-0.16.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:9c8a956fd72c2cf1e730d01ea080341f13aa0a97a4a33b51abebe725b7ae9ca9", size = 351107, upload-time = "2026-06-29T13:03:24.815Z" },
+    { url = "https://files.pythonhosted.org/packages/ec/74/4cd9e0fca65232136400354b630fbfcd2de634e22ccbb96567725981b548/jiter-0.16.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:561926e0573ffe4a32498420a76d64b16c513e1ab413b9d28158a8764ac701e5", size = 388441, upload-time = "2026-06-29T13:03:26.266Z" },
+    { url = "https://files.pythonhosted.org/packages/d9/8c/554691e48bc711299c0a293dd8a6179e24b2d66a54dc295421fcf64569c0/jiter-0.16.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:44d019fa8cdaf89bf29c71b39e3712143fdd0ac76725c6ef954f9957a5ea8730", size = 516354, upload-time = "2026-06-29T13:03:28.02Z" },
+    { url = "https://files.pythonhosted.org/packages/a4/cb/01e9d69dc2cc6759d4f91e230b34489c4fdb2518992650633f9e20bece89/jiter-0.16.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:0df91907609837f33341b8e6fe73b95991fdaa57caf1a0fbd343dffe826f386f", size = 547880, upload-time = "2026-06-29T13:03:29.534Z" },
+    { url = "https://files.pythonhosted.org/packages/79/70/2953195f1c6ad00f49fa67e13df7e60acb3dd4f387101bc15abccddd905e/jiter-0.16.0-cp312-cp312-win32.whl", hash = "sha256:51d7b836acb0108d7c77df1742332cac2a1fa04a74d6dacec46e7091f0e91274", size = 203473, upload-time = "2026-06-29T13:03:31.025Z" },
+    { url = "https://files.pythonhosted.org/packages/2d/05/2909a8b10699a4d560f8c502b6b2c5f3991b682b1922c1eedda242b225bd/jiter-0.16.0-cp312-cp312-win_amd64.whl", hash = "sha256:1878349266f8ee36ecb1375cc5ba2f115f35fd9f0a1a4119e725e379126647f7", size = 196905, upload-time = "2026-06-29T13:03:32.472Z" },
+    { url = "https://files.pythonhosted.org/packages/e9/a9/6b82bb1c8d7790d602489b967b982a909e5d092875a6c2ade96444c8dfc5/jiter-0.16.0-cp312-cp312-win_arm64.whl", hash = "sha256:2ed5738ae4af18271a51a528b8811b0cbfa4a1858de9d83359e4169855d6a331", size = 190618, upload-time = "2026-06-29T13:03:34.672Z" },
+    { url = "https://files.pythonhosted.org/packages/98/ab/664fd8c4be028b2bedd3d2ff08769c4ede23d0dbc87a77c62384a0515b5d/jiter-0.16.0-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:f17d61a28b4b3e0e3e2ba98490c70501403b4d196f78732439160e7fd3678127", size = 303106, upload-time = "2026-06-29T13:05:07.118Z" },
+    { url = "https://files.pythonhosted.org/packages/1a/07/421f1d5b65493a76e16027b848aba6a7d28073ae75944fa4289cc914d39f/jiter-0.16.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:96e38eea538c8ddf853a35727c7be0741c76c13f04148ac5c116222f50ece3b3", size = 304658, upload-time = "2026-06-29T13:05:08.708Z" },
+    { url = "https://files.pythonhosted.org/packages/0a/db/bba1155f01a01c3c37a89425d571da751bbedf5c54247b831a04cb971798/jiter-0.16.0-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d284fb8d94d5855d60c44fefcab4bf966f1da6fada73992b01f6f0c9bc0c6702", size = 339719, upload-time = "2026-06-29T13:05:10.41Z" },
+    { url = "https://files.pythonhosted.org/packages/78/f7/18a1afcd64f35314b68c1f23afcd9994d0bc13e65cc77517afff4e83986d/jiter-0.16.0-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:64d613743df53199b1aa256a7d328340da6d7078aac7705a7db9d7a791e9cfd2", size = 343885, upload-time = "2026-06-29T13:05:12.087Z" },
+]
+
+[[package]]
+name = "jupyter-client"
+version = "8.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "jupyter-core" },
+    { name = "python-dateutil" },
+    { name = "pyzmq" },
+    { name = "tornado" },
+    { name = "traitlets" },
+    { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7d/dc/5512503b088997c2250b8bf18258fba9d9ce5ead641183700960d3c9d342/jupyter_client-8.9.1.tar.gz", hash = "sha256:a58f730dd9e728ba16ba1d62ebccf7ffe1ebbdbce4e95cfae941b7321ae1f4fa", size = 359256, upload-time = "2026-06-09T13:15:01.033Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/3f/6f/56d39bf385c5c27988aebaf0c18a2a17e960575740100973511018bd904e/jupyter_client-8.9.1-py3-none-any.whl", hash = "sha256:0b7a295bc46e8751e9adae84781f726c851c1d911bd793edc4a3bde942e3da81", size = 109828, upload-time = "2026-06-09T13:14:58.835Z" },
+]
+
+[[package]]
+name = "jupyter-core"
+version = "5.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "platformdirs" },
+    { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/02/49/9d1284d0dc65e2c757b74c6687b6d319b02f822ad039e5c512df9194d9dd/jupyter_core-5.9.1.tar.gz", hash = "sha256:4d09aaff303b9566c3ce657f580bd089ff5c91f5f89cf7d8846c3cdf465b5508", size = 89814, upload-time = "2025-10-16T19:19:18.444Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/e7/e7/80988e32bf6f73919a113473a604f5a8f09094de312b9d52b79c2df7612b/jupyter_core-5.9.1-py3-none-any.whl", hash = "sha256:ebf87fdc6073d142e114c72c9e29a9d7ca03fad818c5d300ce2adc1fb0743407", size = 29032, upload-time = "2025-10-16T19:19:16.783Z" },
+]
+
+[[package]]
+name = "jupyterlab-widgets"
+version = "3.0.16"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/26/2d/ef58fed122b268c69c0aa099da20bc67657cdfb2e222688d5731bd5b971d/jupyterlab_widgets-3.0.16.tar.gz", hash = "sha256:423da05071d55cf27a9e602216d35a3a65a3e41cdf9c5d3b643b814ce38c19e0", size = 897423, upload-time = "2025-11-01T21:11:29.724Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/ab/b5/36c712098e6191d1b4e349304ef73a8d06aed77e56ceaac8c0a306c7bda1/jupyterlab_widgets-3.0.16-py3-none-any.whl", hash = "sha256:45fa36d9c6422cf2559198e4db481aa243c7a32d9926b500781c830c80f7ecf8", size = 914926, upload-time = "2025-11-01T21:11:28.008Z" },
+]
+
 [[package]]
 name = "markdown-it-py"
 version = "4.2.0"
@@ -387,6 +644,18 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" },
 ]
 
+[[package]]
+name = "matplotlib-inline"
+version = "0.2.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bd/c0/9f7c9a46090390368a4d7bcb76bb87a4a36c421e4c0792cdb53486ffac7a/matplotlib_inline-0.2.2.tar.gz", hash = "sha256:72f3fe8fce36b70d4a5b612f899090cd0401deddc4ea90e1572b9f4bfb058c79", size = 8150, upload-time = "2026-05-08T17:33:33.49Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/41/09/5b161152e2d90f7b87f781c2e1267494aef9c32498df793f73ad0a0a494a/matplotlib_inline-0.2.2-py3-none-any.whl", hash = "sha256:3c821cf1c209f59fb2d2d64abbf5b23b67bcb2210d663f9918dd851c6da1fcf6", size = 9534, upload-time = "2026-05-08T17:33:32.055Z" },
+]
+
 [[package]]
 name = "mdurl"
 version = "0.1.2"
@@ -405,6 +674,15 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198, upload-time = "2023-03-07T16:47:09.197Z" },
 ]
 
+[[package]]
+name = "nest-asyncio2"
+version = "1.7.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b4/73/731debf26e27e0a0323d7bda270dc2f634b398e38f040a09da1f4351d0aa/nest_asyncio2-1.7.2.tar.gz", hash = "sha256:1921d70b92cc4612c374928d081552efb59b83d91b2b789d935c665fa01729a8", size = 14743, upload-time = "2026-02-13T00:34:04.386Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/c5/3c/3179b85b0e1c3659f0369940200cd6d0fa900e6cefcc7ea0bc6dd0e29ffb/nest_asyncio2-1.7.2-py3-none-any.whl", hash = "sha256:f5dfa702f3f81f6a03857e9a19e2ba578c0946a4ad417b4c50a24d7ba641fe01", size = 7843, upload-time = "2026-02-13T00:34:02.691Z" },
+]
+
 [[package]]
 name = "networkx"
 version = "3.6.1"
@@ -419,22 +697,32 @@ name = "numarkapp"
 version = "0.1.0"
 source = { virtual = "." }
 dependencies = [
+    { name = "accelerate" },
     { name = "flash-attn" },
+    { name = "gguf" },
     { name = "gradio" },
     { name = "huggingface-hub" },
-    { name = "qwen-vl-utils" },
+    { name = "ipykernel" },
+    { name = "ipython" },
+    { name = "ipywidgets" },
+    { name = "openai" },
     { name = "torch" },
     { name = "transformers" },
 ]
 
 [package.metadata]
 requires-dist = [
+    { name = "accelerate", specifier = ">=1.14.0" },
     { name = "flash-attn", url = "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.28/flash_attn-2.8.3+cu130torch2.9-cp312-cp312-win_amd64.whl" },
+    { name = "gguf", specifier = ">=0.19.0" },
     { name = "gradio" },
     { name = "huggingface-hub" },
-    { name = "qwen-vl-utils", specifier = ">=0.0.14" },
-    { name = "torch", specifier = "==2.9.0", index = "https://download.pytorch.org/whl/cu130" },
-    { name = "transformers", specifier = ">=5.14.1" },
+    { name = "ipykernel", specifier = ">=7.3.0" },
+    { name = "ipython", specifier = ">=9.15.0" },
+    { name = "ipywidgets", specifier = ">=8.1.8" },
+    { name = "openai", specifier = ">=2.46.0" },
+    { name = "torch", specifier = "==2.9.1", index = "https://download.pytorch.org/whl/cu130" },
+    { name = "transformers" },
 ]
 
 [[package]]
@@ -605,6 +893,25 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/86/91/8b486ba85f71a2859dd705a4ec6aab38c37a389b8b7f94343db027732999/nvidia_nvtx-13.0.39-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cddd2e08b35144f1000631c3880c9ebbcb8a2863d762e76f92d47d30ecaf87cc", size = 148037, upload-time = "2025-08-04T10:18:31.763Z" },
 ]
 
+[[package]]
+name = "openai"
+version = "2.46.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "anyio" },
+    { name = "distro" },
+    { name = "httpx" },
+    { name = "jiter" },
+    { name = "pydantic" },
+    { name = "sniffio" },
+    { name = "tqdm" },
+    { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/af/ac/f725c4efbda8657d02be684607e5a2e5ce362e4790fdbcbdfb7c15018647/openai-2.46.0.tar.gz", hash = "sha256:0421e0735ac41451cad894af4cddf0435bfbf8cbc538ac0e15b3c062f2ddc06a", size = 1114628, upload-time = "2026-07-17T02:48:06.05Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/ea/7b/206238ebcb50b235942b1c66dba4974776f2057402a8d91c399be587d66a/openai-2.46.0-py3-none-any.whl", hash = "sha256:672381db55efb3a1e2610f29304c130cccdd0b319bace4d492b2443cb64c1e7c", size = 1637556, upload-time = "2026-07-17T02:48:03.695Z" },
+]
+
 [[package]]
 name = "orjson"
 version = "3.11.9"
@@ -658,6 +965,27 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/86/bd/fda8f9705b1b09c6ebe14bfc0fa0e4ec8584d54ea673628f157ff55131af/pandas-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:557409bc4178e70ee8d9ddb494798e51ebf6ea59330f6be22c51bab2a7db6c49", size = 9066158, upload-time = "2026-05-11T18:52:56.038Z" },
 ]
 
+[[package]]
+name = "parso"
+version = "0.8.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/30/4b/90c937815137d43ce71ba043cd3566221e9df6b9c805f24b5d138c9d40a7/parso-0.8.7.tar.gz", hash = "sha256:eaaac4c9fdd5e9e8852dc778d2d7405897ec510f2a298071453e5e3a07914bb1", size = 401824, upload-time = "2026-05-01T23:13:02.138Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/99/5d/8268b644392ee874ee82a635cd0df1773de230bde356c38de28e298392cc/parso-0.8.7-py2.py3-none-any.whl", hash = "sha256:a8926eb2a1b915486941fdbd31e86a4baf88fe8c210f25f2f35ecec5b574ca1c", size = 107025, upload-time = "2026-05-01T23:12:58.867Z" },
+]
+
+[[package]]
+name = "pexpect"
+version = "4.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "ptyprocess", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/42/92/cc564bf6381ff43ce1f4d06852fc19a2f11d180f23dc32d9588bee2f149d/pexpect-4.9.0.tar.gz", hash = "sha256:ee7d41123f3c9911050ea2c2dac107568dc43b2d3b0c7557a33212c398ead30f", size = 166450, upload-time = "2023-11-25T09:07:26.339Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/9e/c3/059298687310d527a58bb01f3b1965787ee3b40dce76752eda8b44e9a2c5/pexpect-4.9.0-py2.py3-none-any.whl", hash = "sha256:7236d1e080e4936be2dc3e326cec0af72acf9212a7e1d060210e70a47e253523", size = 63772, upload-time = "2023-11-25T06:56:14.81Z" },
+]
+
 [[package]]
 name = "pillow"
 version = "12.3.0"
@@ -675,6 +1003,70 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/de/47/4845a0a6c0dbf1db8456bd9fc791f13c5ced7ced20606d08a0aacfd25b49/pillow-12.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:331b624368d4f1d069149002f25f44bc61c8919ce8ddb3c45bdad8f6e2d89510", size = 2568267, upload-time = "2026-07-01T11:54:24.051Z" },
 ]
 
+[[package]]
+name = "platformdirs"
+version = "4.10.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/52/cd/4f25b2f95b23f5d2c9c1fe43e49841bff5800562149b2666afc09309aa8f/platformdirs-4.10.1.tar.gz", hash = "sha256:ceab4084426fe6319ce18e86deada8ab1b7487c7aee7040c55e277c9ae793695", size = 31678, upload-time = "2026-07-18T03:53:43.808Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/ec/73/6fd0bb9ce84138c3857f12e9de63bc901852975a092d545f18087a204aa2/platformdirs-4.10.1-py3-none-any.whl", hash = "sha256:0e4eff26be2d75293977f7cddc153fd9b8eaa7fb0c7b64ffe4076cb443117443", size = 22906, upload-time = "2026-07-18T03:53:42.576Z" },
+]
+
+[[package]]
+name = "prompt-toolkit"
+version = "3.0.52"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "wcwidth" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a1/96/06e01a7b38dce6fe1db213e061a4602dd6032a8a97ef6c1a862537732421/prompt_toolkit-3.0.52.tar.gz", hash = "sha256:28cde192929c8e7321de85de1ddbe736f1375148b02f2e17edd840042b1be855", size = 434198, upload-time = "2025-08-27T15:24:02.057Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/84/03/0d3ce49e2505ae70cf43bc5bb3033955d2fc9f932163e84dc0779cc47f48/prompt_toolkit-3.0.52-py3-none-any.whl", hash = "sha256:9aac639a3bbd33284347de5ad8d68ecc044b91a762dc39b7c21095fcd6a19955", size = 391431, upload-time = "2025-08-27T15:23:59.498Z" },
+]
+
+[[package]]
+name = "psutil"
+version = "7.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" },
+    { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" },
+    { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" },
+    { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" },
+    { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" },
+    { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" },
+    { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" },
+    { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" },
+]
+
+[[package]]
+name = "ptyprocess"
+version = "0.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/20/e5/16ff212c1e452235a90aeb09066144d0c5a6a8c0834397e03f5224495c4e/ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220", size = 70762, upload-time = "2020-12-28T15:15:30.155Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35", size = 13993, upload-time = "2020-12-28T15:15:28.35Z" },
+]
+
+[[package]]
+name = "pure-eval"
+version = "0.2.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cd/05/0a34433a064256a578f1783a10da6df098ceaa4a57bbeaa96a6c0352786b/pure_eval-0.2.3.tar.gz", hash = "sha256:5f4e983f40564c576c7c8635ae88db5956bb2229d7e9237d03b3c0b0190eaf42", size = 19752, upload-time = "2024-07-21T12:58:21.801Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl", hash = "sha256:1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0", size = 11842, upload-time = "2024-07-21T12:58:20.04Z" },
+]
+
+[[package]]
+name = "pycparser"
+version = "3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
+]
+
 [[package]]
 name = "pydantic"
 version = "2.13.4"
@@ -787,18 +1179,24 @@ wheels = [
 ]
 
 [[package]]
-name = "qwen-vl-utils"
-version = "0.0.14"
+name = "pyzmq"
+version = "27.1.0"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "av" },
-    { name = "packaging" },
-    { name = "pillow" },
-    { name = "requests" },
+    { name = "cffi", marker = "implementation_name == 'pypy'" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/b6/b1/ad4fc2260a3badd278b38d642f3b987412f1f6682f0ef2b31b0572d5caa8/qwen_vl_utils-0.0.14.tar.gz", hash = "sha256:9c7cad5ae803b3a10f8bb7194deb12aeacdd032f92f4224e880c73587a7346ad", size = 8453, upload-time = "2025-09-23T09:38:57.532Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/04/0b/3c9baedbdf613ecaa7aa07027780b8867f57b6293b6ee50de316c9f3222b/pyzmq-27.1.0.tar.gz", hash = "sha256:ac0765e3d44455adb6ddbf4417dcce460fc40a05978c08efdf2948072f6db540", size = 281750, upload-time = "2025-09-08T23:10:18.157Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/c4/43/80f67e0336cb2fc725f8e06f7fe35c1d0fe946f4d2b8b2175e797e07349e/qwen_vl_utils-0.0.14-py3-none-any.whl", hash = "sha256:5e28657bfd031e56bd447c5901b58ddfc3835285ed100f4c56580e0ade054e96", size = 8120, upload-time = "2025-09-23T09:38:56.297Z" },
+    { url = "https://files.pythonhosted.org/packages/92/e7/038aab64a946d535901103da16b953c8c9cc9c961dadcbf3609ed6428d23/pyzmq-27.1.0-cp312-abi3-macosx_10_15_universal2.whl", hash = "sha256:452631b640340c928fa343801b0d07eb0c3789a5ffa843f6e1a9cee0ba4eb4fc", size = 1306279, upload-time = "2025-09-08T23:08:03.807Z" },
+    { url = "https://files.pythonhosted.org/packages/e8/5e/c3c49fdd0f535ef45eefcc16934648e9e59dace4a37ee88fc53f6cd8e641/pyzmq-27.1.0-cp312-abi3-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:1c179799b118e554b66da67d88ed66cd37a169f1f23b5d9f0a231b4e8d44a113", size = 895645, upload-time = "2025-09-08T23:08:05.301Z" },
+    { url = "https://files.pythonhosted.org/packages/f8/e5/b0b2504cb4e903a74dcf1ebae157f9e20ebb6ea76095f6cfffea28c42ecd/pyzmq-27.1.0-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3837439b7f99e60312f0c926a6ad437b067356dc2bc2ec96eb395fd0fe804233", size = 652574, upload-time = "2025-09-08T23:08:06.828Z" },
+    { url = "https://files.pythonhosted.org/packages/f8/9b/c108cdb55560eaf253f0cbdb61b29971e9fb34d9c3499b0e96e4e60ed8a5/pyzmq-27.1.0-cp312-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:43ad9a73e3da1fab5b0e7e13402f0b2fb934ae1c876c51d0afff0e7c052eca31", size = 840995, upload-time = "2025-09-08T23:08:08.396Z" },
+    { url = "https://files.pythonhosted.org/packages/c2/bb/b79798ca177b9eb0825b4c9998c6af8cd2a7f15a6a1a4272c1d1a21d382f/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:0de3028d69d4cdc475bfe47a6128eb38d8bc0e8f4d69646adfbcd840facbac28", size = 1642070, upload-time = "2025-09-08T23:08:09.989Z" },
+    { url = "https://files.pythonhosted.org/packages/9c/80/2df2e7977c4ede24c79ae39dcef3899bfc5f34d1ca7a5b24f182c9b7a9ca/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_i686.whl", hash = "sha256:cf44a7763aea9298c0aa7dbf859f87ed7012de8bda0f3977b6fb1d96745df856", size = 2021121, upload-time = "2025-09-08T23:08:11.907Z" },
+    { url = "https://files.pythonhosted.org/packages/46/bd/2d45ad24f5f5ae7e8d01525eb76786fa7557136555cac7d929880519e33a/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:f30f395a9e6fbca195400ce833c731e7b64c3919aa481af4d88c3759e0cb7496", size = 1878550, upload-time = "2025-09-08T23:08:13.513Z" },
+    { url = "https://files.pythonhosted.org/packages/e6/2f/104c0a3c778d7c2ab8190e9db4f62f0b6957b53c9d87db77c284b69f33ea/pyzmq-27.1.0-cp312-abi3-win32.whl", hash = "sha256:250e5436a4ba13885494412b3da5d518cd0d3a278a1ae640e113c073a5f88edd", size = 559184, upload-time = "2025-09-08T23:08:15.163Z" },
+    { url = "https://files.pythonhosted.org/packages/fc/7f/a21b20d577e4100c6a41795842028235998a643b1ad406a6d4163ea8f53e/pyzmq-27.1.0-cp312-abi3-win_amd64.whl", hash = "sha256:9ce490cf1d2ca2ad84733aa1d69ce6855372cb5ce9223802450c9b2a7cba0ccf", size = 619480, upload-time = "2025-09-08T23:08:17.192Z" },
+    { url = "https://files.pythonhosted.org/packages/78/c2/c012beae5f76b72f007a9e91ee9401cb88c51d0f83c6257a03e785c81cc2/pyzmq-27.1.0-cp312-abi3-win_arm64.whl", hash = "sha256:75a2f36223f0d535a0c919e23615fc85a1e23b71f40c7eb43d7b1dedb4d8f15f", size = 552993, upload-time = "2025-09-08T23:08:18.926Z" },
 ]
 
 [[package]]
@@ -925,6 +1323,29 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" },
 ]
 
+[[package]]
+name = "sniffio"
+version = "1.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/87/a6771e1546d97e7e041b6ae58d80074f81b7d5121207425c964ddf5cfdbd/sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc", size = 20372, upload-time = "2024-02-25T23:20:04.057Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" },
+]
+
+[[package]]
+name = "stack-data"
+version = "0.6.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "asttokens" },
+    { name = "executing" },
+    { name = "pure-eval" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/28/e3/55dcc2cfbc3ca9c29519eb6884dd1415ecb53b0e934862d3559ddcb7e20b/stack_data-0.6.3.tar.gz", hash = "sha256:836a778de4fec4dcd1dcd89ed8abff8a221f58308462e1c4aa2a3cf30148f0b9", size = 44707, upload-time = "2023-09-30T13:58:05.479Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/f1/7b/ce1eafaf1a76852e2ec9b22edecf1daa58175c090266e9f6c64afcd81d91/stack_data-0.6.3-py3-none-any.whl", hash = "sha256:d5558e0c25a4cb0853cddad3d77da9891a08cb85dd9f9f91b9f8cd66e511e695", size = 24521, upload-time = "2023-09-30T13:58:03.53Z" },
+]
+
 [[package]]
 name = "starlette"
 version = "1.3.1"
@@ -987,7 +1408,7 @@ wheels = [
 
 [[package]]
 name = "torch"
-version = "2.9.0+cu130"
+version = "2.9.1+cu130"
 source = { registry = "https://download.pytorch.org/whl/cu130" }
 dependencies = [
     { name = "filelock" },
@@ -1015,9 +1436,26 @@ dependencies = [
     { name = "typing-extensions" },
 ]
 wheels = [
-    { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.9.0%2Bcu130-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:3aef05b6247261f4a7c440be9a052c4be36c673c6721920181a4ac9a66d6c2a2", upload-time = "2025-10-14T17:30:43Z" },
-    { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.9.0%2Bcu130-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:cc241ffb20428f6a44c299ca06b934445606cf1fa48f3b68ef3af0a04c86bc3b", upload-time = "2025-10-14T17:30:43Z" },
-    { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.9.0%2Bcu130-cp312-cp312-win_amd64.whl", hash = "sha256:b9979a7c0a1c9544a857fc2390ebc89938f116eaaf6a359a0d46597402ca51da", upload-time = "2025-10-14T17:30:55Z" },
+    { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.9.1%2Bcu130-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:6e7f84cb10c7e7d9f862c318f056d64840544ab4f0bcbf8cf7ed6047fe04051f", upload-time = "2026-01-26T16:37:40Z" },
+    { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.9.1%2Bcu130-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:e70e1b18881e6b3c1ce402d0a989da39f956a3a057526e03c354df23d704ce9b", upload-time = "2026-01-26T16:37:43Z" },
+    { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.9.1%2Bcu130-cp312-cp312-win_amd64.whl", hash = "sha256:cd3232a562ad2a2699d48130255e1b24c07dfe694a40dcd24fad683c752de121", upload-time = "2026-01-26T16:37:50Z" },
+]
+
+[[package]]
+name = "tornado"
+version = "6.5.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/64/24/95ec527ad67b76d59299e5465b3935d05e4294b7e0290a3924b7487df30b/tornado-6.5.7.tar.gz", hash = "sha256:66c513a76cda70d53907bc27cf1447557699c2e95aa48ba27a442ff61c3ddfc2", size = 519252, upload-time = "2026-06-08T17:34:51.232Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/02/dc/c7043cab6fed8ae159fc1923ce829ada35c4dbd797d408a43858ffaf9639/tornado-6.5.7-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:148b2eb15c2c765a50796172c1e499649b35f30d2e3c3d3e15913cfa56bfb163", size = 448543, upload-time = "2026-06-08T17:34:38.052Z" },
+    { url = "https://files.pythonhosted.org/packages/92/4f/090b1431e5a43df696feceffc268c5383cc079ecb5f08ce58f917109aafe/tornado-6.5.7-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:9da38de27f1da3b78a966f0dae12b5a1ea9afe72ca805d84ff06508272ddf100", size = 446707, upload-time = "2026-06-08T17:34:39.594Z" },
+    { url = "https://files.pythonhosted.org/packages/37/d8/ef374952fd5da67d4463122c2b8e5a96536ec10b4b339254c6dcde81d01c/tornado-6.5.7-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:8d759e71906ee783f8867b93bf26a265743da4c1e2f4a018464c1ba019862972", size = 449774, upload-time = "2026-06-08T17:34:41.204Z" },
+    { url = "https://files.pythonhosted.org/packages/35/37/d434c73f4c6e014b745b9b37085f34f40c022f007efff3d7fe65991899f3/tornado-6.5.7-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8a46347a18f23fb92b396beebe0fb78f61dda0cc302445202c16203d8a18848b", size = 450745, upload-time = "2026-06-08T17:34:42.531Z" },
+    { url = "https://files.pythonhosted.org/packages/b6/2b/56b9aff361d7f1ab728a805ec7d7ea835f8807afa9f5cc690ea0e630efb9/tornado-6.5.7-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:7778b30bef919231265e91c69963ce0f49a1e9c07ac900bbe75b19ce2575ba92", size = 450578, upload-time = "2026-06-08T17:34:43.787Z" },
+    { url = "https://files.pythonhosted.org/packages/02/30/a7444fb23aa76860a14198fab96ac79f1866b0a6e19e26c4381b0938e50f/tornado-6.5.7-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e726f0c75da7726eec023aa62751ff8878bd2737e34fbdd33b1ae5897d2200f5", size = 449985, upload-time = "2026-06-08T17:34:45.326Z" },
+    { url = "https://files.pythonhosted.org/packages/5c/42/5f0e56c01e8d9d36f4e23f367b85ae6cae0c1ecddd5e6977d8388ad27488/tornado-6.5.7-cp39-abi3-win32.whl", hash = "sha256:f8de3bf12d3efdd0cbe7c8887868198f8a91415e3f29fcf258d9b8eb7b1d9ae4", size = 451047, upload-time = "2026-06-08T17:34:46.784Z" },
+    { url = "https://files.pythonhosted.org/packages/c9/a4/b393076ffb21b469eec5b328a0534cf03a3b90bfc6b1f09507cdd075d938/tornado-6.5.7-cp39-abi3-win_amd64.whl", hash = "sha256:de942f843533a039ef9fa3d9c88c7cd8a7c94553fb5ad0154270989b3d99a2c4", size = 451485, upload-time = "2026-06-08T17:34:48.248Z" },
+    { url = "https://files.pythonhosted.org/packages/71/2e/7b1c769803121b809112cf9a00681c472eae1d80e32d7ec0e0bd61d0d0e1/tornado-6.5.7-cp39-abi3-win_arm64.whl", hash = "sha256:ff934fce95643af5f11efdae618eaa73d469dc588641e5c8d19295a0c65c4796", size = 450506, upload-time = "2026-06-08T17:34:49.702Z" },
 ]
 
 [[package]]
@@ -1032,6 +1470,15 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/48/5d/acf5905c36149bbaec41ccf7f2b68814647347b72075ac0b1fe3022fdc73/tqdm-4.66.5-py3-none-any.whl", hash = "sha256:90279a3770753eafc9194a0364852159802111925aa30eb3f9d85b0e805ac7cd", size = 78351, upload-time = "2024-08-03T22:35:36.644Z" },
 ]
 
+[[package]]
+name = "traitlets"
+version = "5.15.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/57/a9/a2584b8313b89f94869ddb3c4074617a691de1812a614d2d50e32ca5a7a6/traitlets-5.15.1.tar.gz", hash = "sha256:7b1c07854fe25acb39e009bae49f11b79ff6cbb2f27999104e9110e7a6b53722", size = 163344, upload-time = "2026-06-03T12:26:06.181Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/96/8d/1080ee4c231f361b6ce4470d556c8c435b67c7e0753aaa641497ee92f88b/traitlets-5.15.1-py3-none-any.whl", hash = "sha256:770a53705f84b81ac107e83a1b3328ff2dae16094d8fc3cfc004e4b22dfd8e92", size = 85858, upload-time = "2026-06-03T12:26:04.395Z" },
+]
+
 [[package]]
 name = "transformers"
 version = "5.14.1"
@@ -1054,11 +1501,11 @@ wheels = [
 
 [[package]]
 name = "triton"
-version = "3.5.0"
+version = "3.5.1"
 source = { registry = "https://pypi.org/simple" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/87/9b/30988039e1e84df7554fba24e6a734d2d0e847af33cabdf9b532b3c51456/triton-3.5.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7da21fccceafc163e3a5e857abe34351ef76345af06cabf9637a914742671f0b", size = 159946647, upload-time = "2025-10-15T19:15:56.325Z" },
-    { url = "https://files.pythonhosted.org/packages/f5/3a/e991574f3102147b642e49637e0281e9bb7c4ba254edb2bab78247c85e01/triton-3.5.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c9e71db82261c4ffa3921cd050cd5faa18322d2d405c30eb56084afaff3b0833", size = 170476535, upload-time = "2025-10-13T16:38:05.18Z" },
+    { url = "https://files.pythonhosted.org/packages/db/53/2bcc46879910991f09c063eea07627baef2bc62fe725302ba8f46a2c1ae5/triton-3.5.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:275a045b6ed670dd1bd005c3e6c2d61846c74c66f4512d6f33cc027b11de8fd4", size = 159940689, upload-time = "2025-11-11T17:51:55.938Z" },
+    { url = "https://files.pythonhosted.org/packages/f2/50/9a8358d3ef58162c0a415d173cfb45b67de60176e1024f71fbc4d24c0b6d/triton-3.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d2c6b915a03888ab931a9fd3e55ba36785e1fe70cbea0b40c6ef93b20fc85232", size = 170470207, upload-time = "2025-11-11T17:41:00.253Z" },
 ]
 
 [[package]]
@@ -1127,3 +1574,21 @@ sdist = { url = "https://files.pythonhosted.org/packages/a2/65/b7c6c443ccc58678c
 wheels = [
     { url = "https://files.pythonhosted.org/packages/45/ec/dbb7e5a6b91f86bfb9eb7d2988a2730907b6a729875b949c7f022e8b88fa/uvicorn-0.51.0-py3-none-any.whl", hash = "sha256:5d38af6cd620f2ae3849fb44fd4879e0890aa1febe8d47eb355fb45d93fe6a5b", size = 73219, upload-time = "2026-07-08T10:59:04.44Z" },
 ]
+
+[[package]]
+name = "wcwidth"
+version = "0.8.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/34/74/c6428f875774288bec1396f5bfcbc2d925700a4dad61727fd5f2b12f249d/wcwidth-0.8.2.tar.gz", hash = "sha256:91fbef97204b96a3d4d421609b80340b760cf33e26da123ff243d76b1fda8dda", size = 1466253, upload-time = "2026-06-29T18:11:11.601Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/96/42/3e5985a0a7e57de470b320c6d6a1a67c844f6737a587f3d44dd13d1819e7/wcwidth-0.8.2-py3-none-any.whl", hash = "sha256:d63947694a0539a1d51e01eda7caf800c291020e6cdd7e28ad7b14dd33ad4f85", size = 323166, upload-time = "2026-06-29T18:11:09.888Z" },
+]
+
+[[package]]
+name = "widgetsnbextension"
+version = "4.0.15"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/bd/f4/c67440c7fb409a71b7404b7aefcd7569a9c0d6bd071299bf4198ae7a5d95/widgetsnbextension-4.0.15.tar.gz", hash = "sha256:de8610639996f1567952d763a5a41af8af37f2575a41f9852a38f947eb82a3b9", size = 1097402, upload-time = "2025-11-01T21:15:55.178Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/3f/0e/fa3b193432cfc60c93b42f3be03365f5f909d2b3ea410295cf36df739e31/widgetsnbextension-4.0.15-py3-none-any.whl", hash = "sha256:8156704e4346a571d9ce73b84bee86a29906c9abfd7223b7228a28899ccf3366", size = 2196503, upload-time = "2025-11-01T21:15:53.565Z" },
+]