Add model files

doc_pipeline/README.md

@@ -0,0 +1,162 @@
+# pdf_pipeline
+
+A PDF → Markdown extraction pipeline implementing this flow:
+
+```
+pdf
+ └─ pdfplumber ──────────────┬──────────────────────────────┐
+        │ words+bboxes(xyxy) │ image                        │
+        ▼                    ▼                              │
+  empty words?          pp-doclayout                         │
+   yes │   no            (DocLayoutV3)                       │
+       │     └────────────┐  │ layout_class+bbox+order       │
+       ▼                  ▼  ▼                                
+  OCR(pytesseract/   Plumber words + pp-doclayout matching    
+  paddleocr/rapidocr)  (align_words_to_layout)                
+       │                  │                                   
+       ▼                  ▼                                   
+  Matching ocr bboxes   (class_name, bbox, reading_order, text)
+  and layout                       │
+  (align_ocr_to_layout)            ▼
+       │                   class_name == "table"?
+       │                    yes │      no
+       └───────────┬───────────┘       └─ formatted by class
+                    ▼
+            TableFormerONNX → OTSL → markdown
+```
+
+Both matching branches converge on the same `RegionText` shape and the same
+downstream per-region rendering (`region_to_markdown`), including the
+table → TableFormerONNX → OTSL → markdown branch — the diagram only draws
+that branch coming off the pdfplumber-matching box, but a "table" layout
+region can equally appear on a scanned (OCR'd) page, so it's handled the
+same way in both paths.
+
+## Modules
+
+| File                  | Diagram piece(s) |
+|------------------------|------------------|
+| `layout.py`            | `pp-doclayout` (DocLayoutV3 ONNX wrapper) + both "matching" boxes (`align_words_to_layout`, `align_ocr_to_layout`, sharing the generic `align_tokens_to_layout` core) |
+| `ocr_backends.py`      | `Ocr(pytesseract, paddleocr, rapidocr)` — also reused for table-cell OCR |
+| `table_extraction.py`  | `TableFormerONNX` → OTSL → markdown |
+| `pipeline.py`          | The `if the words are empty` branch, the `if classname is table` branch, and final markdown assembly (`process_pdf_page`, `process_pdf`) |
+
+## What's new vs. the two source notebooks
+
+The two original notebooks (`pdfplumber_.ipynb`, `docling_.ipynb`) already
+had working code for: layout detection, the pdfplumber↔layout matcher, OCR
+backends, and TableFormer→OTSL. What was missing, per the diagram, was:
+
+- **The OCR-fallback matching box.** `align_words_to_layout` only knew how
+  to consume pdfplumber's `extract_words()` output. `align_tokens_to_layout`
+  generalizes the matching algorithm (containment + nearest-centroid
+  fallback + line re-assembly) to take any `text + bbox` token list, in
+  image-pixel space. `align_words_to_layout` and `align_ocr_to_layout` are
+  now both adapters onto that one function.
+- **The `if classname is table` branch wired into the page loop.** The
+  notebook's per-region markdown loop only ever embedded a cropped image for
+  `table`/`image`/`chart` regions; TableFormer was only run manually,
+  separately, against one standalone table image. `region_to_markdown` now
+  checks `class_name == "table"` and runs `table_image_to_otsl` +
+  `otsl_to_markdown` automatically, falling back to an embedded image if no
+  `TableFormerONNX` runner was supplied (or if extraction throws).
+- **Table cells prefer pdfplumber's own words over OCR.** When the PDF has a
+  real text layer, the words pdfplumber already extracted for a table region
+  are reused directly as TableFormer's cell-text source (`tokens=` on
+  `ocr_anchor_cells`/`table_image_to_otsl`) instead of re-OCR-ing the table
+  crop — cheaper and avoids OCR mistakes on text that's already exact.
+  `table_ocr_backend` is now only invoked as a fallback when a table region
+  has no underlying words (e.g. the table is itself a scanned image, or the
+  whole page is scanned and went through the page-level OCR branch).
+- **Standalone image input.** `process_image_page`/`process_images` run the
+  same diagram on plain images (.jpg/.png/etc. — a photographed or scanned
+  page with no PDF structure at all), always via the OCR branch since
+  there's no native text layer to check. `process_document` is a single
+  dispatcher that picks the PDF path or the image path based on the input,
+  so callers don't need to branch on file type themselves. Both share the
+  same per-image core (`_process_rendered_page`) as the PDF path, so a
+  scanned PDF page and a standalone photo of a page are handled identically
+  once OCR kicks in.
+- **Fixed a latent color-channel bug along the way.** `DocLayoutV3._preprocess`
+  always applies a BGR→RGB swap, correct only if its input really is BGR
+  (as `cv2.imread` produces). The original page-rendering code fed it a
+  PIL-derived array (RGB) directly, which silently double-swapped the
+  channels and degraded detection on every PDF page processed through this
+  pipeline. Fixed via a shared `_to_bgr_array` helper used by both the PDF
+  and image entry points.
+- **`otsl_to_markdown`.** The OTSL→DocTagsDocument→DoclingDocument→markdown
+  steps were ad hoc, later notebook cells, applied to one hardcoded `otsl`
+  variable. Factored into one function so it can run once per detected
+  table inside a page loop.
+
+## Requirements
+
+```
+pip install pdfplumber numpy opencv-python onnxruntime pydantic pillow docling-core
+# plus whichever OCR backend(s) you use:
+pip install pytesseract            # needs the tesseract binary too
+pip install rapidocr-onnxruntime
+pip install paddleocr paddlepaddle
+```
+
+You'll also need the model artifacts referenced in the original notebooks:
+- `PP-DocLayout/PP-DocLayoutV3.onnx`
+- `tableformerv1/onnx/<variant>/tableformer_<variant>_{encoder,decoder_step,bbox_decoder}.onnx`
+  + `tableformerv1/tm_config.json`
+
+## Usage
+
+```python
+from pdf_pipeline import DocLayoutV3, TableFormerONNX, get_ocr_backend, process_document
+
+layout_detector = DocLayoutV3("PP-DocLayout/PP-DocLayoutV3.onnx")
+page_ocr_backend = get_ocr_backend("rapidocr")          # used on scanned PDF pages AND any standalone image
+table_runner = TableFormerONNX(artifact_root="tableformerv1")
+table_ocr_backend = get_ocr_backend("paddleocr")          # fallback for tables with no underlying text
+
+# Works the same regardless of input type:
+markdown_doc = process_document(
+    "document.pdf",                       # or "scan.jpg", or ["page1.png", "page2.png"]
+    layout_detector,
+    page_ocr_backend=page_ocr_backend,
+    table_runner=table_runner,
+    table_ocr_backend=table_ocr_backend,
+)
+
+with open("document.md", "w") as f:
+    f.write(markdown_doc)
+```
+
+`process_document` dispatches on what you pass it: a `.pdf` path goes
+through `process_pdf` (pdfplumber, may have a native text layer); a single
+image path (`.jpg`/`.png`/etc.) or a list of image paths goes through
+`process_image_page`/`process_images` (always OCR, since standalone images
+never have a text layer). Call `process_pdf`/`process_image_page` directly
+if you want the type-specific kwargs (e.g. `pages=` or `resolution=` only
+make sense for PDFs) or the full per-page debug dict instead of just the
+markdown string.
+
+For single-page debugging (mirrors the original notebook's main cell), use
+`process_pdf_page(page, layout_detector, doc_path, ...)` directly — it
+returns a dict with `markdown`, `regions`, `layout_result`, `image`, and
+`used_ocr`, so you can inspect intermediate state (e.g. call
+`layout_detector.visualize(result["layout_result"], "debug.png", result["image"])`)
+before trusting the final markdown.
+
+Cropped images (for `image`/`chart`/`table`-without-TableFormer regions) are
+written to `<doc_stem>/images/<n>.png`, same convention as the original
+notebook's `crop_and_save_image`.
+
+## Notes / things to tune for your documents
+
+- `containment_threshold` / `line_tol_ratio` (passed through `process_pdf_page`
+  → `align_words_to_layout` / `align_ocr_to_layout` → `align_tokens_to_layout`)
+  control how aggressively tokens are assigned to boxes and how lines are
+  re-grouped; the defaults (0.5 / 0.5) come from the original notebook.
+- `SKIP_CLASSES` and `IMAGE_LIKE_CLASSES` in `pipeline.py` control which
+  layout classes are dropped vs. rendered as a markdown image link vs.
+  rendered as text — extend these for your document types (e.g. treat
+  `display_formula`/`inline_formula` specially instead of falling through
+  to plain text).
+- If `table_runner`/`table_ocr_backend` are omitted, table regions degrade
+  gracefully to an embedded cropped image instead of raising.

doc_pipeline/__init__.py

@@ -0,0 +1,87 @@
+"""
+pdf_pipeline
+============
+
+PDF -> markdown extraction pipeline:
+
+    pdfplumber (words+bboxes, image)
+        -> [empty words? -> OCR] -> match against pp-doclayout boxes
+        -> per-region markdown, with table regions routed through TableFormerONNX -> OTSL -> markdown
+
+See README.md for the full architecture diagram and a usage walkthrough.
+"""
+
+from .layout import (
+    BBox,
+    DocLayoutResult,
+    DocLayoutV3,
+    LayoutBox,
+    PlacedWord,
+    RegionText,
+    align_ocr_to_layout,
+    align_tokens_to_layout,
+    align_words_to_layout,
+    log_layout_result,
+    pdfplumber_tokens_in_image_space,
+    print_layout_result,
+)
+from .logging_config import setup_pipeline_logging
+from .ocr_backends import (
+    OCRBackend,
+    PytesseractBackend,
+    RapidOCRBackend,
+    get_ocr_backend,
+)
+from .pipeline import (
+    IMAGE_EXTENSIONS,
+    crop_and_save_image,
+    process_document,
+    process_images,
+    process_image_page,
+    process_pdf,
+    process_pdf_page,
+    region_to_markdown,
+    regions_to_markdown,
+)
+from .table_extraction import (
+    TableFormerONNX,
+    ocr_anchor_cells,
+    otsl_to_markdown,
+    seq_to_otsl,
+    table_image_to_otsl,
+)
+
+__all__ = [
+    "BBox",
+    "DocLayoutResult",
+    "DocLayoutV3",
+    "LayoutBox",
+    "PlacedWord",
+    "RegionText",
+    "align_ocr_to_layout",
+    "align_tokens_to_layout",
+    "align_words_to_layout",
+    "pdfplumber_tokens_in_image_space",
+    "log_layout_result",
+    "print_layout_result",
+    "OCRBackend",
+    "PaddleOCRBackend",
+    "PytesseractBackend",
+    "RapidOCRBackend",
+    "get_ocr_backend",
+    "crop_and_save_image",
+    "IMAGE_EXTENSIONS",
+    "process_document",
+    "process_images",
+    "process_image_page",
+    "process_pdf",
+    "process_pdf_page",
+    "region_to_markdown",
+    "regions_to_markdown",
+    "TableFormerONNX",
+    "ocr_anchor_cells",
+    "otsl_to_markdown",
+    "seq_to_otsl",
+    "table_image_to_otsl",
+    "setup_pipeline_logging",
+]

doc_pipeline/layout.py

@@ -0,0 +1,672 @@
+"""
+layout.py
+=========
+
+Two responsibilities, matching the top half of the pipeline diagram:
+
+1. `DocLayoutV3` — thin ONNXRuntime wrapper around PP-DocLayoutV3. Given a
+   page image it returns layout boxes: (class_name, bbox[xyxy], reading_order).
+   This is unchanged from the original pdfplumber_.ipynb.
+
+2. Matching: turn a flat list of "tokens" (word/line-level text + bbox) into
+   per-layout-region text, by containment + nearest-centroid fallback, then
+   re-assembling lines within each region.
+
+   The original notebook only had `align_words_to_layout`, which is hard-wired
+   to pdfplumber's `page.extract_words()` output (point-space, needs scaling
+   to image pixels). The diagram has a *second*, parallel matching box
+   ("Matching ocr bboxes and layout") for the branch where pdfplumber finds no
+   extractable words and an OCR backend is used instead. OCR tokens are
+   already in image-pixel space (OCR runs on the same rendered image that was
+   fed to PP-DocLayout), so no scaling is needed there.
+
+   Rather than duplicating the matching logic, `align_tokens_to_layout` is the
+   single generic implementation. `align_words_to_layout` and
+   `align_ocr_to_layout` are thin adapters that normalize their respective
+   inputs into the same token shape and call it.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+logger = logging.getLogger(__name__)
+
+import cv2
+import numpy as np
+from pydantic import BaseModel, Field
+
+try:
+    import onnxruntime as ort
+except ImportError:  # pragma: no cover - allows importing this module for
+    ort = None        # static inspection without onnxruntime installed.
+
+
+LAYOUT_CLASSES = [
+    "abstract",
+    "algorithm",
+    "aside_text",
+    "chart",
+    "content",
+    "display_formula",
+    "doc_title",
+    "figure_title",
+    "footer",
+    "footer_image",
+    "footnote",
+    "formula_number",
+    "header",
+    "header_image",
+    "image",
+    "inline_formula",
+    "number",
+    "paragraph_title",
+    "reference",
+    "reference_content",
+    "seal",
+    "table",
+    "text",
+    "vertical_text",
+    "vision_footnote",
+]
+
+
+# --------------------------------------------------------------------------- #
+# Layout detection result types
+# --------------------------------------------------------------------------- #
+class BBox(BaseModel):
+    """Axis-aligned bounding box in original image pixel coordinates."""
+
+    xmin: float
+    ymin: float
+    xmax: float
+    ymax: float
+
+    def as_tuple(self) -> Tuple[float, float, float, float]:
+        return (self.xmin, self.ymin, self.xmax, self.ymax)
+
+
+class LayoutBox(BaseModel):
+    """A single detected layout element."""
+
+    class_id: int
+    class_name: str
+    score: float
+    bbox: BBox
+    reading_order: int
+
+    model_config = {"frozen": True}
+
+
+class DocLayoutResult(BaseModel):
+    """Full result for one image."""
+
+    image_path: Optional[str] = None
+    image_width: int
+    image_height: int
+    boxes: List[LayoutBox] = Field(default_factory=list)
+
+    def sorted_by_reading_order(self) -> List[LayoutBox]:
+        return sorted(self.boxes, key=lambda b: b.reading_order)
+
+
+# --------------------------------------------------------------------------- #
+# PP-DocLayoutV3 ONNX wrapper
+# --------------------------------------------------------------------------- #
+
+class DocLayoutV3:
+    """
+    Thin OO wrapper around the PP-DocLayoutV3 ONNX model.
+
+    Usage:
+        detector = DocLayoutV3("PP-DocLayout/PP-DocLayoutV3.onnx")
+        result = detector.predict("page.png")
+        for box in result.sorted_by_reading_order():
+            print(box.class_name, box.bbox, box.reading_order)
+
+        # optional: draw + save a visualization
+        detector.visualize(result, "pp_doclayout.png")
+    """
+
+    def __init__(
+        self,
+        model_path: Union[str, Path],
+        target_input_size: Tuple[int, int] = (800, 800),
+        score_threshold: float = 0.5,
+        class_names: Optional[List[str]] = None,
+        providers: Optional[List[str]] = None,
+        intra_op_num_threads: Optional[int] = None,
+        inter_op_num_threads: Optional[int] = None,
+        enable_mem_pattern: bool = True,
+        graph_optimization_level: "ort.GraphOptimizationLevel" = None,
+    ):
+        if ort is None:
+            raise ImportError("onnxruntime is required for DocLayoutV3")
+
+        graph_optimization_level = (
+            graph_optimization_level or ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+        )
+
+        self.model_path = str(model_path)
+        self.target_input_size = target_input_size
+        self.score_threshold = score_threshold
+        self.class_names = class_names or LAYOUT_CLASSES
+
+        self.session = self._build_session(
+            providers=providers,
+            intra_op_num_threads=intra_op_num_threads,
+            inter_op_num_threads=inter_op_num_threads,
+            enable_mem_pattern=enable_mem_pattern,
+            graph_optimization_level=graph_optimization_level,
+        )
+        self.providers_used = self.session.get_providers()
+
+        self.input_names = [i.name for i in self.session.get_inputs()]
+        self.output_names = [o.name for o in self.session.get_outputs()]
+
+        self._mean = np.array([0.0, 0.0, 0.0], dtype=np.float32)
+        self._std =     std = np.array([1.0, 1.0, 1.0], dtype=np.float32)
+
+
+    # ------------------------------------------------------------------ #
+    def _build_session(
+        self,
+        providers: Optional[List[str]],
+        intra_op_num_threads: Optional[int],
+        inter_op_num_threads: Optional[int],
+        enable_mem_pattern: bool,
+        graph_optimization_level,
+    ) -> "ort.InferenceSession":
+        available = ort.get_available_providers()
+
+        if providers is not None:
+            resolved = [p for p in providers if p in available]
+            if not resolved:
+                raise ValueError(
+                    f"None of the requested providers {providers} are "
+                    f"available. Available providers: {available}"
+                )
+        else:
+            preferred_order = ["CUDAExecutionProvider", "CPUExecutionProvider"]
+            resolved = [p for p in preferred_order if p in available]
+            if not resolved:
+                resolved = available or ["CPUExecutionProvider"]
+
+        sess_options = ort.SessionOptions()
+        sess_options.graph_optimization_level = graph_optimization_level
+        sess_options.enable_mem_pattern = enable_mem_pattern
+        sess_options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
+
+        if intra_op_num_threads is not None:
+            sess_options.intra_op_num_threads = intra_op_num_threads
+        if inter_op_num_threads is not None:
+            sess_options.inter_op_num_threads = inter_op_num_threads
+
+        session = ort.InferenceSession(
+            self.model_path,
+            sess_options=sess_options,
+            providers=resolved,
+        )
+
+        actual = session.get_providers()
+        if resolved[0] not in actual:
+            import warnings
+
+            warnings.warn(
+                f"Requested provider order {resolved} but session is "
+                f"actually running on {actual}.",
+                RuntimeWarning,
+                stacklevel=2,
+            )
+
+        return session
+    
+    #--------------------------------------------------------------------#
+    def add_border_to_image(self, image: np.ndarray) -> np.ndarray:
+        """
+        Add a border to the image to ensure it is square.
+        """
+        orig_h, orig_w = image.shape[:2]
+        if orig_h > orig_w:
+            pad_h = (orig_h - orig_w) // 2
+            pad_v = (orig_h - orig_w) // 2
+        else:
+            pad_h = (orig_w - orig_h) // 2
+            pad_v = (orig_w - orig_h) // 2
+        return cv2.copyMakeBorder(
+            image,
+            pad_v,
+            pad_v,
+            pad_h,
+            pad_h,
+            cv2.BORDER_CONSTANT,
+            value=(0, 0, 0),
+        )
+        height, width = image.shape[:2]
+        max_dim = max(height, width)
+        border = (max_dim - height) // 2, (max_dim - width) // 2
+        image = cv2.copyMakeBorder(
+            image,
+            border[0],
+            border[0],  
+            border[1],
+            border[1],
+            cv2.BORDER_CONSTANT,
+            value=[255, 255, 255],
+        )
+        return image    
+    # ------------------------------------------------------------------ #
+    def median_filter(self, image:np.ndarray):
+        """
+        Apply median filter to the image.
+        """
+        return cv2.medianBlur(image, 3)
+    def _preprocess(self, image: np.ndarray) -> Tuple[np.ndarray, float, float]:
+        orig_h, orig_w = image.shape[:2]
+        target_h, target_w = self.target_input_size
+        #image = self.add_border_to_image(image)
+        
+
+        scale_h = target_h / orig_h
+        scale_w = target_w / orig_w
+
+        
+        resized = cv2.resize(image, (target_w, target_h), interpolation=cv2.INTER_CUBIC)
+
+        rgb = cv2.cvtColor(resized, cv2.COLOR_BGR2RGB)
+        blob = rgb.astype(np.float32) / 255.0
+        blob = (blob - self._mean) / self._std
+
+        blob = blob.transpose(2, 0, 1)[np.newaxis, ...]
+        return blob, scale_h, scale_w
+
+    def _infer(self, image: np.ndarray) -> Tuple[np.ndarray, int, int]:
+        orig_h, orig_w = image.shape[:2]
+        input_blob, scale_h, scale_w = self._preprocess(image)
+
+        target_h, target_w = self.target_input_size
+        preprocess_shape = [np.array([target_h, target_w], dtype=np.float32)]
+
+        input_feed = {
+            self.input_names[0]: preprocess_shape,
+            self.input_names[1]: input_blob,
+            self.input_names[2]: [[scale_h, scale_w]],
+        }
+
+        # shape=(N, 7): [label_index, score, xmin, ymin, xmax, ymax, read_order]
+        output = self.session.run(self.output_names, input_feed)[0]
+        return output, orig_h, orig_w
+
+    def _to_result(
+        self,
+        raw_boxes: np.ndarray,
+        orig_h: int,
+        orig_w: int,
+        image_path: Optional[str],
+    ) -> DocLayoutResult:
+        logger.info(f"raw_boxes: {raw_boxes}")
+        
+        filtered = raw_boxes[raw_boxes[:, 1] > self.score_threshold]
+        filtered = filtered[np.argsort(filtered[:, 5])]
+        
+
+        boxes: List[LayoutBox] = []
+        for row in filtered:
+            cls_id = int(row[0])
+            boxes.append(
+                LayoutBox(
+                    class_id=cls_id,
+                    class_name=self.class_names[cls_id]
+                    if 0 <= cls_id < len(self.class_names)
+                    else str(cls_id),
+                    score=float(row[1]),
+                    bbox=BBox(
+                        xmin=float(row[2]),
+                        ymin=float(row[3]),
+                        xmax=float(row[4]),
+                        ymax=float(row[5]),
+                    ),
+                    reading_order=int(row[6]),
+                )
+            )
+
+        result = DocLayoutResult(
+            image_path=image_path,
+            image_width=orig_w,
+            image_height=orig_h,
+            boxes=boxes,
+        )
+        logger.info(
+            "DocLayoutV3: %dx%d image -> %d boxes (threshold=%.2f)",
+            orig_w,
+            orig_h,
+            len(boxes),
+            self.score_threshold,
+        )
+        return result
+
+    # ------------------------------------------------------------------ #
+    # Public API
+    # ------------------------------------------------------------------ #
+    def predict(self, image: Union[str, Path, np.ndarray]) -> DocLayoutResult:
+        image_path: Optional[str] = None
+        if isinstance(image, (str, Path)):
+            image_path = str(image)
+            img_array = cv2.imread(image_path)
+            if img_array is None:
+                raise FileNotFoundError(f"Could not read image: {image_path}")
+        else:
+            img_array = image
+
+        raw_boxes, orig_h, orig_w = self._infer(img_array)
+        return self._to_result(raw_boxes, orig_h, orig_w, image_path)
+
+    def predict_batch(
+        self, images: List[Union[str, Path, np.ndarray]]
+    ) -> List[DocLayoutResult]:
+        return [self.predict(img) for img in images]
+
+    def visualize(
+        self,
+        result: DocLayoutResult,
+        output_path: Union[str, Path],
+        source_image: Optional[Union[str, Path, np.ndarray]] = None,
+    ) -> str:
+        src = source_image if source_image is not None else result.image_path
+        if src is None:
+            raise ValueError(
+                "No source image available to draw on; pass source_image=..."
+            )
+
+        if isinstance(src, (str, Path)):
+            img = cv2.imread(str(src))
+            if img is None:
+                raise FileNotFoundError(f"Could not read image: {src}")
+        else:
+            img = src.copy()
+
+        for box in result.boxes:
+            x0, y0, x1, y1 = (int(round(v)) for v in box.bbox.as_tuple())
+            cv2.rectangle(img, (x0, y0), (x1, y1), (0, 0, 255), 2)
+            label = f"{box.reading_order}|{box.class_name}"
+            text_y = max(y0 - 10, 0)
+            cv2.putText(
+                img,
+                label,
+                (x0, text_y),
+                cv2.FONT_HERSHEY_SIMPLEX,
+                0.5,
+                (255, 0, 0),
+                1,
+                cv2.LINE_AA,
+            )
+
+        output_path = str(output_path)
+        cv2.imwrite(output_path, img)
+        return output_path
+
+
+def log_layout_result(result: DocLayoutResult) -> None:
+    """Log every detected layout box at INFO level."""
+    for box in result.sorted_by_reading_order():
+        b = box.bbox
+        logger.info(
+            "  layout [%d] %s score=%.3f bbox=(%.0f, %.0f, %.0f, %.0f)",
+            box.reading_order,
+            box.class_name,
+            box.score,
+            b.xmin,
+            b.ymin,
+            b.xmax,
+            b.ymax,
+        )
+
+
+def print_layout_result(result: DocLayoutResult) -> None:
+    """Print layout boxes to stdout (legacy helper; prefer ``log_layout_result``)."""
+    log_layout_result(result)
+
+
+# --------------------------------------------------------------------------- #
+# Token <-> layout matching (the two "matching" boxes in the diagram)
+# --------------------------------------------------------------------------- #
+@dataclass(frozen=True)
+class PlacedWord:
+    text: str
+    x0: float
+    x1: float
+    top: float
+    bottom: float
+
+
+@dataclass
+class RegionText:
+    reading_order: int
+    class_name: str
+    bbox: tuple  # (xmin, ymin, xmax, ymax) in image-pixel space
+    text: str
+    words: List[PlacedWord]
+
+
+def _line_cluster_fast(words: List[PlacedWord], tol: float) -> List[List[PlacedWord]]:
+    """Group words into lines by `top` proximity, then sort each line left-to-right.
+    `tol` is computed once globally by the caller instead of per-region."""
+    if not words:
+        return []
+    ordered = sorted(words, key=lambda w: w.top)
+    lines: List[List[PlacedWord]] = []
+    current = [ordered[0]]
+    current_top = ordered[0].top
+    for w in ordered[1:]:
+        if abs(w.top - current_top) <= tol:
+            current.append(w)
+        else:
+            lines.append(current)
+            current = [w]
+            current_top = w.top
+    lines.append(current)
+    for line in lines:
+        line.sort(key=lambda w: w.x0)
+    return lines
+
+
+def align_tokens_to_layout(
+    tokens: Sequence[PlacedWord],
+    layout_result: DocLayoutResult,
+    containment_threshold: float = 0.5,
+    line_tol_ratio: float = 0.5,
+) -> List[RegionText]:
+    """
+    Generic matcher: given tokens already expressed in image-pixel space
+    (same coordinate system as `layout_result`), assign each token to its
+    best-matching layout box (smallest box containing it above
+    `containment_threshold`, falling back to nearest centroid for tokens
+    that don't sit inside any box), then re-assemble lines of text per box.
+
+    This is the shared core behind both `align_words_to_layout` (pdfplumber
+    path) and `align_ocr_to_layout` (OCR fallback path) in the diagram.
+    """
+    boxes = layout_result.boxes
+    if not boxes or not tokens:
+        logger.debug(
+            "align_tokens_to_layout: skipped (boxes=%d, tokens=%d)",
+            len(boxes) if boxes else 0,
+            len(tokens) if tokens else 0,
+        )
+        return []
+
+    logger.debug(
+        "align_tokens_to_layout: %d tokens -> %d layout boxes (containment=%.2f)",
+        len(tokens),
+        len(boxes),
+        containment_threshold,
+    )
+
+    wx0 = np.fromiter((t.x0 for t in tokens), dtype=np.float64, count=len(tokens))
+    wx1 = np.fromiter((t.x1 for t in tokens), dtype=np.float64, count=len(tokens))
+    wtop = np.fromiter((t.top for t in tokens), dtype=np.float64, count=len(tokens))
+    wbot = np.fromiter((t.bottom for t in tokens), dtype=np.float64, count=len(tokens))
+    word_area = np.clip((wx1 - wx0) * (wbot - wtop), 1e-6, None)
+
+    bx0 = np.array([b.bbox.xmin for b in boxes])
+    by0 = np.array([b.bbox.ymin for b in boxes])
+    bx1 = np.array([b.bbox.xmax for b in boxes])
+    by1 = np.array([b.bbox.ymax for b in boxes])
+    box_area = np.clip((bx1 - bx0) * (by1 - by0), 1e-6, None)
+
+    ix0 = np.maximum(wx0[:, None], bx0[None, :])
+    iy0 = np.maximum(wtop[:, None], by0[None, :])
+    ix1 = np.minimum(wx1[:, None], bx1[None, :])
+    iy1 = np.minimum(wbot[:, None], by1[None, :])
+    inter = np.clip(ix1 - ix0, 0, None) * np.clip(iy1 - iy0, 0, None)
+    containment = inter / word_area[:, None]
+
+    masked_area = np.where(containment >= containment_threshold, box_area[None, :], np.inf)
+    best = np.argmin(masked_area, axis=1)
+    matched = np.isfinite(masked_area[np.arange(len(tokens)), best])
+    if not matched.all():
+        bcx, bcy = (bx0 + bx1) / 2.0, (by0 + by1) / 2.0
+        wcx, wcy = (wx0 + wx1) / 2.0, (wtop + wbot) / 2.0
+        dist = (wcx[:, None] - bcx[None, :]) ** 2 + (wcy[:, None] - bcy[None, :]) ** 2
+        best = np.where(matched, best, np.argmin(dist, axis=1))
+
+    global_tol = float(np.median(wbot - wtop)) * line_tol_ratio
+
+    order = np.argsort(best, kind="stable")
+    sorted_idx = best[order]
+    split_points = np.searchsorted(sorted_idx, np.arange(len(boxes)))
+    split_points = np.append(split_points, len(order))
+
+    results = []
+    for box_idx, box in enumerate(boxes):
+        idxs = order[split_points[box_idx]:split_points[box_idx + 1]]
+        box_words = [tokens[i] for i in idxs]
+        lines = _line_cluster_fast(box_words, global_tol)
+        text = "\n".join(" ".join(w.text for w in line) for line in lines)
+        results.append(
+            RegionText(box.reading_order, box.class_name, box.bbox.as_tuple(), text, box_words)
+        )
+        logger.debug(
+            "  matched box [%d] %s: %d words, text_len=%d",
+            box.reading_order,
+            box.class_name,
+            len(box_words),
+            len(text),
+        )
+    results.sort(key=lambda r: r.reading_order)
+    return results
+
+
+def pdfplumber_tokens_in_image_space(
+    page, image_width: int, image_height: int
+) -> List[PlacedWord]:
+    """
+    Extract a pdfplumber page's words and scale them from PDF point-space
+    into the pixel space of a page image rendered at
+    (image_width, image_height) — e.g. via `page.to_image(resolution=...)`,
+    the same image PP-DocLayout ran on.
+
+    Returns `[]` if `extract_words()` finds nothing (a scanned page with no
+    text layer). Factored out of `align_words_to_layout` so callers that need
+    these same scaled tokens for something else (e.g. pulling exact words for
+    a table region instead of OCR-ing it) don't have to re-derive them.
+    """
+    raw_words = page.extract_words(
+        x_tolerance=2,
+        y_tolerance=3,
+        keep_blank_chars=False,
+        use_text_flow=False,
+    )
+    if not raw_words:
+        logger.debug("pdfplumber extract_words: 0 words (scanned or image-only page)")
+        return []
+
+    scale_x = image_width / page.width
+    scale_y = image_height / page.height
+    logger.debug(
+        "pdfplumber extract_words: %d words scaled to %dx%d (scale %.3f, %.3f)",
+        len(raw_words),
+        image_width,
+        image_height,
+        scale_x,
+        scale_y,
+    )
+    return [
+        PlacedWord(
+            text=w["text"],
+            x0=w["x0"] * scale_x,
+            x1=w["x1"] * scale_x,
+            top=w["top"] * scale_y,
+            bottom=w["bottom"] * scale_y,
+        )
+        for w in raw_words
+    ]
+
+
+def align_words_to_layout(
+    layout_result: DocLayoutResult,
+    page,
+    containment_threshold: float = 0.5,
+    line_tol_ratio: float = 0.5,
+) -> List[RegionText]:
+    """
+    Plumber-path matcher ("Plumber_bboxes_words and pp-doclayout matching" box).
+
+    `page` is a pdfplumber Page. Its `extract_words()` output is in PDF point
+    space, so it's scaled up to the layout image's pixel space before matching.
+    """
+    if not layout_result.boxes:
+        return []
+
+    tokens = pdfplumber_tokens_in_image_space(
+        page, layout_result.image_width, layout_result.image_height
+    )
+    if not tokens:
+        return []
+    return align_tokens_to_layout(tokens, layout_result, containment_threshold, line_tol_ratio)
+
+
+def align_ocr_to_layout(
+    layout_result: DocLayoutResult,
+    ocr_tokens: Sequence[Dict],
+    containment_threshold: float = 0.5,
+    line_tol_ratio: float = 0.5,
+) -> List[RegionText]:
+    """
+    OCR-fallback matcher ("Matching ocr bboxes and layout" box) — used when
+    pdfplumber's `extract_words()` returns nothing (e.g. a scanned page).
+
+    `ocr_tokens` is the output of an `OCRBackend.get_text_boxes(image)` call
+    (see ocr_backends.py): a list of {"text": str, "bbox": [x1, y1, x2, y2]}
+    dicts already in the rendered image's pixel space — the same space the
+    layout image was detected in — so no scaling is required here.
+    """
+    if not layout_result.boxes or not ocr_tokens:
+        logger.debug(
+            "align_ocr_to_layout: skipped (boxes=%d, ocr_tokens=%d)",
+            len(layout_result.boxes),
+            len(ocr_tokens),
+        )
+        return []
+
+    logger.info(
+        "OCR matching: %d OCR tokens -> %d layout boxes",
+        len(ocr_tokens),
+        len(layout_result.boxes),
+    )
+
+    tokens = [
+        PlacedWord(
+            text=t["text"],
+            x0=float(t["bbox"][0]),
+            x1=float(t["bbox"][2]),
+            top=float(t["bbox"][1]),
+            bottom=float(t["bbox"][3]),
+        )
+        for t in ocr_tokens
+        if t.get("text")
+    ]
+    return align_tokens_to_layout(tokens, layout_result, containment_threshold, line_tol_ratio)

doc_pipeline/logging_config.py

@@ -0,0 +1,44 @@
+"""
+logging_config.py
+=================
+
+Central logging setup for the pdf_pipeline package. Call ``setup_pipeline_logging``
+once at process start (e.g. from ``example_usage.py``) to enable step-by-step
+logs from pipeline, layout, OCR, and table extraction modules.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Optional, Union
+
+
+def setup_pipeline_logging(
+    level: Union[int, str] = logging.INFO,
+    log_file: Optional[str] = None,
+) -> None:
+    """
+    Configure root logging for the pipeline.
+
+    Parameters
+    ----------
+    level:
+        Logging level (``logging.DEBUG``, ``logging.INFO``, etc.) or a level name
+        such as ``"DEBUG"``.
+    log_file:
+        Optional path to also write logs to a file.
+    """
+    if isinstance(level, str):
+        level = getattr(logging, level.upper(), logging.INFO)
+
+    fmt = "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s"
+    handlers: list[logging.Handler] = [logging.StreamHandler()]
+    if log_file:
+        handlers.append(logging.FileHandler(log_file, encoding="utf-8"))
+
+    logging.basicConfig(level=level, format=fmt, handlers=handlers, force=True)
+
+    # Keep noisy third-party libraries quieter unless DEBUG is requested.
+    if level > logging.DEBUG:
+        for name in ("PIL", "onnxruntime", "urllib3"):
+            logging.getLogger(name).setLevel(logging.WARNING)

doc_pipeline/ocr_backends.py

@@ -0,0 +1,112 @@
+"""
+ocr_backends.py
+================
+
+OCR backends with a common interface: `get_text_boxes(image) -> list[dict]`,
+each dict being `{"text": str, "bbox": [x1, y1, x2, y2]}` in the input
+image's pixel space.
+
+These backends are used in *two* places in the pipeline (diagram):
+
+1. "Ocr(pytesseract, paddleocr, rapidocr)" — page-level OCR fallback when
+   pdfplumber finds no extractable words (scanned pages).
+2. Table-cell OCR inside TableFormerONNX's `ocr_anchor_cells` step (see
+   table_extraction.py) — same backends, reused as-is.
+
+Unchanged in substance from docling_.ipynb, just split out into its own
+module and with the legacy crop-based `.read()` method dropped (it was a
+no-op stub for every backend already — `get_text_boxes` was the only one
+actually implemented and used).
+"""
+
+from __future__ import annotations
+
+import logging
+from abc import ABC, abstractmethod
+
+import cv2
+import numpy as np
+from PIL import Image
+import os
+logger = logging.getLogger(__name__)
+
+
+
+
+
+
+def _log_ocr_tokens(backend_name: str, image: Image.Image, tokens: list[dict]) -> None:
+    logger.debug("OCR (%s) on image %dx%d", backend_name, image.width, image.height)
+    logger.info("OCR (%s) returned %d text boxes", backend_name, len(tokens))
+    if logger.isEnabledFor(logging.DEBUG):
+        for t in tokens[:10]:
+            logger.debug("  %r @ %s", t["text"], t["bbox"])
+
+
+class OCRBackend(ABC):
+    @abstractmethod
+    def get_text_boxes(self, image: Image.Image) -> list[dict]:
+        """Run OCR on the full image once; return all text tokens with
+        their global pixel-space bounding boxes [x1, y1, x2, y2]."""
+        raise NotImplementedError
+
+
+class RapidOCRBackend(OCRBackend):
+    def __init__(self,det_model_path,rec_model_path,rec_keys_path):
+        from rapidocr_onnxruntime import RapidOCR
+
+        self._engine = RapidOCR( det_model_path=det_model_path,
+    rec_model_path=rec_model_path,
+    rec_keys_path=rec_keys_path)
+
+    @staticmethod
+    def _to_bgr(image: Image.Image) -> np.ndarray:
+        rgb = np.array(image.convert("RGB"))
+        return cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
+
+    def get_text_boxes(self, image: Image.Image) -> list[dict]:
+        result, _ = self._engine(self._to_bgr(image), use_det=True, use_cls=False)
+        tokens = []
+        if result:
+            for item in result:
+                pts = np.array(item[0])  # 4 corners
+                x1, y1 = pts.min(axis=0)
+                x2, y2 = pts.max(axis=0)
+                tokens.append(
+                    {"text": str(item[1]).strip(), "bbox": [float(x1), float(y1), float(x2), float(y2)]}
+                )
+        _log_ocr_tokens("rapidocr", image, tokens)
+        return tokens
+
+
+class PytesseractBackend(OCRBackend):
+    def __init__(self, lang: str = "fra+eng"):
+        import pytesseract
+
+        self._pt = pytesseract
+        self.lang = lang
+
+    def get_text_boxes(self, image: Image.Image) -> list[dict]:
+        data = self._pt.image_to_data(image, lang=self.lang, output_type=self._pt.Output.DICT)
+        tokens = []
+        for i in range(len(data["text"])):
+            text = data["text"][i].strip()
+            if text:
+                x = data["left"][i]
+                y = data["top"][i]
+                w = data["width"][i]
+                h = data["height"][i]
+                tokens.append({"text": text, "bbox": [float(x), float(y), float(x + w), float(y + h)]})
+        _log_ocr_tokens("pytesseract", image, tokens)
+        return tokens
+
+
+
+
+def get_ocr_backend(name: str, **kwargs) -> OCRBackend:
+    if name == "rapidocr":
+        return RapidOCRBackend(**kwargs)
+    if name == "pytesseract":
+        return PytesseractBackend(**kwargs)
+   
+    raise ValueError(f"unknown OCR backend: {name!r} (use 'rapidocr', 'pytesseract')")

doc_pipeline/pipeline.py

@@ -0,0 +1,505 @@
+"""
+pipeline.py
+===========
+
+Wires together layout.py, ocr_backends.py, and table_extraction.py into the
+exact flow drawn in the diagram:
+
+    pdf -> pdfplumber -> words+bboxes, image
+                            |                \\
+                 if words empty?            pp-doclayout -> layout boxes
+                  yes /        \\no                  |        |
+                 OCR        Plumber+layout match     |   OCR+layout match
+                  \\__________________________________/________/
+                                    |
+                         (class_name, bbox, reading_order, text) per region
+                                    |
+                        if class_name == "table" -> TableFormerONNX -> OTSL -> markdown
+                        else                      -> markdown formatted by class_name
+
+Two entry points sit on top of a shared per-image core (`_process_rendered_page`):
+
+- `process_pdf_page` / `process_pdf` — PDF input via pdfplumber. May have
+  native words (Plumber-path matching) or not (OCR fallback).
+- `process_image_page` / `process_images` — standalone image input
+  (.jpg/.png/etc., e.g. a photographed or scanned page with no PDF
+  structure at all). These never have native words, so they always go
+  through the OCR branch — the same code path a scanned PDF page uses.
+
+`process_document` is a single dispatcher that picks the right one based on
+the input (a .pdf path, an image path, or a list of image paths treated as
+ordered pages of one document).
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+from pathlib import Path
+from typing import List, Optional, Sequence, Union
+import base64
+import io
+import cv2
+import numpy as np
+from PIL import Image
+
+from .layout import (
+    DocLayoutV3,
+    PlacedWord,
+    RegionText,
+    align_ocr_to_layout,
+    align_tokens_to_layout,
+    log_layout_result,
+    pdfplumber_tokens_in_image_space,
+)
+from .ocr_backends import OCRBackend
+from .table_extraction import TableFormerONNX, otsl_to_markdown, table_image_to_otsl
+
+logger = logging.getLogger(__name__)
+
+# Layout classes that contribute nothing to the output markdown.
+SKIP_CLASSES = {"header", "footer", "number", "seal"}
+
+# Layout classes rendered as a cropped image link rather than as text.
+IMAGE_LIKE_CLASSES = {"chart", "header_image", "footer_image", "vision_footnote"}
+
+# Extensions routed through the standalone-image path by process_document.
+IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".bmp", ".tif", ".tiff", ".webp", ".gif"}
+
+
+def _to_bgr_array(img: Image.Image) -> np.ndarray:
+    """
+    Convert a PIL image to a BGR ndarray.
+
+    `DocLayoutV3._preprocess` always applies a BGR->RGB channel swap, which
+    is only correct if the array it receives is actually BGR (as
+    `cv2.imread` produces). PIL images are RGB, so feeding one in directly
+    — as the original page-rendering code did — silently double-swaps the
+    channels and degrades detection. Converting explicitly here keeps every
+    caller (PDF-rendered pages and standalone images alike) consistent with
+    what `DocLayoutV3` actually expects from an in-memory array.
+    """
+    return cv2.cvtColor(np.array(img.convert("RGB")), cv2.COLOR_RGB2BGR)
+
+
+def crop_and_save_image(image: Image.Image, bbox: tuple, index: int, doc_path: str) -> str:
+    """Crop `bbox` (xyxy, pixel space) out of `image` and save it under
+    `<doc_stem>/images/<index>.png`, returning that path."""
+    stem = doc_path.split(".")[0]
+    images_dir = os.path.join(stem, "images")
+    os.makedirs(images_dir, exist_ok=True)
+    cropped = image.crop(bbox)
+    out_path = os.path.join(images_dir, f"{index}.png")
+    cropped.save(out_path)
+    return out_path
+
+
+
+def crop_to_base64(image: Image.Image, bbox: tuple, fmt: str = "PNG") -> str:
+    """Crop `bbox` (xyxy, pixel space) out of `image` and return a base64
+    data URI, ready to embed directly in markdown — no file written to disk."""
+    cropped = image.crop(bbox)
+    buffer = io.BytesIO()
+    cropped.save(buffer, format=fmt)
+    b64_data = base64.b64encode(buffer.getvalue()).decode("ascii")
+    return f"data:image/{fmt.lower()};base64,{b64_data}"
+
+
+def _table_tokens_from_page(
+    bbox: tuple, page_tokens: Optional[List[PlacedWord]]
+) -> List[dict]:
+    """
+    Pull the pdfplumber word tokens whose centroid falls inside a table
+    region's `bbox` (xyxy, full-page image pixel space — the same space
+    `page_tokens` is already expressed in) and re-express them relative to
+    the cropped table image's own origin (top-left of `bbox`), which is the
+    coordinate convention `table_image_to_otsl`/`ocr_anchor_cells` expects.
+
+    Returns `[]` if there's no text layer (`page_tokens` is None/empty) or
+    no words happen to land inside this particular region — callers should
+    fall back to OCR in that case.
+    """
+    if not page_tokens:
+        return []
+    xmin, ymin, xmax, ymax = bbox
+    local_tokens = []
+    for t in page_tokens:
+        cx, cy = (t.x0 + t.x1) / 2.0, (t.top + t.bottom) / 2.0
+        if xmin <= cx <= xmax and ymin <= cy <= ymax:
+            local_tokens.append(
+                {"text": t.text, "bbox": [t.x0 - xmin, t.top - ymin, t.x1 - xmin, t.bottom - ymin]}
+            )
+    return local_tokens
+
+
+def region_to_markdown(
+    region: RegionText,
+    index: int,
+    img: Image.Image,
+    doc_path: str,
+    table_runner: Optional[TableFormerONNX] = None,
+    table_ocr_backend: Optional[OCRBackend] = None,
+    page_tokens: Optional[List[PlacedWord]] = None,
+) -> str:
+    label = region.class_name
+    logger.debug("Rendering region %d: class=%s bbox=%s", index, label, region.bbox)
+
+    if label in SKIP_CLASSES:
+        logger.debug("Skipping region %d (%s)", index, label)
+        return ""
+
+    if label == "table":
+        if table_runner is not None:
+            table_tokens = _table_tokens_from_page(region.bbox, page_tokens)
+            tmp_path = None
+            try:
+                import tempfile
+                cropped = img.crop(region.bbox)
+                with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmp:
+                    cropped.save(tmp, format="PNG")
+                    tmp_path = tmp.name
+
+                if table_tokens:
+                    logger.info(
+                        "Table region %d: using %d pdfplumber tokens (no OCR)",
+                        index, len(table_tokens),
+                    )
+                    otsl = table_image_to_otsl(table_runner, tmp_path, tokens=table_tokens)
+                elif table_ocr_backend is not None:
+                    logger.info(
+                        "Table region %d: no pdfplumber tokens — OCR via %s",
+                        index, type(table_ocr_backend).__name__,
+                    )
+                    otsl = table_image_to_otsl(table_runner, tmp_path, backend=table_ocr_backend)
+                else:
+                    raise ValueError(
+                        "no pdfplumber words found in this table region and no "
+                        "table_ocr_backend was provided to fall back on"
+                    )
+                table_md, df = otsl_to_markdown(otsl)
+                if df is not None:
+                    logger.info(
+                        "Table region %d: extracted %d rows x %d cols (Excel export disabled)",
+                        index, df.shape[0], df.shape[1],
+                    )
+                    logger.debug("Table region %d dataframe:\n%s", index, df)
+                    # Excel export intentionally disabled for now.
+
+                return f"{table_md}\n\n"
+            except Exception as exc:
+                logger.warning(
+                    "Table region %d: extraction failed (%s) — falling back to image embed",
+                    index, exc,
+                )
+                image_b64 = crop_to_base64(img, region.bbox)
+                return f"<!-- table extraction failed: {exc} -->\n![table]({image_b64})\n\n"
+            finally:
+                if tmp_path and os.path.exists(tmp_path):
+                    os.remove(tmp_path)
+        logger.info("Table region %d: no TableFormer configured — embedding crop", index)
+        image_b64 = crop_to_base64(img, region.bbox)
+        return f"![table]({image_b64})\n\n"
+
+    if label in IMAGE_LIKE_CLASSES:
+        image_b64 = crop_to_base64(img, region.bbox)
+        logger.info("Image-like region %d (%s): embedded as base64", index, label)
+        return f"![{label}]({image_b64})\n\n"
+
+    text = region.text
+    if not text:
+        return ""
+
+    if label == "doc_title":
+        return f"# {text}\n\n"
+    if label == "paragraph_title":
+        return f"## {text}\n\n"
+    if label == "aside_text":
+        return f"> {text}\n\n"
+    if label == "figure_title":
+        return f"### {text}\n\n"
+    if label == "image":
+        image_b64 = crop_to_base64(img, region.bbox)
+        output = f"![{label}]({image_b64})\n\n"
+        if text.strip():
+            output += f"{text}\n\n"
+        return output
+
+    return f"{text}\n\n"
+
+def regions_to_markdown(
+    regions: List[RegionText],
+    img: Image.Image,
+    doc_path: str,
+    table_runner: Optional[TableFormerONNX] = None,
+    table_ocr_backend: Optional[OCRBackend] = None,
+    page_tokens: Optional[List[PlacedWord]] = None,
+) -> str:
+    """Render a full list of reading-order-sorted regions to one markdown string."""
+    parts = []
+    for i, region in enumerate(regions):
+        parts.append(
+            region_to_markdown(region, i, img, doc_path, table_runner, table_ocr_backend, page_tokens)
+        )
+    return "".join(parts)
+
+
+def _process_rendered_page(
+    img: Image.Image,
+    layout_detector: DocLayoutV3,
+    doc_path: str,
+    page_index: int,
+    page_tokens: Optional[List[PlacedWord]],
+    page_ocr_backend: Optional[OCRBackend],
+    table_runner: Optional[TableFormerONNX],
+    table_ocr_backend: Optional[OCRBackend],
+) -> dict:
+    """
+    Shared core of the diagram's bottom half, given an already-loaded page
+    image: run layout detection, then route to either the pdfplumber-words
+    matcher or the OCR-fallback matcher, then render per-region markdown.
+
+    `page_tokens=None` (or `[]`) always forces the OCR path — this is what
+    a standalone image (no native text layer at all) and a scanned PDF page
+    (pdfplumber found no words) both end up doing identically.
+    """
+    logger.info("Page %d: running layout detection on %s", page_index, doc_path)
+    layout_result = layout_detector.predict(_to_bgr_array(img))
+    log_layout_result(layout_result)
+
+    used_ocr = False
+    if page_tokens:
+        logger.info(
+            "Page %d: matching path pdfplumber (%d tokens)",
+            page_index,
+            len(page_tokens),
+        )
+        if logger.isEnabledFor(logging.DEBUG):
+            for t in page_tokens[:5]:
+                logger.debug("  sample token: %r @ (%.0f, %.0f)", t.text, t.x0, t.top)
+        regions = align_tokens_to_layout(page_tokens, layout_result)
+    else:
+        used_ocr = True
+        if page_ocr_backend is None:
+            raise ValueError(
+                "No native text/tokens available (scanned page or standalone "
+                "image input) and no page_ocr_backend was provided to fall back on."
+            )
+        logger.warning(
+            "Page %d: no text layer — OCR via %s",
+            page_index,
+            type(page_ocr_backend).__name__,
+        )
+        ocr_tokens = page_ocr_backend.get_text_boxes(img)
+        regions = align_ocr_to_layout(layout_result, ocr_tokens)
+
+    logger.info("Page %d: matched %d regions (used_ocr=%s)", page_index, len(regions), used_ocr)
+    for r in regions:
+        preview = (r.text[:80] + "...") if len(r.text) > 80 else r.text
+        logger.info(
+            "  region [%d] %s words=%d text=%r",
+            r.reading_order,
+            r.class_name,
+            len(r.words),
+            preview,
+        )
+
+    markdown_doc = regions_to_markdown(
+        regions, img, doc_path, table_runner, table_ocr_backend, page_tokens=page_tokens or None
+    )
+
+    return {
+        "page_index": page_index,
+        "used_ocr": used_ocr,
+        "layout_result": layout_result,
+        "regions": regions,
+        "markdown": markdown_doc,
+        "image": img,
+    }
+
+
+def process_pdf_page(
+    page,
+    layout_detector: DocLayoutV3,
+    doc_path: str,
+    page_index: int = 0,
+    resolution: int = 150,
+    page_ocr_backend: Optional[OCRBackend] = None,
+    table_runner: Optional[TableFormerONNX] = None,
+    table_ocr_backend: Optional[OCRBackend] = None,
+) -> dict:
+    """
+    Run the full diagram for a single pdfplumber Page.
+
+    - page_ocr_backend: used for the "Ocr(...)" box, only invoked if
+      page.extract_words() comes back empty (scanned page / no text layer).
+    - table_runner / table_ocr_backend: used for the "TableFormerONNX" box,
+      only invoked for regions whose class_name == "table", and only as a
+      fallback when no pdfplumber words land inside that region. If neither
+      is usable, table regions fall back to an embedded cropped image instead.
+
+    Returns a dict with the page's markdown plus the intermediate regions,
+    so callers can inspect/debug a single page without re-running detection.
+    """
+    page_image = page.to_image(resolution=resolution)
+    img = page_image.original.copy()
+
+    # Computed once: reused both for matching regions (Plumber-path) and,
+    # later, for pulling exact words inside any "table" region instead of
+    # re-OCR-ing it. `[]` here means "no text layer" (e.g. a scanned page).
+    img_w, img_h = img.size
+    logger.info(
+        "Processing PDF page %d of %s (resolution=%d, image=%dx%d)",
+        page_index,
+        doc_path,
+        resolution,
+        img_w,
+        img_h,
+    )
+    page_tokens = pdfplumber_tokens_in_image_space(page, img_w, img_h)
+    logger.info("Page %d: pdfplumber extracted %d word tokens", page_index, len(page_tokens))
+
+    return _process_rendered_page(
+        img, layout_detector, doc_path, page_index, page_tokens,
+        page_ocr_backend, table_runner, table_ocr_backend,
+    )
+
+
+def process_pdf(
+    doc_path: str,
+    layout_detector: DocLayoutV3,
+    pages: Optional[List[int]] = None,
+    resolution: int = 150,
+    page_ocr_backend: Optional[OCRBackend] = None,
+    table_runner: Optional[TableFormerONNX] = None,
+    table_ocr_backend: Optional[OCRBackend] = None,
+) -> str:
+    """
+    Run the full diagram over an entire PDF (or a subset of page indices)
+    and return one concatenated markdown document.
+    """
+    import pdfplumber
+
+    logger.info("Processing PDF: %s (pages=%s)", doc_path, pages if pages is not None else "all")
+    markdown_chunks = []
+    with pdfplumber.open(doc_path) as pdf:
+        page_indices = pages if pages is not None else range(len(pdf.pages))
+        for i in page_indices:
+            page = pdf.pages[i]
+            result = process_pdf_page(
+                page,
+                layout_detector,
+                doc_path,
+                page_index=i,
+                resolution=resolution,
+                page_ocr_backend=page_ocr_backend,
+                table_runner=table_runner,
+                table_ocr_backend=table_ocr_backend,
+            )
+            markdown_chunks.append(result["markdown"])
+
+    logger.info("Finished PDF: %s (%d pages)", doc_path, len(markdown_chunks))
+    return "\n".join(markdown_chunks)
+
+
+def process_image_page(
+    image_path: Union[str, Path],
+    layout_detector: DocLayoutV3,
+    doc_path: Optional[str] = None,
+    page_index: int = 0,
+    page_ocr_backend: Optional[OCRBackend] = None,
+    table_runner: Optional[TableFormerONNX] = None,
+    table_ocr_backend: Optional[OCRBackend] = None,
+) -> dict:
+    """
+    Run the diagram on a single standalone image (a photo or scan, not a
+    PDF page). There's no PDF text layer to check, so this always takes the
+    OCR branch — equivalent to a PDF page where `extract_words()` came back
+    empty. `page_ocr_backend` is therefore required, not optional, here.
+
+    `doc_path` controls where cropped table/image regions get saved
+    (`<doc_stem>/images/<n>.png`); defaults to `image_path` itself.
+    """
+    if page_ocr_backend is None:
+        raise ValueError("process_image_page requires page_ocr_backend — standalone images have no text layer.")
+
+    logger.info("Processing standalone image page %d: %s", page_index, image_path)
+    img = Image.open(image_path).convert("RGB")
+    doc_path = doc_path or str(image_path)
+
+    return _process_rendered_page(
+        img, layout_detector, doc_path, page_index, None,
+        page_ocr_backend, table_runner, table_ocr_backend,
+    )
+
+
+def process_images(
+    image_paths: Sequence[Union[str, Path]],
+    layout_detector: DocLayoutV3,
+    page_ocr_backend: Optional[OCRBackend] = None,
+    table_runner: Optional[TableFormerONNX] = None,
+    table_ocr_backend: Optional[OCRBackend] = None,
+) -> str:
+    """
+    Run the diagram over a batch of standalone images (e.g. photographed
+    pages of one physical document) in the given order, and return one
+    concatenated markdown document — the image-input analog of `process_pdf`.
+    """
+    logger.info("Processing %d standalone images", len(image_paths))
+    chunks = []
+    for i, path in enumerate(image_paths):
+        result = process_image_page(
+            path, layout_detector, doc_path=str(path), page_index=i,
+            page_ocr_backend=page_ocr_backend, table_runner=table_runner,
+            table_ocr_backend=table_ocr_backend,
+        )
+        chunks.append(result["markdown"])
+    logger.info("Finished image batch (%d pages)", len(chunks))
+    return "\n".join(chunks)
+
+
+def process_document(
+    path: Union[str, Path, Sequence[Union[str, Path]]],
+    layout_detector: DocLayoutV3,
+    pages: Optional[List[int]] = None,
+    resolution: int = 150,
+    page_ocr_backend: Optional[OCRBackend] = None,
+    table_runner: Optional[TableFormerONNX] = None,
+    table_ocr_backend: Optional[OCRBackend] = None,
+) -> str:
+    """
+    Single entry point for the whole pipeline — dispatches to the PDF path
+    or the image path based on the input, so callers don't need to care
+    which kind of file they have:
+
+    - a path to a `.pdf` -> `process_pdf`
+    - a path to a single image (.jpg/.png/etc.) -> `process_image_page`
+    - a list of image paths -> `process_images` (treated as ordered pages
+      of one document)
+    """
+    if isinstance(path, (list, tuple)):
+        logger.info("process_document: dispatching to process_images (%d paths)", len(path))
+        return process_images(
+            path, layout_detector, page_ocr_backend=page_ocr_backend,
+            table_runner=table_runner, table_ocr_backend=table_ocr_backend,
+        )
+
+    suffix = Path(path).suffix.lower()
+    if suffix == ".pdf":
+        logger.info("process_document: dispatching to process_pdf (%s)", path)
+        return process_pdf(
+            str(path), layout_detector, pages=pages, resolution=resolution,
+            page_ocr_backend=page_ocr_backend, table_runner=table_runner,
+            table_ocr_backend=table_ocr_backend,
+        )
+    if suffix in IMAGE_EXTENSIONS:
+        logger.info("process_document: dispatching to process_image_page (%s)", path)
+        result = process_image_page(
+            path, layout_detector, page_ocr_backend=page_ocr_backend,
+            table_runner=table_runner, table_ocr_backend=table_ocr_backend,
+        )
+        return result["markdown"]
+
+    raise ValueError(
+        f"Unsupported file type {suffix!r} for {path!r}; expected a .pdf or "
+        f"an image ({sorted(IMAGE_EXTENSIONS)})."
+    )

doc_pipeline/table_extraction.py

@@ -0,0 +1,565 @@
+"""
+table_extraction.py
+====================
+
+Right-hand branch of the diagram: when a matched region's class is "table",
+its cropped image goes through TableFormerONNX -> OTSL string -> markdown.
+
+`TableFormerONNX`, `ocr_anchor_cells`, `seq_to_otsl`, and `table_image_to_otsl`
+are carried over from docling_.ipynb essentially unchanged (decoupled from
+that notebook's specific OCR-backend import style; backends now come from
+ocr_backends.py). `otsl_to_markdown` is new — it's the "otsl -> markdown" step
+in the diagram, which in the original notebook was done ad hoc in later
+cells (build a <doctag>, run DocTagsDocument/DoclingDocument). Folding it
+into one function makes it reusable per-table inside the full-page pipeline.
+It returns `(markdown, dataframe)` — the dataframe is built from
+`TableItem.export_to_dataframe()` with `collapse_spanned_columns` applied,
+since a raw rectangular DataFrame otherwise duplicates a colspan'd cell's
+value into every column it visually spans (see that function's docstring
+for why, and why `pd.read_html(export_to_html())` has the identical issue).
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import time
+from pathlib import Path
+from typing import Optional, Union
+from urllib.parse import urlparse
+
+import cv2
+import numpy as np
+import onnxruntime as ort
+import pandas as pd
+from PIL import Image
+
+from .ocr_backends import OCRBackend
+
+logger = logging.getLogger(__name__)
+
+ANCHOR_TAGS = {"fcel", "ecel", "ched", "rhed", "srow"}  # tags that occupy their own grid slot
+EMPTY_OTSL_PLACEHOLDER = "?"  # inserted when a non-empty-tagged cell OCRs to nothing,
+                              # so the docling_core parser never sees an empty content cell
+
+MEAN = np.array([0.94247851, 0.94254675, 0.94292611], dtype=np.float32)
+STD = np.array([0.17910956, 0.17940403, 0.17931663], dtype=np.float32)
+IMAGE_SIZE = 448
+
+
+# --------------------------------------------------------------------------- #
+# bbox merge helpers (lcel chains)
+# --------------------------------------------------------------------------- #
+def merge_bboxes(box1: np.ndarray, box2: np.ndarray) -> np.ndarray:
+    """Union the cxcywh boxes of an lcel chain's stub anchor and the real cell that closes it."""
+    new_w = (box2[0] + box2[2] / 2) - (box1[0] - box1[2] / 2)
+    new_h = (box2[1] + box2[3] / 2) - (box1[1] - box1[3] / 2)
+    new_left = box1[0] - box1[2] / 2
+    new_top = min((box2[1] - box2[3] / 2), (box1[1] - box1[3] / 2))
+    return np.array(
+        [new_left + new_w / 2, new_top + new_h / 2, new_w, new_h], dtype=np.float32
+    )
+
+
+def apply_bbox_merge(coords: np.ndarray, bboxes_to_merge: dict) -> np.ndarray:
+    coords = [np.asarray(c, dtype=np.float32) for c in coords]
+    merged, skip = [], set()
+    for i, box1 in enumerate(coords):
+        if i in bboxes_to_merge:
+            j = bboxes_to_merge[i]
+            if j >= 0:
+                skip.add(j)
+                merged.append(merge_bboxes(box1, coords[j]))
+        elif i not in skip:
+            merged.append(box1)
+    return np.stack(merged) if merged else np.empty((0, 4), dtype=np.float32)
+
+
+def make_session(path: str, threads: int = 2) -> ort.InferenceSession:
+    opts = ort.SessionOptions()
+    opts.inter_op_num_threads = 2
+    opts.intra_op_num_threads = 4
+    opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+    opts.enable_mem_pattern = True
+    opts.enable_mem_reuse = True
+    opts.enable_profiling = False
+    return ort.InferenceSession(path, sess_options=opts, providers=["CPUExecutionProvider"])
+
+
+# --------------------------------------------------------------------------- #
+# TableFormer ONNX runner
+# --------------------------------------------------------------------------- #
+class TableFormerONNX:
+    """
+    encoder + decoder-step (autoregressive loop, KV-cache) + bbox_decoder,
+    all ONNX Runtime. Produces a structure sequence (OTSL tags) plus a
+    normalized cxcywh bbox per anchor cell.
+
+    Usage:
+        runner = TableFormerONNX(artifact_root="tableformerv1", variant="accurate")
+    """
+
+    def __init__(self, artifact_root: str = "tableformerv1", variant: str = "accurate", threads: int = 2):
+        root = Path(artifact_root)
+        onnx_dir = root / "onnx" / variant
+        self.enc = make_session(str(onnx_dir / f"tableformer_{variant}_encoder.onnx"), threads)
+        self.dec = make_session(str(onnx_dir / f"tableformer_{variant}_decoder_step.onnx"), threads)
+        self.bbox = make_session(str(onnx_dir / f"tableformer_{variant}_bbox_decoder.onnx"), threads)
+
+        with open(root / "tm_config.json", encoding="utf-8") as f:
+            self.tag_map = json.load(f)["dataset_wordmap"]["word_map_tag"]
+        self.rev_tag = {v: k for k, v in self.tag_map.items()}
+
+        cache_input = next(i for i in self.dec.get_inputs() if i.name == "cache")
+        self.num_layers = int(cache_input.shape[0])
+        self.embed_dim = int(cache_input.shape[3])
+
+    @staticmethod
+    def is_valid_url(url_string: str) -> bool:
+        try:
+            result = urlparse(url_string)
+            return all([result.scheme in ("http", "https"), result.netloc])
+        except ValueError:
+            return False
+
+    def image_preprocess(self, img) -> np.ndarray:
+        """Accepts a path, URL, PIL.Image, or HWC RGB ndarray."""
+        if isinstance(img, str) and self.is_valid_url(img):
+            import requests
+
+            img = Image.open(requests.get(img, stream=True, timeout=30).raw)
+        if isinstance(img, (str, Path)):
+            img = Image.open(img)
+        if isinstance(img, Image.Image):
+            img = np.array(img.convert("RGB"))
+        elif isinstance(img, np.ndarray) and img.ndim == 3 and img.shape[2] == 3:
+            pass
+        else:
+            raise TypeError(f"Unsupported image type: {type(img)}")
+
+        img = (img.astype(np.float32) - 255.0 * MEAN) / STD
+        img = cv2.resize(img, (IMAGE_SIZE, IMAGE_SIZE), interpolation=cv2.INTER_CUBIC)
+        img = img.transpose(2, 1, 0) / 255.0
+        return np.expand_dims(img.astype(np.float32), axis=0)
+
+    def draw_bbox(self, img_path: str, bboxs: np.ndarray, out_path: str = "draw_p.png") -> None:
+        """Debug helper — draws predicted cxcywh boxes on the ORIGINAL-resolution image."""
+        from PIL import ImageDraw
+
+        pil_image = Image.open(img_path)
+        width, height = pil_image.size
+        draw = ImageDraw.Draw(pil_image)
+        for cx, cy, w, h in np.asarray(bboxs):
+            xyxy = (
+                int((cx - 0.5 * w) * width), int((cy - 0.5 * h) * height),
+                int((cx + 0.5 * w) * width), int((cy + 0.5 * h) * height),
+            )
+            draw.rectangle(xyxy, outline="red", width=2)
+        pil_image.save(out_path)
+
+    def predict(self, img: np.ndarray, max_steps: int = 1024) -> dict:
+        """img: float32 [1, 3, 448, 448] (preprocessed). Returns seq / outputs_class /
+        outputs_coord (cxcywh, normalized [0,1]) / bboxes_to_merge / timings."""
+        t0 = time.time()
+        enc_out, memory = self.enc.run(None, {"image": img})
+        t_enc = time.time() - t0
+
+        wm = self.tag_map
+        decoded_tags = np.array([[wm["<start>"]]], dtype=np.int64)
+        cache = np.zeros((self.num_layers, 0, 1, self.embed_dim), dtype=np.float32)
+        skip_next_tag = True
+        prev_tag_ucel = False
+        line_num = 0
+        first_lcel = True
+        bbox_ind = 0
+        cur_bbox_ind = -1
+        bboxes_to_merge: dict = {}
+        tag_H_buf = []
+
+        t1 = time.time()
+        for _ in range(max_steps):
+            logits, last_hidden, cache = self.dec.run(
+                None, {"decoded_tags": decoded_tags, "memory": memory, "cache": cache}
+            )
+            new_tag = int(np.argmax(logits, axis=1)[0])
+
+            if line_num == 0 and new_tag == wm["xcel"]:
+                new_tag = wm["lcel"]
+            if prev_tag_ucel and new_tag == wm["lcel"]:
+                new_tag = wm["fcel"]
+
+            if new_tag == wm["<end>"]:
+                decoded_tags = np.concatenate(
+                    [decoded_tags, np.array([[new_tag]], dtype=np.int64)], axis=0
+                )
+                break
+
+            if not skip_next_tag:
+                if new_tag in (wm["fcel"], wm["ecel"], wm["ched"], wm["rhed"],
+                               wm["srow"], wm["nl"], wm["ucel"]):
+                    tag_H_buf.append(last_hidden[:, 0, :].copy())
+                    if first_lcel is not True:
+                        bboxes_to_merge[cur_bbox_ind] = bbox_ind
+                    bbox_ind += 1
+
+            if new_tag != wm["lcel"]:
+                first_lcel = True
+            else:
+                if first_lcel:
+                    tag_H_buf.append(last_hidden[:, 0, :].copy())
+                    first_lcel = False
+                    cur_bbox_ind = bbox_ind
+                    bboxes_to_merge[cur_bbox_ind] = -1
+                    bbox_ind += 1
+
+            skip_next_tag = new_tag in (wm["nl"], wm["ucel"], wm["xcel"])
+            prev_tag_ucel = new_tag == wm["ucel"]
+            if new_tag == wm["nl"]:
+                line_num += 1
+
+            decoded_tags = np.concatenate(
+                [decoded_tags, np.array([[new_tag]], dtype=np.int64)], axis=0
+            )
+        t_dec = time.time() - t1
+
+        seq = decoded_tags.squeeze().tolist()
+
+        if tag_H_buf:
+            tag_H_stacked = np.stack(
+                [h[None, ...] if h.ndim == 1 else h for h in tag_H_buf], axis=0
+            ).reshape(-1, 1, self.embed_dim).astype(np.float32)
+            t2 = time.time()
+            cls_logits, coord = self.bbox.run(None, {"enc_out": enc_out, "tag_H_stacked": tag_H_stacked})
+            t_bbox = time.time() - t2
+            coord = apply_bbox_merge(coord, bboxes_to_merge)
+        else:
+            cls_logits = np.empty((0, 3), dtype=np.float32)
+            coord = np.empty((0, 4), dtype=np.float32)
+            t_bbox = 0.0
+
+        return {
+            "seq": seq,
+            "outputs_class": cls_logits,
+            "outputs_coord": coord,
+            "bboxes_to_merge": bboxes_to_merge,
+            "timings": {"encoder": t_enc, "decoder": t_dec, "bbox": t_bbox,
+                        "total": t_enc + t_dec + t_bbox},
+        }
+
+
+# --------------------------------------------------------------------------- #
+# OTSL assembly: structure tokens + per-cell OCR -> docling-ready OTSL string
+# --------------------------------------------------------------------------- #
+def compute_ioa(ocr_bbox: list[float], cell_bbox: list[float]) -> float:
+    """Intersection over OCR-Token Area (IoA): maps a token to a cell if most
+    of the token sits inside it."""
+    ox1, oy1, ox2, oy2 = ocr_bbox
+    cx1, cy1, cx2, cy2 = cell_bbox
+
+    ix1 = max(ox1, cx1)
+    iy1 = max(oy1, cy1)
+    ix2 = min(ox2, cx2)
+    iy2 = min(oy2, cy2)
+
+    if ix2 <= ix1 or iy2 <= iy1:
+        return 0.0
+
+    inter_area = (ix2 - ix1) * (iy2 - iy1)
+    ocr_area = (ox2 - ox1) * (oy2 - oy1)
+    return inter_area / max(ocr_area, 1e-6)
+
+
+def ocr_anchor_cells(
+    orig_image_path: str,
+    seq: list,
+    box_o: np.ndarray,
+    rev_tag: dict,
+    backend: Optional[OCRBackend] = None,
+    tokens: Optional[list[dict]] = None,
+    pad: int = 4,
+    narrow_width: float = 0.08,
+) -> list:
+    """Maps text tokens into TableFormer cell boundaries via spatial intersection.
+
+    Either `backend` (run OCR on the table crop) or pre-extracted `tokens`
+    (e.g. pdfplumber words already expressed in the crop's local pixel space —
+    see pipeline.py) must be supplied. `tokens` takes priority: when a PDF
+    has a real text layer, reusing those exact words is both cheaper and more
+    accurate than re-OCRing a table crop pdfplumber already read correctly.
+    """
+    if tokens is None and backend is None:
+        raise ValueError("ocr_anchor_cells needs either `tokens` or `backend`")
+
+    orig = Image.open(orig_image_path).convert("RGB")
+    orig_w, orig_h = orig.size
+
+    anchor_tags = [rev_tag[t] for t in seq[1:-1] if rev_tag[t] in ANCHOR_TAGS]
+    assert len(anchor_tags) == box_o.shape[0], (
+        f"{len(anchor_tags)} anchor tags vs {box_o.shape[0]} boxes — structural alignment issue."
+    )
+
+    ocr_tokens = tokens if tokens is not None else backend.get_text_boxes(orig)
+    source = "pdfplumber tokens" if tokens is not None else type(backend).__name__
+    logger.debug(
+        "ocr_anchor_cells: %d anchor cells, %d tokens from %s",
+        len(anchor_tags),
+        len(ocr_tokens),
+        source,
+    )
+    texts = []
+
+    for i, (tag, (cx, cy, w, h)) in enumerate(zip(anchor_tags, box_o)):
+        if tag == "ecel":
+            texts.append("")
+            continue
+
+        x1 = max(0, (cx - w / 2) * orig_w - pad)
+        y1 = max(0, (cy - h / 2) * orig_h - pad)
+        x2 = min(orig_w, (cx + w / 2) * orig_w + pad)
+        y2 = min(orig_h, (cy + h / 2) * orig_h + pad)
+        cell_bbox = [x1, y1, x2, y2]
+
+        matched_tokens = [t for t in ocr_tokens if compute_ioa(t["bbox"], cell_bbox) >= 0.45]
+
+        if not matched_tokens:
+            texts.append("")
+            logger.debug("  cell[%d] tag=%s (empty)", i, tag)
+            continue
+
+        matched_tokens.sort(key=lambda t: (int(t["bbox"][1] / 6), t["bbox"][0]))
+        cell_string = " ".join(t["text"] for t in matched_tokens)
+        texts.append(cell_string.strip())
+        logger.debug("  cell[%d] tag=%s text=%r", i, tag, cell_string.strip()[:60])
+
+    return texts
+
+
+def _otsl_cell_text(tag: str, text: str) -> str | None:
+    if tag == "ecel":
+        return None
+    cleaned = (text or "").strip()
+    return cleaned if cleaned else EMPTY_OTSL_PLACEHOLDER
+
+
+def seq_to_otsl(seq: list, rev_tag: dict, texts: list) -> str:
+    """Re-walk seq, inserting OCR text right after each anchor tag."""
+    out, ti = [], 0
+    for tok in seq[1:-1]:
+        name = rev_tag[tok]
+        if name == "nl":
+            out.append("<nl>")
+            continue
+        out.append(f"<{name}>")
+        if name in ANCHOR_TAGS:
+            cell_text = _otsl_cell_text(name, texts[ti])
+            if cell_text is not None:
+                out.append(cell_text)
+            ti += 1
+    return "".join(out)
+
+
+def table_image_to_otsl(
+    runner: TableFormerONNX,
+    orig_image_path: str,
+    backend: Optional[OCRBackend] = None,
+    tokens: Optional[list[dict]] = None,
+) -> str:
+    """End-to-end: image -> ONNX TableFormer -> anchors filled from `tokens`
+    (preferred, e.g. pdfplumber words) or OCR'd via `backend` -> OTSL string."""
+    logger.info("TableFormer: processing %s", orig_image_path)
+    preproc = runner.image_preprocess(orig_image_path)
+    out = runner.predict(preproc)
+    logger.debug(
+        "TableFormer: seq length=%d, anchor boxes=%d",
+        len(out["seq"]),
+        out["outputs_coord"].shape[0],
+    )
+    texts = ocr_anchor_cells(
+        orig_image_path, out["seq"], out["outputs_coord"], runner.rev_tag,
+        backend=backend, tokens=tokens,
+    )
+    otsl = seq_to_otsl(out["seq"], runner.rev_tag, texts)
+    logger.debug("TableFormer OTSL preview: %s", otsl[:200])
+    return otsl
+
+
+def collapse_spanned_columns(df: "pd.DataFrame") -> "pd.DataFrame":
+    """
+    Collapse adjacent columns that are exact duplicates (same header text,
+    identical value in every row) back into a single column.
+
+    A plain rectangular DataFrame has no native way to represent a cell that
+    spans multiple columns (e.g. an OTSL `lcel` chain / HTML `colspan`).
+    Both `TableItem.export_to_dataframe()` and the `export_to_html()` +
+    `pd.read_html()` round-trip handle this the same way: they repeat the
+    spanned cell's value into every column it covers, so a 2-column-wide
+    "Description" header ends up as two separate "Description" columns
+    with identical content. This collapses those back into one.
+
+    Note: this is a heuristic — if two genuinely distinct adjacent columns
+    happen to share both the same header text and identical values in
+    every row of this specific table, they'll also get collapsed. That's
+    expected to be rare in practice (real tables don't usually have two
+    different columns with the same name and identical data), but worth
+    knowing if a table doesn't collapse cleanly.
+    """
+    cols = list(df.columns)
+    if len(cols) <= 1:
+        return df
+
+    keep = [0]
+    for i in range(1, len(cols)):
+        prev = keep[-1]
+        # pandas suffixes repeated header names with ".1", ".2", ... on
+        # read_html; export_to_dataframe doesn't, so strip defensively.
+        prev_header = str(cols[prev]).split(".")[0]
+        cur_header = str(cols[i]).split(".")[0]
+        same_header = prev_header == cur_header
+        same_values = df.iloc[:, prev].astype(str).equals(df.iloc[:, i].astype(str))
+        if same_header and same_values:
+            continue  # duplicate of the column we're keeping — drop it
+        keep.append(i)
+
+    out = df.iloc[:, keep].copy()
+    out.columns = [str(c).split(".")[0] for c in out.columns]
+    return out
+
+
+def _parse_md_table_row(line: str) -> list[str]:
+    """Split one markdown table row into its cell strings, stripping whitespace."""
+    return [c.strip() for c in line.strip().strip("|").split("|")]
+
+
+def _is_separator_row(cells: list[str]) -> bool:
+    """Return True if every cell is a markdown alignment/separator marker (e.g. `---`, `:---:`)."""
+    return all(set(c.replace(":", "").replace("-", "")) == set() and len(c) > 0 for c in cells)
+
+
+def collapse_markdown_table(md: str) -> str:
+    """
+    Apply the same duplicate-adjacent-column collapse to every markdown table
+    inside `md`, in-place. This is necessary because markdown has no colspan
+    syntax — docling_core's `export_to_markdown()` represents a colspan-N cell
+    by repeating its value in N consecutive columns, which looks like
+    `| Description | Description | Total |` for a 2-column-wide "Description"
+    header. This function detects and removes those duplicate columns from the
+    markdown string so the rendered table matches what the source document
+    actually meant.
+
+    The collapse criterion is identical to `collapse_spanned_columns`: two
+    adjacent columns are merged when they share the same header text (after
+    stripping pandas `.1`/`.2` dedup suffixes, though docling markdown doesn't
+    add those) AND every data row has the same value in both columns.
+
+    Non-table lines (headings, paragraphs, etc.) are passed through unchanged.
+    """
+    lines = md.splitlines()
+    out_lines: list[str] = []
+    i = 0
+
+    while i < len(lines):
+        line = lines[i]
+
+        # Detect the start of a markdown table: a line containing at least one |
+        # that isn't just punctuation.
+        if "|" not in line:
+            out_lines.append(line)
+            i += 1
+            continue
+
+        # Collect the full table block (contiguous lines containing |).
+        table_block: list[str] = []
+        while i < len(lines) and "|" in lines[i]:
+            table_block.append(lines[i])
+            i += 1
+
+        if len(table_block) < 2:
+            # Not a real table (need at least header + separator).
+            out_lines.extend(table_block)
+            continue
+
+        # Parse into header, optional separator, and data rows.
+        header_cells = _parse_md_table_row(table_block[0])
+        sep_idx = 1 if _is_separator_row(_parse_md_table_row(table_block[1])) else None
+
+        data_rows: list[list[str]] = []
+        for row_line in table_block[(2 if sep_idx is not None else 1):]:
+            data_rows.append(_parse_md_table_row(row_line))
+
+        if not data_rows:
+            out_lines.extend(table_block)
+            continue
+
+        # Pad short rows so indexing is safe.
+        n_cols = len(header_cells)
+        data_rows = [r + [""] * max(0, n_cols - len(r)) for r in data_rows]
+
+        # Determine which columns to keep — same logic as collapse_spanned_columns.
+        keep: list[int] = [0]
+        for j in range(1, n_cols):
+            prev = keep[-1]
+            prev_header = header_cells[prev].split(".")[0]
+            cur_header = header_cells[j].split(".")[0]
+            same_header = prev_header == cur_header
+            same_values = all(
+                (row[prev] if prev < len(row) else "") == (row[j] if j < len(row) else "")
+                for row in data_rows
+            )
+            if same_header and same_values:
+                continue
+            keep.append(j)
+
+        if len(keep) == n_cols:
+            # Nothing to collapse — pass through unchanged.
+            out_lines.extend(table_block)
+            continue
+
+        # Rebuild the table with only the kept columns.
+        def _fmt_row(cells: list[str], indices: list[int]) -> str:
+            return "| " + " | ".join(cells[k] for k in indices) + " |"
+
+        def _fmt_sep(indices: list[int]) -> str:
+            return "| " + " | ".join("---" for _ in indices) + " |"
+
+        out_lines.append(_fmt_row(header_cells, keep))
+        if sep_idx is not None:
+            out_lines.append(_fmt_sep(keep))
+        for row in data_rows:
+            out_lines.append(_fmt_row(row, keep))
+
+    return "\n".join(out_lines)
+
+
+def otsl_to_markdown(
+    otsl: str, dummy_image_size: tuple[int, int] = (512, 512)
+) -> tuple[str, "pd.DataFrame"]:
+    """
+    Wrap a raw OTSL tag string as <doctag><otsl>...</otsl></doctag>, load it
+    through docling_core's DocTagsDocument/DoclingDocument, and export both
+    markdown and a DataFrame for the (first) detected table.
+
+    Returns `(markdown, dataframe)` — both have duplicate columns from
+    colspan cells collapsed:
+    - markdown: via `collapse_markdown_table` (markdown has no colspan syntax
+      so docling repeats the spanned cell's text into every column it covers)
+    - dataframe: via `TableItem.export_to_dataframe()` + `collapse_spanned_columns`
+      (same duplication issue at the DataFrame level, same fix)
+    """
+    from docling_core.types.doc.document import DocTagsDocument, DoclingDocument
+
+    otsl_doc = f"<doctag><otsl>{otsl}</otsl></doctag>"
+    dummy_image = Image.new("RGB", dummy_image_size, color="black")
+    doctags_doc = DocTagsDocument.from_doctags_and_image_pairs([otsl_doc], [dummy_image])
+    doc = DoclingDocument.load_from_doctags(doctags_doc)
+
+    md = collapse_markdown_table(doc.export_to_markdown())
+
+    if doc.tables:
+        df = doc.tables[0].export_to_dataframe(doc=doc)
+        df = collapse_spanned_columns(df)
+        logger.info("OTSL -> markdown: %d rows x %d cols", df.shape[0], df.shape[1])
+    else:
+        df = pd.DataFrame()
+        logger.warning("OTSL -> markdown: no tables detected in OTSL output")
+
+    return md, df