doc_pipeline/README.md

pdf_pipeline

A PDF → Markdown extraction pipeline implementing this flow:

pdf
 └─ pdfplumber ──────────────┬──────────────────────────────┐
        │ words+bboxes(xyxy) │ image                        │
        ▼                    ▼                              │
  empty words?          pp-doclayout                         │
   yes │   no            (DocLayoutV3)                       │
       │     └────────────┐  │ layout_class+bbox+order       │
       ▼                  ▼  ▼                                
  OCR(pytesseract/   Plumber words + pp-doclayout matching    
  paddleocr/rapidocr)  (align_words_to_layout)                
       │                  │                                   
       ▼                  ▼                                   
  Matching ocr bboxes   (class_name, bbox, reading_order, text)
  and layout                       │
  (align_ocr_to_layout)            ▼
       │                   class_name == "table"?
       │                    yes │      no
       └───────────┬───────────┘       └─ formatted by class
                    ▼
            TableFormerONNX → OTSL → markdown

Both matching branches converge on the same RegionText shape and the same downstream per-region rendering (region_to_markdown), including the table → TableFormerONNX → OTSL → markdown branch — the diagram only draws that branch coming off the pdfplumber-matching box, but a "table" layout region can equally appear on a scanned (OCR'd) page, so it's handled the same way in both paths.

Modules

File	Diagram piece(s)
layout.py	pp-doclayout (DocLayoutV3 ONNX wrapper) + both "matching" boxes (align_words_to_layout, align_ocr_to_layout, sharing the generic align_tokens_to_layout core)
ocr_backends.py	Ocr(pytesseract, paddleocr, rapidocr) — also reused for table-cell OCR
table_extraction.py	TableFormerONNX → OTSL → markdown
pipeline.py	The if the words are empty branch, the if classname is table branch, and final markdown assembly (process_pdf_page, process_pdf)

What's new vs. the two source notebooks

The two original notebooks (pdfplumber_.ipynb, docling_.ipynb) already had working code for: layout detection, the pdfplumber↔layout matcher, OCR backends, and TableFormer→OTSL. What was missing, per the diagram, was:

• The OCR-fallback matching box. align_words_to_layout only knew how to consume pdfplumber's extract_words() output. align_tokens_to_layout generalizes the matching algorithm (containment + nearest-centroid fallback + line re-assembly) to take any text + bbox token list, in image-pixel space. align_words_to_layout and align_ocr_to_layout are now both adapters onto that one function.
• The if classname is table branch wired into the page loop. The notebook's per-region markdown loop only ever embedded a cropped image for table/image/chart regions; TableFormer was only run manually, separately, against one standalone table image. region_to_markdown now checks class_name == "table" and runs table_image_to_otsl + otsl_to_markdown automatically, falling back to an embedded image if no TableFormerONNX runner was supplied (or if extraction throws).
• Table cells prefer pdfplumber's own words over OCR. When the PDF has a real text layer, the words pdfplumber already extracted for a table region are reused directly as TableFormer's cell-text source (tokens= on ocr_anchor_cells/table_image_to_otsl) instead of re-OCR-ing the table crop — cheaper and avoids OCR mistakes on text that's already exact. table_ocr_backend is now only invoked as a fallback when a table region has no underlying words (e.g. the table is itself a scanned image, or the whole page is scanned and went through the page-level OCR branch).
• Standalone image input. process_image_page/process_images run the same diagram on plain images (.jpg/.png/etc. — a photographed or scanned page with no PDF structure at all), always via the OCR branch since there's no native text layer to check. process_document is a single dispatcher that picks the PDF path or the image path based on the input, so callers don't need to branch on file type themselves. Both share the same per-image core (_process_rendered_page) as the PDF path, so a scanned PDF page and a standalone photo of a page are handled identically once OCR kicks in.
• Fixed a latent color-channel bug along the way. DocLayoutV3._preprocess always applies a BGR→RGB swap, correct only if its input really is BGR (as cv2.imread produces). The original page-rendering code fed it a PIL-derived array (RGB) directly, which silently double-swapped the channels and degraded detection on every PDF page processed through this pipeline. Fixed via a shared _to_bgr_array helper used by both the PDF and image entry points.
• otsl_to_markdown. The OTSL→DocTagsDocument→DoclingDocument→markdown steps were ad hoc, later notebook cells, applied to one hardcoded otsl variable. Factored into one function so it can run once per detected table inside a page loop.

Requirements

pip install pdfplumber numpy opencv-python onnxruntime pydantic pillow docling-core
# plus whichever OCR backend(s) you use:
pip install pytesseract            # needs the tesseract binary too
pip install rapidocr-onnxruntime
pip install paddleocr paddlepaddle

You'll also need the model artifacts referenced in the original notebooks:

• PP-DocLayout/PP-DocLayoutV3.onnx
• tableformerv1/onnx/<variant>/tableformer_<variant>_{encoder,decoder_step,bbox_decoder}.onnx
  • tableformerv1/tm_config.json

Usage

from pdf_pipeline import DocLayoutV3, TableFormerONNX, get_ocr_backend, process_document

layout_detector = DocLayoutV3("PP-DocLayout/PP-DocLayoutV3.onnx")
page_ocr_backend = get_ocr_backend("rapidocr")          # used on scanned PDF pages AND any standalone image
table_runner = TableFormerONNX(artifact_root="tableformerv1")
table_ocr_backend = get_ocr_backend("paddleocr")          # fallback for tables with no underlying text

# Works the same regardless of input type:
markdown_doc = process_document(
    "document.pdf",                       # or "scan.jpg", or ["page1.png", "page2.png"]
    layout_detector,
    page_ocr_backend=page_ocr_backend,
    table_runner=table_runner,
    table_ocr_backend=table_ocr_backend,
)

with open("document.md", "w") as f:
    f.write(markdown_doc)

process_document dispatches on what you pass it: a .pdf path goes through process_pdf (pdfplumber, may have a native text layer); a single image path (.jpg/.png/etc.) or a list of image paths goes through process_image_page/process_images (always OCR, since standalone images never have a text layer). Call process_pdf/process_image_page directly if you want the type-specific kwargs (e.g. pages= or resolution= only make sense for PDFs) or the full per-page debug dict instead of just the markdown string.

For single-page debugging (mirrors the original notebook's main cell), use process_pdf_page(page, layout_detector, doc_path, ...) directly — it returns a dict with markdown, regions, layout_result, image, and used_ocr, so you can inspect intermediate state (e.g. call layout_detector.visualize(result["layout_result"], "debug.png", result["image"])) before trusting the final markdown.

Cropped images (for image/chart/table-without-TableFormer regions) are written to <doc_stem>/images/<n>.png, same convention as the original notebook's crop_and_save_image.

Notes / things to tune for your documents

• containment_threshold / line_tol_ratio (passed through process_pdf_page → align_words_to_layout / align_ocr_to_layout → align_tokens_to_layout) control how aggressively tokens are assigned to boxes and how lines are re-grouped; the defaults (0.5 / 0.5) come from the original notebook.
• SKIP_CLASSES and IMAGE_LIKE_CLASSES in pipeline.py control which layout classes are dropped vs. rendered as a markdown image link vs. rendered as text — extend these for your document types (e.g. treat display_formula/inline_formula specially instead of falling through to plain text).
• If table_runner/table_ocr_backend are omitted, table regions degrade gracefully to an embedded cropped image instead of raising.