---
library_name: transformers
license: mit
datasets:
  - PRAIG/JAZZMUS
  - PRAIG/JAZZMUS_staffLevel
  - PRAIG/JAZZMUS_Synthetic
language:
  - en
tags:
  - optical-music-recognition
  - music-transcription
  - jazz
  - lead-sheets
  - kern
  - sheet-music-transformer
base_model: antoniorv6/smt-grandstaff
pipeline_tag: image-to-text
metrics:
  - wer
  - cer
---

# JAZZMUS — OMR for Jazz Lead Sheets

An end-to-end **Optical Music Recognition** model that transcribes handwritten jazz lead sheet images (melody + chords) into Humdrum `**kern` / `**mxhm` notation.

> **Paper:** [Optical Music Recognition of Jazz Lead Sheets](https://arxiv.org/abs/2509.05329) — ISMIR 2025

## Overview

This model addresses the challenge of recognising handwritten jazz lead sheets, a score type that encodes **melody** and **chord symbols**. Chords are a score component not handled by existing OMR systems, and handwritten images exhibit high variability and quality issues.

The model is a **Sheet Music Transformer** (SMT) — a ConvNeXt encoder paired with a Transformer decoder — pretrained on polyphonic piano scores ([`antoniorv6/smt-grandstaff`](https://huggingface.co/antoniorv6/smt-grandstaff)) and fine-tuned on the JAZZMUS dataset (handwritten + synthetic lead sheets) using **medium-level tokenisation**.

A **YOLOv11** staff detector is bundled in this repository to enable full-page inference.

## Quick Start

### Command-line inference

Clone the repository and install dependencies:

```bash
git clone https://github.com/JuanCarlosMartinezSevilla/ISMIR-Jazzmus.git
cd ISMIR-Jazzmus
pip install -e ".[predict]"
```

Transcribe a full-page lead sheet image:

```bash
# Print to stdout
python predict.py page.jpg

# Save to a .krn file
python predict.py page.jpg -o output.krn

# Print and save
python predict.py page.jpg -o output.krn -p

# Process multiple pages
python predict.py page1.jpg page2.jpg -o results/
```

The script automatically downloads the SMT model and YOLO staff detector weights from this repository.

### Programmatic usage

```python
import torch
from safetensors.torch import load_file
from torch.nn import Conv1d
from huggingface_hub import hf_hub_download

from jazzmus.model.smt.configuration_smt import SMTConfig
from jazzmus.model.smt.modeling_smt import SMTModelForCausalLM

REPO_ID = "JuanCarlosMartinezSevilla/jazzmus-model"

# Load config and weights
config = SMTConfig.from_pretrained(REPO_ID)
weights_path = hf_hub_download(repo_id=REPO_ID, filename="model.safetensors")
sd = load_file(weights_path)

# Build model (handles pretrained embedding / output layer size mismatch)
embed_size = sd["decoder.embedding.weight"].shape[0]
out_size = sd["decoder.out_layer.weight"].shape[0]
config.out_categories = embed_size
model = SMTModelForCausalLM(config)
if embed_size != out_size:
    model.decoder.out_layer = Conv1d(config.d_model, out_size, kernel_size=1)
model.load_state_dict(sd, strict=True)
config.out_categories = out_size

model.eval()
```

## Datasets

This model was trained on:

| Dataset | Description |
|---|---|
| [PRAIG/JAZZMUS](https://huggingface.co/datasets/PRAIG/JAZZMUS) | 293 handwritten jazz lead sheet page images with ground truth |
| [PRAIG/JAZZMUS_staffLevel](https://huggingface.co/datasets/PRAIG/JAZZMUS_staffLevel) | 2021 region-level staff crops used for training |
| [PRAIG/JAZZMUS_Synthetic](https://huggingface.co/datasets/PRAIG/JAZZMUS_Synthetic) | 326 synthetic lead sheet images (Classical + MuseJazz fonts) |

## Limitations

- The model operates at **staff-region level**; full-page inference depends on the bundled YOLO staff detector.
- Chord recognition is harder than melody recognition — most errors involve chord symbols (see qualitative analysis in the paper).
- Implicit accidentals from key signatures are sometimes missed.
- Performance may degrade on score styles not represented in the training set (e.g., printed commercial lead sheets, non-jazz genres).

## Citation

```bibtex
@inproceedings{juan_carlos_martinez_sevilla_2025_17811464,
  author       = {Juan Carlos Martinez-Sevilla and
                  Francesco Foscarin and
                  Patricia Garcia-Iasci and
                  David Rizo and
                  Jorge Calvo-Zaragoza and
                  Gerhard Widmer},
  title        = {Optical Music Recognition of Jazz Lead Sheets},
  booktitle    = {Proceedings of the 26th International Society for
                   Music Information Retrieval Conference},
  year         = 2025,
  pages        = {710-716},
  publisher    = {ISMIR},
  month        = sep,
  venue        = {Daejeon, South Korea and Online},
  doi          = {10.5281/zenodo.17811464},
  url          = {https://doi.org/10.5281/zenodo.17811464},
}
```

## Links

- **Paper:** [zenodo.17811464](https://doi.org/10.5281/zenodo.17811464)
- **ArXiv:** [arXiv 2509.05329](https://arxiv.org/abs/2509.05329)
- **Code:** [GitHub — ISMIR-Jazzmus](https://github.com/JuanCarlosMartinezSevilla/ISMIR-Jazzmus)
- **Project page:** [https://grfia.dlsi.ua.es/jazz-omr/](https://grfia.dlsi.ua.es/jazz-omr/)
- **Pretrained base model:** [antoniorv6/smt-grandstaff](https://huggingface.co/antoniorv6/smt-grandstaff)