peiyugeorgia/qwen2vl-crochetbench-task-c-finetuned

Qwen2-VL-7B finetuned on crochetBench Task C

This model is a fine-tuned version of Qwen/Qwen2-VL-7B-Instruct, trained on the crochetBench Task C benchmark.

• Paper: CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?
• Repository: GitHub

Task Description

Task C — Image-to-Pattern Generation: Given an image of a finished crochet product, the model generates a complete set of crochet instructions in standard published-pattern format.

The output follows these conventions:

• Standard abbreviations: sc (single crochet), hdc (half double crochet), dc (double crochet), tr (treble), ch (chain), sl st (slip stitch), rep (repeat)
• Organized row-by-row or round-by-round (e.g., Rnd 1: ..., Row 2: ...)
• Concise and precise, targeting experienced crocheters
• Pattern instructions only — no commentary or extra text

Model Details

Property	Value
Base model	Qwen/Qwen2-VL-7B-Instruct
Fine-tuning method	Full fine-tuning
Optimizer	AdamW
Learning rate	1e-5
Epochs	3
Effective batch size	8
Best checkpoint	Step 1500
Training date	December 13, 2025

Usage

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from PIL import Image
import torch

model_id = "peiyugeorgia/qwen2vl-crochetbench-task-c-finetuned"

model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("your_crochet_image.jpg").convert("RGB")

messages = [
    {
        "role": "system",
        "content": (
            "You are a professional crochet pattern writer. "
            "Examine the image of the finished crochet product carefully. "
            "Write a complete set of crochet instructions in the standard style used in published patterns. "
            "Use standard abbreviations: sc (single crochet), hdc (half double crochet), dc (double crochet), "
            "tr (treble), ch (chain), sl st (slip stitch), rep (repeat). "
            "Organize the instructions row by row or round by round (e.g., 'Rnd 1: ...', 'Row 2: ...'). "
            "Keep the instructions concise and precise, as if for experienced crocheters. "
            "Output only the crochet pattern. Do not add any explanations, commentary, or extra text."
        )
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "Generate step-by-step crochet instructions for this image."}
        ]
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(**inputs, max_new_tokens=1024)

generated = processor.batch_decode(output_ids, skip_special_tokens=True)[0]
print(generated)

Citation

If you use this model, please cite the crochetBench benchmark.

@inproceedings{peiyuli2026,
  title     = {CrochetBench: Can Vision-Language Models Move from Describing to
               Doing in Crochet Domain?},
  author    = {Li, Peiyu and Huang, Xiaobao and Hua, Ting and Chawla, Nitesh V.},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for
               Computational Linguistics (Volume 1: Long Papers)},
  year      = {2026}
}