Create README.md

README.md

@@ -0,0 +1,106 @@
+# Usage
+
+This model outputs a reward for each reasoning step evaluating it.
+
+`Babelscape/Qwen2.5-Math-PRM-7B-PDDL-r` is a **Process Reward Model (PRM)** obtained by continual fine-tuning from **Qwen/Qwen2.5-Math-PRM-7B** with the planning-based supervision introduced in **PDDL2PRM**.
+
+Unlike the other PRM checkpoints in this release, this model is not trained from the base/instruct model with a newly added scalar reward head. Instead, it starts from the original **Qwen2.5-Math-PRM-7B** checkpoint and continues its training on PDDL2PRM data. For this reason, it follows the original Qwen PRM reward interface: reasoning steps must be separated with the `<extra_0>` marker, and rewards are obtained from the positive-class probability at marker positions.
+
+PDDL2PRM is the dataset introduced in:
+
+**Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards**
+Raffaele Pisano and Roberto Navigli, ACL 2026
+
+Project page & paper: https://babelscape.github.io/prm-meets-planning/
+arXiv: https://arxiv.org/abs/2604.17957
+
+The paper proposes using symbolic planning problems written in **Planning Domain Definition Language (PDDL)** to generate precise step-level rewards for reasoning trajectories. In PDDL, actions, states, preconditions, effects, and goals are explicitly defined, so intermediate reasoning steps can be evaluated automatically.
+
+## Example
+
+```python
+import torch
+import torch.nn.functional as F
+from transformers import AutoTokenizer, AutoModel
+
+repo_id = "Babelscape/Qwen2.5-Math-PRM-7B-PDDL-r"
+
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()
+
+
+def build_messages(problem, steps):
+    return [
+        {
+            "role": "system",
+            "content": "Please reason step by step, and put your final answer within \\boxed{}."
+        },
+        {
+            "role": "user",
+            "content": problem
+        },
+        {
+            "role": "assistant",
+            "content": "<extra_0>".join(steps) + "<extra_0>"
+        }
+    ]
+
+
+def get_step_rewards(logits, marker_positions):
+    probs = F.softmax(logits, dim=-1)
+    # Positive-class probability at each <extra_0> marker position
+    return probs[0, marker_positions, 1].detach().cpu().tolist()
+
+
+problem = "If x + 3 = 10, find x."
+steps = [
+    "Subtract 3 from both sides: x = 10 - 3.",
+    "So x = 7."
+]
+
+messages = build_messages(problem, steps)
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=False
+)
+
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model(**inputs)
+
+logits = outputs.logits if hasattr(outputs, "logits") else outputs[0]
+
+marker_id = tokenizer.encode("<extra_0>", add_special_tokens=False)[0]
+marker_positions = (inputs["input_ids"][0] == marker_id).nonzero(as_tuple=True)[0]
+
+step_scores = get_step_rewards(logits, marker_positions)
+
+print("Step scores:", step_scores)
+
+first_bad = next((i for i, score in enumerate(step_scores) if score < 0.5), -1)
+print("First failing step index:", first_bad)
+```
+
+# Notes
+
+* The marker `<extra_0>` must appear after every reasoning step.
+* This model follows the reward format of `Qwen/Qwen2.5-Math-PRM-7B`.
+* Rewards are computed from the positive-class probability at `<extra_0>` marker positions.
+* A threshold such as 0.5 can be used to identify potentially incorrect steps.
+* This differs from the PRM800K-based checkpoints with a scalar reward head, where `pred_scalar` is read at marker positions.
+
+# Citation
+
+If you use this model or the PDDL2PRM dataset in your work, please cite:
+
+```bibtex
+@inproceedings{pisano2026prmplanning,
+  title={Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards},
+  author={Pisano, Raffaele and Navigli, Roberto},
+  booktitle={Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)},
+  year={2026},
+  note={Accepted}
+}
+```