---
library_name: peft
base_model: Qwen/Qwen2.5-32B-Instruct
tags:
- trl
- grpo
- math
- reasoning
- qwen
license: apache-2.0
language:
- en
datasets:
- openai/gsm8k
---

# Qwen2.5-32B GRPO Math Reasoner

This is a fine-tuned LoRA adapter for `Qwen/Qwen2.5-32B-Instruct` that significantly enhances its mathematical logic and reasoning capabilities. The model was trained using **Group Relative Policy Optimization (GRPO)**, a cutting-edge reinforcement learning technique. 

Rather than just learning to output answers, the model was actively rewarded for formulating structured algebraic approaches and strictly formatting its final deductions inside `<answer>...</answer>` tags.

## 🏆 Model Details

- **Developer:** koushikkb12
- **Model Type:** Causal Language Model with LoRA adapter
- **Base Model:** `Qwen/Qwen2.5-32B-Instruct`
- **Language:** English
- **License:** Apache 2.0
- **Training Algorithm:** GRPO (Reinforcement Learning)

## 🎯 Intended Use

This model excels at breaking down complex word problems and physics/math scenarios into step-by-step logical deductions. It is highly resistant to "hallucinating" math, as it was trained to self-correct during the generation phase.

**Formatting Requirement:** When prompting the model, instruct it to wrap its final answer in `<answer>` tags. It has been highly conditioned to follow this structural constraint.

## 🚀 How to Get Started with the Model

Use the code below to dynamically load this LoRA adapter over the base Qwen 32B model.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load the native 16-bit base model
base_model_id = "Qwen/Qwen2.5-32B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)

# Attach the GRPO-trained LoRA adapter
adapter_id = "koushikkb12/Qwen2.5-32B-GRPO-Math"
model = PeftModel.from_pretrained(model, adapter_id)

# Inference
prompt = "A train leaves New York at 8:00 AM traveling 60 mph. Another leaves at 9:00 AM at 80 mph. When do they meet?"
messages = [
    {"role": "system", "content": "You are a logical math assistant. Wrap your final answer in <answer> tags."},
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## 🧠 Training Details

### Training Data
Trained on the **GSM8K** (Grade School Math 8K) dataset, evaluating against thousands of diverse algebraic and logical reasoning problems.

### Hardware & Compute Infrastructure
- **GPU:** 1x NVIDIA RTX PRO 6000 (96GB VRAM)
- **Precision:** Pure `bfloat16` (Native 16-bit, NO 4-bit quantization used)
- **Attention:** PyTorch Native SDPA (Scaled Dot Product Attention)
- **Training Time:** ~2 hours (200 GRPO Steps / 3,200 Evaluated Episodes)

### Training Hyperparameters
- **Batch Size:** 1 per device
- **Gradient Accumulation Steps:** 16
- **Generations per Prompt:** 4
- **LoRA Rank (r):** 16
- **LoRA Alpha:** 32
- **Learning Rate:** 2e-5
- **Optimizer:** AdamW (8-bit)
- **Max Sequence Length:** 2048 tokens

## 📦 Framework Versions
- `peft`: 0.19.1
- `trl`: Latest
- `transformers`: Latest
- `torch`: 2.8.0+cu128