---
library_name: transformers
license: apache-2.0
base_model: Qwen/Qwen3-4B
tags:
- llama-factory
- full
- generated_from_trainer
- decor
- baseline
model-index:
- name: decor-qwen3-4b-original
  results: []
---

# DecoR: Qwen3-4B Fine-tuned on Original LIMO (Baseline)

This model is **Qwen3-4B** fine-tuned on the **original** (unmodified) [LIMO](https://arxiv.org/abs/2502.03387) dataset. It serves as the baseline for comparison with DSE-cleaned variants.

## Key Results

| Benchmark | Base Qwen3-4B | **Original LIMO SFT** | DSE LIMO SFT |
|:----------|:---:|:---:|:---:|
| MATH-500 | 56.6% | 69.6% | **72.8%** |
| AIME 2025 | 13.3% | 40.0% | **43.3%** |
| AIME 2026 | 36.7% | 46.7% | **53.3%** |
| GPQA Diamond | 43.4% | **55.6%** | 49.0% |

## Training Details

- **Base model**: [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B)
- **Training data**: LIMO-Original (817 samples, unmodified)
- **Framework**: LLaMA-Factory + DeepSpeed ZeRO-2
- **Hardware**: 2× NVIDIA A100 80GB (CSD3 HPC)
- **Hyperparameters**:
  - Learning rate: 5e-6 (cosine schedule, 10% warmup)
  - Batch size: 8 (effective)
  - Epochs: 15
  - Max sequence length: 16384
  - Full fine-tuning (no LoRA)

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Ciaranshu/decor-qwen3-4b-original")
tokenizer = AutoTokenizer.from_pretrained("Ciaranshu/decor-qwen3-4b-original")
```

### Framework versions

- Transformers 4.52.4
- Pytorch 2.5.1+cu124
- Datasets 3.6.0
- Tokenizers 0.21.1