---
license: apache-2.0
base_model: Qwen/Qwen2.5-7B-Instruct
tags:
  - sdft
  - self-distillation
  - tool-use
  - continual-learning
  - sdft-reproduction
datasets:
  - ToolAlpaca
---

# Qwen2.5-7B-Instruct — SDFT on Tool Use (Step 1000, Best)

Best checkpoint from SDFT (Self-Distillation Fine-Tuning) reproduction of ["Self-Distillation Enables Continual Learning"](https://arxiv.org/abs/2601.19897).

## Results

| Metric | Base | This Model | Paper |
|--------|------|------------|-------|
| Greedy Accuracy | 54.4% | **64.7%** | 70.6% |
| Pass@1 | 52.6% | 56.2% | — |
| Pass@5 | 61.5% | 70.1% | — |
| Pass@10 | 64.4% | 74.4% | — |
| Pass@50 | 70.6% | 79.4% | — |

## Training Details

| Parameter | Value |
|-----------|-------|
| Base model | Qwen/Qwen2.5-7B-Instruct |
| Method | On-policy Self-Distillation (SDFT) |
| Dataset | ToolAlpaca (4046 train, 68 test) |
| Learning rate | 1e-5 |
| Batch size | 32 |
| Epochs | 2 |
| EMA alpha | 0.01 |
| Step | 1000 (best of 1011) |
| Hardware | L40S 48GB |

## All Checkpoints

| Step | Greedy Acc |
|------|-----------|
| [100](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-100) | 55.9% |
| [200](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-200) | 48.5% |
| [300](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-300) | 44.1% |
| [400](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-400) | 47.1% |
| [500](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-500) | 57.4% |
| [600](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-600) | 47.1% |
| [700](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-700) | 54.4% |
| [800](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-800) | 52.9% |
| [900](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-900) | 57.4% |
| **[1000](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-1000)** | **64.7%** |
| [1011](https://huggingface.co/Ayushnangia/qwen2.5-7b-instruct-sdft-tooluse-step-1011) | 57.4% |

## Related

- **Eval results:** [Ayushnangia/sdft-reproduction-eval-results](https://huggingface.co/datasets/Ayushnangia/sdft-reproduction-eval-results)
- **Code:** [github.com/ayushnangia/Self-Distillation](https://github.com/ayushnangia/Self-Distillation)