| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - qwen |
| - layer-pruning |
| - qlora |
| - compression |
| - reasoning |
| pipeline_tag: text-generation |
| base_model: qwen2.5-7b-instruct |
| --- |
| |
| # Iterative-Qwen-7layers-SGR-Dolci |
|
|
| This model is a **depth-pruned** version of **Qwen2.5-7B-Instruct**, obtained via **Iterative layer pruning** and post-trained using **QLoRA** with **Self-Generated Responses (SGR)** on the **Dolci** dataset. |
|
|
| It was released as part of our study on the limits of layer pruning for generative reasoning. |
|
|
| ## Summary |
| - **Base model:** Qwen2.5-7B-Instruct |
| - **Pruning:** Iterative |
| - **Depth:** 7 layers |
| - **Post-training:** QLoRA |
| - **Supervision:** Self-Generated Responses (SGR) |
| - **Data:** Dolci (SGR variant) |
|
|
| This checkpoint is intended for **research and analysis of pruning and recovery**, not as a production model. |
|
|
| ## Paper |
|
|
| **On the Limits of Layer Pruning for Generative Reasoning in LLMs** |
| https://arxiv.org/abs/2602.01997 |
|
|
| ```bibtex |
| @misc{shrestha2026limitslayerpruninggenerative, |
| title={On the Limits of Layer Pruning for Generative Reasoning in LLMs}, |
| author={Safal Shrestha and Anubhav Shrestha and Aadim Nepal and Minwu Kim and Keith Ross}, |
| year={2026}, |
| eprint={2602.01997}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.LG}, |
| url={https://arxiv.org/abs/2602.01997}, |
| } |
| |
