BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation
Paper • 2402.10631 • Published • 2
| PPL | arc_easy | arc_challenge | piqa | winogrande | hellaswag | mmlu | QA Avg |
|---|---|---|---|---|---|---|---|
| 74461.16 | 25.29 ± 0.89 | 23.38 ± 1.24 | 52.45 ± 1.17 | 50.99 ± 1.40 | 25.54 ± 0.44 | - | 35.53 |
Training method based on BitDistiller Paper
Base model
TinyLlama/TinyLlama_v1.1