File size: 3,432 Bytes
9fc8c19 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | # Qwen3.5 Classification Distillation
This package turns the text backbone of `Qwen/Qwen3.5-0.8B` into smaller
classification-ready backbones and task-specific classifiers.
Two workflows are intentionally separate:
1. **Task-agnostic base:** hidden/interface distillation on unlabeled text.
It produces a headless text backbone and never reads task labels.
2. **Task-specific classifier:** attach a single-label or multilabel head,
train a 24-layer teacher on a user dataset, then transfer hard labels,
teacher logits, and hidden/interface representations through
`24L → 8L → 6L → 4L`.
The published default maps are structural defaults, not a claim that they are
universally optimal:
| Stage | Source layers retained | Target layer types |
|---|---|---|
| 24L → 8L | `0,4,6,11,13,16,20,23` | `L,L,L,F,L,L,L,F` |
| 8L → 6L | `0,1,3,4,6,7` | `L,L,F,L,L,F` |
| 6L → 4L | `0,2,3,5` | `L,F,L,F` |
The final 6L→4L numeric map overlaps the earlier SemEval experiment, but the
task-agnostic base uses the fixed structural contract above and no SemEval
labels, examples, logits, thresholds, or span objectives. Architecture choice
and task-free weight training are reported as separate provenance claims.
## Dataset formats
Unlabeled:
```json
{"id":"u-1","text":"Unlabeled text used only for representation distillation."}
```
Single-label:
```json
{"id":"s-1","text":"Example text.","label":"class_a"}
```
Multilabel:
```json
{"id":"m-1","text":"Example text.","labels":["class_a","class_c"]}
```
Each split is a separate JSONL file. `id` must be unique across all splits.
An optional `group_id` is checked for cross-split leakage.
## Initial commands
```bash
python -m qwen35_distill.cli validate-dataset \
--mode unlabeled --train examples/unlabeled.jsonl
python -m qwen35_distill.cli extract-text \
--source /path/to/Qwen3.5-0.8B \
--output /path/to/text-24l
python -m qwen35_distill.cli materialize \
--source /path/to/text-24l \
--output /path/to/text-8l \
--stage 24to8
python -m qwen35_distill.cli distill \
--teacher /path/to/text-24l \
--student /path/to/text-8l \
--data /path/to/unlabeled.parquet \
--output /path/to/text-8l-kd \
--stage 24to8 --device cuda --dtype bfloat16
```
Every output is exclusive: existing directories are never overwritten.
## Build a classifier from your own labels
Put the exact label order in `labels.json`, then train a teacher or attach a
head directly to the published 4-layer base:
```bash
qwen35-distill finetune \
--checkpoint /path/to/text-24l \
--train train.jsonl --validation validation.jsonl \
--labels labels.json --mode multilabel \
--output classifier-24l
```
To preserve the classifier head while following the full compression chain,
repeat these two commands for `24to8`, `8to6`, and `6to4`:
```bash
qwen35-distill materialize-classifier \
--source classifier-24l --stage 24to8 \
--output classifier-8l-init
qwen35-distill distill-classifier \
--teacher classifier-24l --checkpoint classifier-8l-init \
--stage 24to8 --train train.jsonl --validation validation.jsonl \
--labels labels.json --mode multilabel \
--output classifier-8l
```
The task-specific distillation objective combines the user labels, teacher
logits, and aligned hidden boundaries. The teacher hashes, dataset hashes,
label order, environment, loss history, and fresh-reload check are written to
each output manifest.
|