distillation/README.md

Qwen3.5 Classification Distillation

This package turns the text backbone of Qwen/Qwen3.5-0.8B into smaller classification-ready backbones and task-specific classifiers.

Two workflows are intentionally separate:

1. Task-agnostic base: hidden/interface distillation on unlabeled text. It produces a headless text backbone and never reads task labels.
2. Task-specific classifier: attach a single-label or multilabel head, train a 24-layer teacher on a user dataset, then transfer hard labels, teacher logits, and hidden/interface representations through 24L → 8L → 6L → 4L.

The published default maps are structural defaults, not a claim that they are universally optimal:

Stage	Source layers retained	Target layer types
24L → 8L	0,4,6,11,13,16,20,23	L,L,L,F,L,L,L,F
8L → 6L	0,1,3,4,6,7	L,L,F,L,L,F
6L → 4L	0,2,3,5	L,F,L,F

The final 6L→4L numeric map overlaps the earlier SemEval experiment, but the task-agnostic base uses the fixed structural contract above and no SemEval labels, examples, logits, thresholds, or span objectives. Architecture choice and task-free weight training are reported as separate provenance claims.

Dataset formats

Unlabeled:

{"id":"u-1","text":"Unlabeled text used only for representation distillation."}

Single-label:

{"id":"s-1","text":"Example text.","label":"class_a"}

Multilabel:

{"id":"m-1","text":"Example text.","labels":["class_a","class_c"]}

Each split is a separate JSONL file. id must be unique across all splits. An optional group_id is checked for cross-split leakage.

Initial commands

python -m qwen35_distill.cli validate-dataset \
  --mode unlabeled --train examples/unlabeled.jsonl

python -m qwen35_distill.cli extract-text \
  --source /path/to/Qwen3.5-0.8B \
  --output /path/to/text-24l

python -m qwen35_distill.cli materialize \
  --source /path/to/text-24l \
  --output /path/to/text-8l \
  --stage 24to8

python -m qwen35_distill.cli distill \
  --teacher /path/to/text-24l \
  --student /path/to/text-8l \
  --data /path/to/unlabeled.parquet \
  --output /path/to/text-8l-kd \
  --stage 24to8 --device cuda --dtype bfloat16

Every output is exclusive: existing directories are never overwritten.

Build a classifier from your own labels

Put the exact label order in labels.json, then train a teacher or attach a head directly to the published 4-layer base:

qwen35-distill finetune \
  --checkpoint /path/to/text-24l \
  --train train.jsonl --validation validation.jsonl \
  --labels labels.json --mode multilabel \
  --output classifier-24l

To preserve the classifier head while following the full compression chain, repeat these two commands for 24to8, 8to6, and 6to4:

qwen35-distill materialize-classifier \
  --source classifier-24l --stage 24to8 \
  --output classifier-8l-init

qwen35-distill distill-classifier \
  --teacher classifier-24l --checkpoint classifier-8l-init \
  --stage 24to8 --train train.jsonl --validation validation.jsonl \
  --labels labels.json --mode multilabel \
  --output classifier-8l

The task-specific distillation objective combines the user labels, teacher logits, and aligned hidden boundaries. The teacher hashes, dataset hashes, label order, environment, loss history, and fresh-reload check are written to each output manifest.