# Qwen3.5 Classification Distillation

This package turns the text backbone of `Qwen/Qwen3.5-0.8B` into smaller
classification-ready backbones and task-specific classifiers.

Two workflows are intentionally separate:

1. **Task-agnostic base:** hidden/interface distillation on unlabeled text.
   It produces a headless text backbone and never reads task labels.
2. **Task-specific classifier:** attach a single-label or multilabel head,
   train a 24-layer teacher on a user dataset, then transfer hard labels,
   teacher logits, and hidden/interface representations through
   `24L → 8L → 6L → 4L`.

The published default maps are structural defaults, not a claim that they are
universally optimal:

| Stage | Source layers retained | Target layer types |
|---|---|---|
| 24L → 8L | `0,4,6,11,13,16,20,23` | `L,L,L,F,L,L,L,F` |
| 8L → 6L | `0,1,3,4,6,7` | `L,L,F,L,L,F` |
| 6L → 4L | `0,2,3,5` | `L,F,L,F` |

The final 6L→4L numeric map overlaps the earlier SemEval experiment, but the
task-agnostic base uses the fixed structural contract above and no SemEval
labels, examples, logits, thresholds, or span objectives. Architecture choice
and task-free weight training are reported as separate provenance claims.

## Dataset formats

Unlabeled:

```json
{"id":"u-1","text":"Unlabeled text used only for representation distillation."}
```

Single-label:

```json
{"id":"s-1","text":"Example text.","label":"class_a"}
```

Multilabel:

```json
{"id":"m-1","text":"Example text.","labels":["class_a","class_c"]}
```

Each split is a separate JSONL file. `id` must be unique across all splits.
An optional `group_id` is checked for cross-split leakage.

## Initial commands

```bash
python -m qwen35_distill.cli validate-dataset \
  --mode unlabeled --train examples/unlabeled.jsonl

python -m qwen35_distill.cli extract-text \
  --source /path/to/Qwen3.5-0.8B \
  --output /path/to/text-24l

python -m qwen35_distill.cli materialize \
  --source /path/to/text-24l \
  --output /path/to/text-8l \
  --stage 24to8

python -m qwen35_distill.cli distill \
  --teacher /path/to/text-24l \
  --student /path/to/text-8l \
  --data /path/to/unlabeled.parquet \
  --output /path/to/text-8l-kd \
  --stage 24to8 --device cuda --dtype bfloat16
```

Every output is exclusive: existing directories are never overwritten.

## Build a classifier from your own labels

Put the exact label order in `labels.json`, then train a teacher or attach a
head directly to the published 4-layer base:

```bash
qwen35-distill finetune \
  --checkpoint /path/to/text-24l \
  --train train.jsonl --validation validation.jsonl \
  --labels labels.json --mode multilabel \
  --output classifier-24l
```

To preserve the classifier head while following the full compression chain,
repeat these two commands for `24to8`, `8to6`, and `6to4`:

```bash
qwen35-distill materialize-classifier \
  --source classifier-24l --stage 24to8 \
  --output classifier-8l-init

qwen35-distill distill-classifier \
  --teacher classifier-24l --checkpoint classifier-8l-init \
  --stage 24to8 --train train.jsonl --validation validation.jsonl \
  --labels labels.json --mode multilabel \
  --output classifier-8l
```

The task-specific distillation objective combines the user labels, teacher
logits, and aligned hidden boundaries. The teacher hashes, dataset hashes,
label order, environment, loss history, and fresh-reload check are written to
each output manifest.