---
license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B
pipeline_tag: feature-extraction
library_name: transformers
language:
  - en
tags:
  - qwen3.5
  - classification-backbone
  - knowledge-distillation
  - model-compression
  - edge-ai
---

# Qwen3.5 Standalone 4L Classification Base

This repository publishes a **headless, classification-ready four-layer
Qwen3.5 text backbone**. The root model has no task labels and no classification
head. It is intended as a compact starting point for single-label or multilabel
classifiers, not as a chat model or a ready-made universal classifier.

The repository follows one integrated 1-1-1 layout:

- **Root model:** task-agnostic four-layer base trained on unlabeled general
  text.
- **`models/semeval-propaganda/`:** the existing SemEval-derived propaganda
  classifier as a separate task-specific lineage.
- **`distillation/`:** a reusable 24L→8L→6L→4L platform for your own data.

## Load the headless base

```python
from transformers import AutoModel, AutoTokenizer

model_id = "mp-juuuns/qwen35-standalone4l-classification-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
backbone = AutoModel.from_pretrained(model_id, trust_remote_code=False)
```

The root uses `Qwen3_5TextModel` in Transformers 5.13.0. It has four text
layers in the sequence `linear → full → linear → full`, hidden size 1,024, and
the full upstream Qwen3.5 tokenizer. The published weight file contains no
`score.weight`.

## Make your own classifier

```bash
git clone https://huggingface.co/mp-juuuns/qwen35-standalone4l-classification-base
cd qwen35-standalone4l-classification-base/distillation
python -m pip install -e .

qwen35-distill finetune \
  --checkpoint .. \
  --train examples/multilabel_train.jsonl \
  --labels examples/labels.json \
  --mode multilabel \
  --output my-classifier-4l
```

For a full task-specific compression run, first fine-tune a 24-layer teacher,
then repeat `materialize-classifier` and `distill-classifier` for `24to8`,
`8to6`, and `6to4`. Exact JSONL formats and commands are in
[`distillation/README.md`](distillation/README.md).

## How the root weights were made

The text backbone and full tokenizer were extracted from
[`Qwen/Qwen3.5-0.8B`](https://huggingface.co/Qwen/Qwen3.5-0.8B). Students were
trained sequentially through 24→8→6→4 using hidden-boundary and final
representation matching on unlabeled text.

The task-free training set was exactly 4,096 deterministically shuffled,
non-empty rows from the first locally cached WikiText-103 raw training shard.
No SemEval examples, labels, logits, thresholds, or evidence/span objectives
were read while training these root weights.

WikiText is not redistributed here. Its current Hugging Face page has a license
wording discrepancy: metadata lists CC BY-SA 3.0 and GFDL, while the prose says
CC BY-SA 4.0. Users should review the
[`Salesforce/wikitext`](https://huggingface.co/datasets/Salesforce/wikitext)
terms directly.

## One-task transfer check

We attached a fresh 14-label head and used the frozen SemEval-derived
article-level protocol only as a transfer probe.

| Initialization | Opened-test Macro-F1, mean ± sample SD |
|---|---:|
| Task-agnostic 4L base | **0.58748 ± 0.01579** |
| Same 4L structural copy without task-free KD | 0.51809 ± 0.01631 |
| Existing separate specialized 4L lineage | 0.58757 ± 0.00717 |

Results use seeds 41/42/43 and a previously opened 55-article public test split.
They are exploratory, not untouched confirmatory evidence. The derived task
measures article-level presence of 14 techniques and is not identical to the
official SemEval span- or fragment-level task. Three seed values are not a
confidence interval.

This check supports transfer on one English classification task only. It does
not establish universal classification quality, multilingual generalization,
calibration, safety, or production suitability.

## Separate SemEval model

The ready-to-use propaganda model is under
[`models/semeval-propaganda/`](models/semeval-propaganda/). It preserves the
currently published seed-41/42/43 checkpoints and links to the original
repository at immutable revision
[`e7e0ff1`](https://huggingface.co/mp-juuuns/qwen35-standalone4l-propaganda-classifier/tree/e7e0ff16828052687d2e8dd7849e7a521629cf38).

That specialized model is **not** claimed to descend from this new task-free
base. It has its own SemEval task-specific shrink/distillation and fine-tuning
history and a reduced 128k vocabulary, while the root base retains the full
upstream tokenizer.

## Limitations

- A classification head and labeled training are required before root-model
  predictions have task meaning.
- The task-free corpus is small: 4,096 rows from one WikiText shard.
- Only one downstream transfer task was measured.
- Default layer maps are documented structural choices, not universal optima.
- Long-context, multilingual, calibration, robustness, fairness, and device
  performance of the new root base have not been established.
- Do not use either model as a fact checker, safety oracle, or autonomous
  decision maker.

## License and attribution

The released code and model artifacts use Apache-2.0, subject to upstream model
and data terms. Qwen3.5 is attributed to Qwen. Transformers, PyTorch,
Hugging Face Hub, WikiText, and SemEval remain the work of their respective
authors. No WikiText or SemEval source records are redistributed.

Machine-readable training manifests, transfer summaries, hashes, and the
release inventory are included under `provenance/`, `benchmark/`, and
`release_manifest.json`.