Feature Extraction
Transformers
Safetensors
multilingual
qwen3_5_text
qwen3.5
classification-backbone
text-classification
knowledge-distillation
model-compression
edge-ai
Instructions to use mp-juuuns/qwen35-standalone4l-classification-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mp-juuuns/qwen35-standalone4l-classification-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="mp-juuuns/qwen35-standalone4l-classification-base")# Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("mp-juuuns/qwen35-standalone4l-classification-base") model = AutoModel.from_pretrained("mp-juuuns/qwen35-standalone4l-classification-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add complete frozen benchmark metrics and bounded multilingual metadata
Browse files- .gitattributes +3 -1
- README.md +202 -59
- SHA256SUMS +22 -11
- benchmark/BENCHMARK_CARD.md +68 -0
- benchmark/arm_summary.csv +13 -0
- benchmark/compression_ladder.csv +5 -0
- benchmark/per_label_metrics.csv +127 -0
- benchmark/per_label_summary.csv +43 -0
- benchmark/reports/existing_specialized_separate_lineage-seed41.json +0 -0
- benchmark/reports/existing_specialized_separate_lineage-seed42.json +0 -0
- benchmark/reports/existing_specialized_separate_lineage-seed43.json +0 -0
- benchmark/reports/structural_copy_control-seed41.json +0 -0
- benchmark/reports/structural_copy_control-seed42.json +0 -0
- benchmark/reports/structural_copy_control-seed43.json +0 -0
- benchmark/reports/task_agnostic_base-seed41.json +0 -0
- benchmark/reports/task_agnostic_base-seed42.json +0 -0
- benchmark/reports/task_agnostic_base-seed43.json +0 -0
- benchmark/resource_metrics.csv +10 -0
- benchmark/seed_metrics.csv +10 -0
- benchmark/semeval-transfer-summary.json +876 -32
- benchmark/summary.json +891 -0
- docs/RELEASE_CONTRACT.md +63 -26
- models/semeval-propaganda/README.md +4 -2
- models/semeval-propaganda/seeds/seed42/tokenizer.json +0 -0
- models/semeval-propaganda/seeds/seed43/tokenizer.json +0 -0
- models/semeval-propaganda/tokenizer.json +0 -0
- release_manifest.json +101 -22
.gitattributes
CHANGED
|
@@ -1,2 +1,4 @@
|
|
| 1 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 2 |
-
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 1 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
models/semeval-propaganda/seeds/seed42/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
models/semeval-propaganda/seeds/seed43/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
models/semeval-propaganda/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -4,10 +4,12 @@ base_model: Qwen/Qwen3.5-0.8B
|
|
| 4 |
pipeline_tag: feature-extraction
|
| 5 |
library_name: transformers
|
| 6 |
language:
|
| 7 |
-
-
|
| 8 |
tags:
|
|
|
|
| 9 |
- qwen3.5
|
| 10 |
- classification-backbone
|
|
|
|
| 11 |
- knowledge-distillation
|
| 12 |
- model-compression
|
| 13 |
- edge-ai
|
|
@@ -15,18 +17,34 @@ tags:
|
|
| 15 |
|
| 16 |
# Qwen3.5 Standalone 4L Classification Base
|
| 17 |
|
| 18 |
-
This repository publishes a **headless,
|
| 19 |
-
Qwen3.5 text backbone**. The root model has no task labels and no
|
| 20 |
-
head. It is
|
| 21 |
-
classifiers, not
|
|
|
|
| 22 |
|
| 23 |
The repository follows one integrated 1-1-1 layout:
|
| 24 |
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
## Load the headless base
|
| 32 |
|
|
@@ -38,10 +56,8 @@ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
|
|
| 38 |
backbone = AutoModel.from_pretrained(model_id, trust_remote_code=False)
|
| 39 |
```
|
| 40 |
|
| 41 |
-
The root
|
| 42 |
-
|
| 43 |
-
the full upstream Qwen3.5 tokenizer. The published weight file contains no
|
| 44 |
-
`score.weight`.
|
| 45 |
|
| 46 |
## Make your own classifier
|
| 47 |
|
|
@@ -58,72 +74,199 @@ qwen35-distill finetune \
|
|
| 58 |
--output my-classifier-4l
|
| 59 |
```
|
| 60 |
|
| 61 |
-
For a full task-specific compression run,
|
| 62 |
-
|
| 63 |
-
|
| 64 |
[`distillation/README.md`](distillation/README.md).
|
| 65 |
|
| 66 |
## How the root weights were made
|
| 67 |
|
| 68 |
-
The text backbone and full tokenizer were extracted from
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
representation matching on unlabeled text.
|
| 72 |
|
| 73 |
The task-free training set was exactly 4,096 deterministically shuffled,
|
| 74 |
non-empty rows from the first locally cached WikiText-103 raw training shard.
|
| 75 |
-
No SemEval examples, labels, logits, thresholds,
|
| 76 |
-
were read while training
|
| 77 |
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
|
| 84 |
-
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
-
|
| 87 |
-
article-level protocol only as a transfer probe.
|
| 88 |
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
confidence interval.
|
| 100 |
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
|
| 105 |
## Separate SemEval model
|
| 106 |
|
| 107 |
-
The ready-to-use propaganda
|
| 108 |
[`models/semeval-propaganda/`](models/semeval-propaganda/). It preserves the
|
| 109 |
-
|
| 110 |
-
|
| 111 |
[`e7e0ff1`](https://huggingface.co/mp-juuuns/qwen35-standalone4l-propaganda-classifier/tree/e7e0ff16828052687d2e8dd7849e7a521629cf38).
|
| 112 |
|
| 113 |
-
That specialized model is **not** claimed to descend from this
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
|
|
|
| 117 |
|
| 118 |
## Limitations
|
| 119 |
|
| 120 |
- A classification head and labeled training are required before root-model
|
| 121 |
predictions have task meaning.
|
| 122 |
-
-
|
| 123 |
-
- Only one downstream
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
- Default layer maps are documented structural choices, not universal optima.
|
| 125 |
-
- Long-context,
|
| 126 |
-
|
| 127 |
- Do not use either model as a fact checker, safety oracle, or autonomous
|
| 128 |
decision maker.
|
| 129 |
|
|
@@ -134,7 +277,7 @@ and data terms. Qwen3.5 is attributed to Qwen. Transformers, PyTorch,
|
|
| 134 |
Hugging Face Hub, WikiText, and SemEval remain the work of their respective
|
| 135 |
authors. No WikiText or SemEval source records are redistributed.
|
| 136 |
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
`
|
| 140 |
-
|
|
|
|
| 4 |
pipeline_tag: feature-extraction
|
| 5 |
library_name: transformers
|
| 6 |
language:
|
| 7 |
+
- multilingual
|
| 8 |
tags:
|
| 9 |
+
- multilingual
|
| 10 |
- qwen3.5
|
| 11 |
- classification-backbone
|
| 12 |
+
- text-classification
|
| 13 |
- knowledge-distillation
|
| 14 |
- model-compression
|
| 15 |
- edge-ai
|
|
|
|
| 17 |
|
| 18 |
# Qwen3.5 Standalone 4L Classification Base
|
| 19 |
|
| 20 |
+
This repository publishes a **headless, multilingual-input, classification-ready
|
| 21 |
+
four-layer Qwen3.5 text backbone**. The root model has no task labels and no
|
| 22 |
+
classification head. It is a compact starting point for user-trained
|
| 23 |
+
single-label or multilabel classifiers, not a chat model or a ready-made
|
| 24 |
+
universal classifier.
|
| 25 |
|
| 26 |
The repository follows one integrated 1-1-1 layout:
|
| 27 |
|
| 28 |
+
1. **Repository root:** task-agnostic four-layer base trained on unlabeled
|
| 29 |
+
general text.
|
| 30 |
+
2. **`models/semeval-propaganda/`:** ready-to-use SemEval-derived propaganda
|
| 31 |
+
classifier preserved as a separate task-specific lineage.
|
| 32 |
+
3. **`distillation/`:** reusable 24L→8L→6L→4L platform for user-owned data.
|
| 33 |
+
|
| 34 |
+
## Model at a glance
|
| 35 |
+
|
| 36 |
+
| Item | Root model |
|
| 37 |
+
|---|---|
|
| 38 |
+
| Upstream | [`Qwen/Qwen3.5-0.8B`](https://huggingface.co/Qwen/Qwen3.5-0.8B) |
|
| 39 |
+
| Runtime class | `Qwen3_5TextModel` in Transformers 5.13.0 |
|
| 40 |
+
| Text layers | 4: `linear_attention → full_attention → linear_attention → full_attention` |
|
| 41 |
+
| Hidden / FFN | 1,024 / 3,584 |
|
| 42 |
+
| Vocabulary | full upstream Qwen3.5 tokenizer, 248,320 entries |
|
| 43 |
+
| Root parameters | 334,096,704 |
|
| 44 |
+
| Root BF16 weights | 668,198,976 bytes (637.24 MiB) |
|
| 45 |
+
| Output | hidden states; no LM head, task head, labels, or thresholds |
|
| 46 |
+
| Task-free KD data | 4,096 unlabeled WikiText-103 raw rows |
|
| 47 |
+
| Downstream evidence | one English 14-label SemEval-derived transfer probe |
|
| 48 |
|
| 49 |
## Load the headless base
|
| 50 |
|
|
|
|
| 56 |
backbone = AutoModel.from_pretrained(model_id, trust_remote_code=False)
|
| 57 |
```
|
| 58 |
|
| 59 |
+
The published root weight file contains no `score.weight`. A fresh task head and
|
| 60 |
+
labeled fine-tuning are required before outputs have class meaning.
|
|
|
|
|
|
|
| 61 |
|
| 62 |
## Make your own classifier
|
| 63 |
|
|
|
|
| 74 |
--output my-classifier-4l
|
| 75 |
```
|
| 76 |
|
| 77 |
+
For a full task-specific compression run, fine-tune a 24-layer teacher, then
|
| 78 |
+
repeat `materialize-classifier` and `distill-classifier` for `24to8`, `8to6`,
|
| 79 |
+
and `6to4`. Exact JSONL formats, split rules, manifests, and commands are in
|
| 80 |
[`distillation/README.md`](distillation/README.md).
|
| 81 |
|
| 82 |
## How the root weights were made
|
| 83 |
|
| 84 |
+
The text backbone and full tokenizer were extracted from the upstream
|
| 85 |
+
Qwen3.5-0.8B model. Students were trained sequentially through 24→8→6→4 using
|
| 86 |
+
hidden-boundary and final-representation matching on unlabeled text.
|
|
|
|
| 87 |
|
| 88 |
The task-free training set was exactly 4,096 deterministically shuffled,
|
| 89 |
non-empty rows from the first locally cached WikiText-103 raw training shard.
|
| 90 |
+
No SemEval examples, labels, logits, thresholds, classification heads, or
|
| 91 |
+
evidence/span objectives were read while training the root weights.
|
| 92 |
|
| 93 |
+
| KD stage | Selected source layers | Mean total loss | Mean interface loss | Mean final loss |
|
| 94 |
+
|---|---|---:|---:|---:|
|
| 95 |
+
| 24→8 | `0,4,6,11,13,16,20,23` | 0.176227 | 0.215628 | 0.166264 |
|
| 96 |
+
| 8→6 | `0,1,3,4,6,7` | 0.033821 | 0.044861 | 0.031147 |
|
| 97 |
+
| 6→4 | `0,2,3,5` | 0.047020 | 0.070484 | 0.043120 |
|
| 98 |
|
| 99 |
+
These are training-objective values, not downstream quality scores. Exact
|
| 100 |
+
parameter counts, BF16 file sizes, hashes, and stage values are in
|
| 101 |
+
[`benchmark/compression_ladder.csv`](benchmark/compression_ladder.csv) and
|
| 102 |
+
[`provenance/task_agnostic_stages/`](provenance/task_agnostic_stages/).
|
| 103 |
|
| 104 |
+
## Downstream benchmark contract
|
|
|
|
| 105 |
|
| 106 |
+
Because the root is headless, it cannot be scored as a classifier without
|
| 107 |
+
adaptation. Its task-appropriate benchmark is therefore a **controlled transfer
|
| 108 |
+
probe**:
|
| 109 |
+
|
| 110 |
+
- attach a fresh randomly initialized 14-label classification head;
|
| 111 |
+
- train all parameters for five epochs on the same SemEval-derived train split;
|
| 112 |
+
- select one global threshold on the epoch-5 calibration split;
|
| 113 |
+
- evaluate once per seed on the already-opened 55-article test split;
|
| 114 |
+
- compare against the same four-layer structural copy without task-free KD.
|
| 115 |
+
|
| 116 |
+
This derived task measures article-level presence of 14 propaganda techniques.
|
| 117 |
+
It is not the official SemEval span- or fragment-level task. Results are
|
| 118 |
+
exploratory because the public test split had already been opened.
|
| 119 |
+
|
| 120 |
+
### Frozen protocol
|
| 121 |
+
|
| 122 |
+
| Setting | Value |
|
| 123 |
+
|---|---|
|
| 124 |
+
| Seeds | 41, 42, 43 |
|
| 125 |
+
| Train / calibration / test articles | 260 / 56 / 55 |
|
| 126 |
+
| Test windows | 434 |
|
| 127 |
+
| Epochs | 5, no early stopping |
|
| 128 |
+
| Input windows | max length 256, overlap stride 128 |
|
| 129 |
+
| Article aggregation | label-wise maximum probability |
|
| 130 |
+
| Batch | micro 1, gradient accumulation 32, effective 32 |
|
| 131 |
+
| Optimizer | AdamW, LR 2e-5, weight decay 0.01, gradient clip 1.0 |
|
| 132 |
+
| Precision / device | BF16 / NVIDIA GeForce RTX 5070 Ti |
|
| 133 |
+
| Threshold selection | epoch-5 calibration Macro-F1, then Micro-F1 |
|
| 134 |
+
|
| 135 |
+
### Overall results by seed
|
| 136 |
+
|
| 137 |
+
| Initialization | Seed | Threshold | Macro-F1 | Micro-F1 | Exact match | Predicted positive rate |
|
| 138 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 139 |
+
| Task-agnostic KD base | 41 | 0.15 | 0.58831 | 0.64681 | 0.05455 | 0.57143 |
|
| 140 |
+
| Task-agnostic KD base | 42 | 0.30 | 0.60284 | 0.66552 | 0.03636 | 0.40909 |
|
| 141 |
+
| Task-agnostic KD base | 43 | 0.10 | 0.57128 | 0.64653 | 0.00000 | 0.51558 |
|
| 142 |
+
| Structural copy, no task-free KD | 41 | 0.05 | 0.51694 | 0.57740 | 0.00000 | 0.71299 |
|
| 143 |
+
| Structural copy, no task-free KD | 42 | 0.20 | 0.53494 | 0.60834 | 0.00000 | 0.62078 |
|
| 144 |
+
| Structural copy, no task-free KD | 43 | 0.20 | 0.50238 | 0.58564 | 0.00000 | 0.59610 |
|
| 145 |
+
|
| 146 |
+
### Three-seed summary
|
| 147 |
+
|
| 148 |
+
Values are mean ± sample SD across the three training seeds. The SD is not a
|
| 149 |
confidence interval.
|
| 150 |
|
| 151 |
+
| Initialization | Macro-F1 | Micro-F1 | Exact match | Predicted positive rate |
|
| 152 |
+
|---|---:|---:|---:|---:|
|
| 153 |
+
| **Task-agnostic KD base** | **0.58748 ± 0.01579** | **0.65295 ± 0.01088** | **0.03030 ± 0.02777** | 0.49870 ± 0.08248 |
|
| 154 |
+
| Structural copy, no task-free KD | 0.51809 ± 0.01631 | 0.59046 ± 0.01603 | 0.00000 ± 0.00000 | 0.64329 ± 0.06161 |
|
| 155 |
+
| Existing specialized 4L, separate lineage | 0.58757 ± 0.00717 | see nested reports | see nested reports | see nested reports |
|
| 156 |
+
|
| 157 |
+
The task-agnostic KD initialization improved same-seed mean Macro-F1 by
|
| 158 |
+
`+0.06939` over the structural-copy control. This is an observed comparison
|
| 159 |
+
under this protocol, not a significance or universal-superiority claim.
|
| 160 |
+
|
| 161 |
+
### Task-agnostic base: per-label test results
|
| 162 |
+
|
| 163 |
+
Precision, recall, and F1 are three-seed means. Support is the fixed number of
|
| 164 |
+
positive test articles, not a three-seed sum.
|
| 165 |
+
|
| 166 |
+
| Label | Support | Precision | Recall | F1 |
|
| 167 |
+
|---|---:|---:|---:|---:|
|
| 168 |
+
| `Appeal_to_Authority` | 11 | 0.3959 | 0.5758 | 0.4657 |
|
| 169 |
+
| `Appeal_to_fear-prejudice` | 24 | 0.5540 | 0.7361 | 0.6278 |
|
| 170 |
+
| `Bandwagon,Reductio_ad_hitlerum` | 7 | 0.4852 | 0.8095 | 0.6007 |
|
| 171 |
+
| `Black-and-White_Fallacy` | 12 | 0.3668 | 0.4722 | 0.4054 |
|
| 172 |
+
| `Causal_Oversimplification` | 18 | 0.5429 | 0.7963 | 0.6447 |
|
| 173 |
+
| `Doubt` | 22 | 0.5224 | 0.9848 | 0.6802 |
|
| 174 |
+
| `Exaggeration,Minimisation` | 22 | 0.4621 | 0.7727 | 0.5727 |
|
| 175 |
+
| `Flag-Waving` | 17 | 0.5643 | 0.8235 | 0.6465 |
|
| 176 |
+
| `Loaded_Language` | 45 | 0.8649 | 0.9852 | 0.9209 |
|
| 177 |
+
| `Name_Calling,Labeling` | 33 | 0.7381 | 0.9091 | 0.8125 |
|
| 178 |
+
| `Repetition` | 23 | 0.5197 | 0.7536 | 0.6148 |
|
| 179 |
+
| `Slogans` | 14 | 0.5161 | 0.7381 | 0.5835 |
|
| 180 |
+
| `Thought-terminating_Cliches` | 7 | 0.3492 | 0.3333 | 0.3190 |
|
| 181 |
+
| `Whataboutism,Straw_Men,Red_Herring` | 10 | 0.2762 | 0.5000 | 0.3304 |
|
| 182 |
+
|
| 183 |
+
Per-seed values and sample SDs for all labels and all arms are available in
|
| 184 |
+
[`benchmark/per_label_metrics.csv`](benchmark/per_label_metrics.csv) and
|
| 185 |
+
[`benchmark/per_label_summary.csv`](benchmark/per_label_summary.csv).
|
| 186 |
+
|
| 187 |
+
### Efficiency and resource measurements
|
| 188 |
+
|
| 189 |
+
These numbers belong to the fresh-head transfer checkpoints, not a standalone
|
| 190 |
+
headless-root latency benchmark.
|
| 191 |
+
|
| 192 |
+
| Measure | Task-agnostic transfer model |
|
| 193 |
+
|---|---:|
|
| 194 |
+
| Parameters with 14-label head | 334,111,040 |
|
| 195 |
+
| Test inference | 0.08131 ± 0.00243 s/article |
|
| 196 |
+
| Test batch | 55 articles / 434 windows |
|
| 197 |
+
| Peak CUDA allocated | 3,380,975,616 bytes (3.15 GiB) |
|
| 198 |
+
| Total train-and-evaluate wall time | 1,591.69 s mean (26.53 min) |
|
| 199 |
+
|
| 200 |
+
The timing is specific to the recorded RTX 5070 Ti, BF16, batch-1,
|
| 201 |
+
sliding-window protocol and includes article aggregation. It is not a claim
|
| 202 |
+
about CPU, mobile, UNO Q, or other deployment performance. Full seed-level
|
| 203 |
+
resource fields are in
|
| 204 |
+
[`benchmark/resource_metrics.csv`](benchmark/resource_metrics.csv).
|
| 205 |
+
|
| 206 |
+
### Machine-readable benchmark bundle
|
| 207 |
+
|
| 208 |
+
- [`benchmark/BENCHMARK_CARD.md`](benchmark/BENCHMARK_CARD.md): generated
|
| 209 |
+
benchmark report and interpretation
|
| 210 |
+
- [`benchmark/summary.json`](benchmark/summary.json): complete aggregate
|
| 211 |
+
- [`benchmark/seed_metrics.csv`](benchmark/seed_metrics.csv): overall metrics
|
| 212 |
+
per arm and seed
|
| 213 |
+
- [`benchmark/arm_summary.csv`](benchmark/arm_summary.csv): three-seed means
|
| 214 |
+
and sample SDs
|
| 215 |
+
- [`benchmark/per_label_metrics.csv`](benchmark/per_label_metrics.csv):
|
| 216 |
+
per-label values per arm and seed
|
| 217 |
+
- [`benchmark/per_label_summary.csv`](benchmark/per_label_summary.csv):
|
| 218 |
+
per-label means and sample SDs
|
| 219 |
+
- [`benchmark/resource_metrics.csv`](benchmark/resource_metrics.csv):
|
| 220 |
+
parameters, CUDA memory, wall time, and test timing
|
| 221 |
+
- [`benchmark/reports/`](benchmark/reports/): normalized full frozen reports,
|
| 222 |
+
including article-level probability vectors
|
| 223 |
+
|
| 224 |
+
## Multilingual scope
|
| 225 |
+
|
| 226 |
+
The root retains the upstream Qwen3.5 tokenizer and multilingual architecture.
|
| 227 |
+
The [official Qwen3.5-0.8B model card](https://huggingface.co/Qwen/Qwen3.5-0.8B)
|
| 228 |
+
states expanded support for **201 languages and dialects** and reports
|
| 229 |
+
multilingual upstream benchmarks such as MMMLU, MMLU-ProX, NOVA-63, INCLUDE,
|
| 230 |
+
Global PIQA, PolyMATH, WMT24++, and MAXIFE. This is why this repository carries
|
| 231 |
+
the Hugging Face `multilingual` language and model tag.
|
| 232 |
+
|
| 233 |
+
However, this four-layer root was distilled on English WikiText and its only
|
| 234 |
+
downstream transfer probe is English SemEval-derived data. **Tokenizer coverage
|
| 235 |
+
and inherited architecture support do not establish retained classification
|
| 236 |
+
quality in all 201 languages.** No multilingual downstream score is reported
|
| 237 |
+
for this root release. Users should fine-tune and evaluate on each intended
|
| 238 |
+
language and domain before making performance claims.
|
| 239 |
|
| 240 |
## Separate SemEval model
|
| 241 |
|
| 242 |
+
The ready-to-use propaganda classifier is under
|
| 243 |
[`models/semeval-propaganda/`](models/semeval-propaganda/). It preserves the
|
| 244 |
+
published seed-41/42/43 checkpoints and links to the original repository at
|
| 245 |
+
immutable revision
|
| 246 |
[`e7e0ff1`](https://huggingface.co/mp-juuuns/qwen35-standalone4l-propaganda-classifier/tree/e7e0ff16828052687d2e8dd7849e7a521629cf38).
|
| 247 |
|
| 248 |
+
That specialized model is **not** claimed to descend from this task-free base.
|
| 249 |
+
It has its own SemEval task-specific shrink/distillation and fine-tuning history
|
| 250 |
+
and a reduced 128k vocabulary, while the root retains the full upstream
|
| 251 |
+
tokenizer. Its complete reports remain under
|
| 252 |
+
[`models/semeval-propaganda/benchmark/`](models/semeval-propaganda/benchmark/).
|
| 253 |
|
| 254 |
## Limitations
|
| 255 |
|
| 256 |
- A classification head and labeled training are required before root-model
|
| 257 |
predictions have task meaning.
|
| 258 |
+
- Task-free KD used only 4,096 rows from one English WikiText shard.
|
| 259 |
+
- Only one downstream task and one hardware/software setting were measured.
|
| 260 |
+
- The 55-article test split was previously opened; results are exploratory.
|
| 261 |
+
- Per-label support ranges from 7 to 45 articles and rare-label estimates are
|
| 262 |
+
unstable.
|
| 263 |
+
- Three seeds describe run-to-run variation; they are not a confidence
|
| 264 |
+
interval.
|
| 265 |
+
- Multilingual input support has not been validated as multilingual downstream
|
| 266 |
+
classification quality for this four-layer root.
|
| 267 |
- Default layer maps are documented structural choices, not universal optima.
|
| 268 |
+
- Long-context, calibration, robustness, fairness, and production safety have
|
| 269 |
+
not been established.
|
| 270 |
- Do not use either model as a fact checker, safety oracle, or autonomous
|
| 271 |
decision maker.
|
| 272 |
|
|
|
|
| 277 |
Hugging Face Hub, WikiText, and SemEval remain the work of their respective
|
| 278 |
authors. No WikiText or SemEval source records are redistributed.
|
| 279 |
|
| 280 |
+
The WikiText page currently has a license wording discrepancy: metadata lists
|
| 281 |
+
CC BY-SA 3.0 and GFDL, while prose says CC BY-SA 4.0. Users should inspect
|
| 282 |
+
[`Salesforce/wikitext`](https://huggingface.co/datasets/Salesforce/wikitext)
|
| 283 |
+
directly.
|
SHA256SUMS
CHANGED
|
@@ -1,13 +1,24 @@
|
|
| 1 |
ee05350c7fda0af4da116020dd7b976d731de3beed4f5e717d5bcc95ead24550 .gitattributes
|
| 2 |
77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049 LICENSE
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
04b007131663760bf3e581e5a953be77044014e87efe1d2a6ca4b72ec0eac978 chat_template.jinja
|
| 12 |
bb33f3a52d787e15db7d516c6de36a7ddd2c675e7964d5b9309131c4e5d365f0 config.json
|
| 13 |
f5b7248f7781412e8b5d8f2fb3b0c309568e4e75baf55029b26a6fa985d9d608 distillation/README.md
|
|
@@ -25,11 +36,11 @@ a9c49af0ee14b307a08c096bcd1ae55fa9bc67bdb807fa9383fc313246e7edd7 distillation/q
|
|
| 25 |
d9b135cf5b0adf4bbfc321200395fbd6112e49fd1a027b30262bcff1c4c0b654 distillation/qwen35_distill/losses.py
|
| 26 |
00731ae7838317105f9fff960e3da210b2a55954feb37b3ac793bcfc81fdab2d distillation/qwen35_distill/schema.py
|
| 27 |
4dc432acb70e03848d2e5964e8550065490fda3857aac32fe3b79895ef72026f distillation/qwen35_distill/training.py
|
| 28 |
-
|
| 29 |
a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d merges.txt
|
| 30 |
2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3 model.safetensors
|
| 31 |
77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049 models/semeval-propaganda/LICENSE
|
| 32 |
-
|
| 33 |
0bae347bb54f8089a5fa5450f8366b5c56eb3e598beaf9a87f4385363a402bcb models/semeval-propaganda/benchmark/final4l_aggregate.json
|
| 34 |
c1bd379680abcf50b47b7d23e4aafbc8b5e7f40a0b0e9700bde21a8ce49af803 models/semeval-propaganda/benchmark/seed41.json
|
| 35 |
a30e38433d008db4aa7e5fbfd477ab80a0daf50e970f8e7b04397efc1a47843a models/semeval-propaganda/benchmark/seed42.json
|
|
@@ -59,7 +70,7 @@ ec734855870758cbfef99031dac2ded85b54c496532e5c514a9a4f63287c3493 provenance/bas
|
|
| 59 |
b9ce99b993c4f4563bc2521fc731bfd82f5e8149c141133599fbd8f982ba43d9 provenance/task_agnostic_stages/24to8.json
|
| 60 |
ec734855870758cbfef99031dac2ded85b54c496532e5c514a9a4f63287c3493 provenance/task_agnostic_stages/6to4.json
|
| 61 |
65e5383afbbcfe0b7dd78a5411c319ed6bf2586659fb1892aca9ecbd5dfd4993 provenance/task_agnostic_stages/8to6.json
|
| 62 |
-
|
| 63 |
7b48e7811e56e552fc4aef44189e1d7c9b1352c85d2b3f10ff0bdd1ea5cdd824 requirements.txt
|
| 64 |
06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523 tokenizer.json
|
| 65 |
5ab9bed0a4d27949672f65ba1141d6dd5b0514fb9091d3426506ecebb5d5e294 tokenizer_config.json
|
|
|
|
| 1 |
ee05350c7fda0af4da116020dd7b976d731de3beed4f5e717d5bcc95ead24550 .gitattributes
|
| 2 |
77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049 LICENSE
|
| 3 |
+
5fcf007de6b6e828711c874310de1e042f1dada6989c70930ac0881285a9dc61 README.md
|
| 4 |
+
bf4073de38c3905effd2dcce6f38894f325c8a0a79bedfb2779f0545c86b9c9f benchmark/BENCHMARK_CARD.md
|
| 5 |
+
e01bd7ac59a3212e44b77e5fcfbc7c41ec91d74c998306d2438209d6ef1a3aa7 benchmark/arm_summary.csv
|
| 6 |
+
2c544c3907c4f592d724f658850b8c6b9634845cb5b50a66ebd2ea6c5d48ce0b benchmark/compression_ladder.csv
|
| 7 |
+
ffc21be32b2106c7bf1df3854097a975ac370da01ac1b732eb5c2ba9075ddeda benchmark/per_label_metrics.csv
|
| 8 |
+
c8e3876abf6cfe960c9332d3bc9d1fbde2f3213f3111351acf83b21bbb184049 benchmark/per_label_summary.csv
|
| 9 |
+
c1bd379680abcf50b47b7d23e4aafbc8b5e7f40a0b0e9700bde21a8ce49af803 benchmark/reports/existing_specialized_separate_lineage-seed41.json
|
| 10 |
+
a30e38433d008db4aa7e5fbfd477ab80a0daf50e970f8e7b04397efc1a47843a benchmark/reports/existing_specialized_separate_lineage-seed42.json
|
| 11 |
+
96466290d09d27b8bc044a9f9a95765ce12b4d7b02dfb0123e436e7535623353 benchmark/reports/existing_specialized_separate_lineage-seed43.json
|
| 12 |
+
816c3249d3ec281ef73d34dc3ab7576a48415d250c32977eb94ca37f73b0b810 benchmark/reports/structural_copy_control-seed41.json
|
| 13 |
+
0b19eabe1e31d0012c754db3b31173f8a6fcdee30973f1a1bebd3a370d11064c benchmark/reports/structural_copy_control-seed42.json
|
| 14 |
+
6dde34173462eac50141ff082c18194f3233f8c06853532d9b12756562e69e2e benchmark/reports/structural_copy_control-seed43.json
|
| 15 |
+
6b3a2e0372a83e1e71c2eca9596efc8cd1a3e39c28e25e3373f195431b5858a4 benchmark/reports/task_agnostic_base-seed41.json
|
| 16 |
+
42dc788983716be0e9847aad87ff79c7e7297884f2635a8d098639be77326639 benchmark/reports/task_agnostic_base-seed42.json
|
| 17 |
+
7ee52dc94a152a2f767dcff171f3c9cc1072fabd3ebc7b14db53617b7b1d33d5 benchmark/reports/task_agnostic_base-seed43.json
|
| 18 |
+
bf5cefdec3b2f3646114bd18bbf8e5e39721bde768ba968d73f22e96fc9ca8cf benchmark/resource_metrics.csv
|
| 19 |
+
e0b08a958d25243158a03dff76ca153d2ded76cc3f1771e4046a0068688ad6a8 benchmark/seed_metrics.csv
|
| 20 |
+
3d2c9fcf9c6cc7da1a885968b70a42eaa38659d317c9d3290a2b3066f3f9ee6a benchmark/semeval-transfer-summary.json
|
| 21 |
+
3d2c9fcf9c6cc7da1a885968b70a42eaa38659d317c9d3290a2b3066f3f9ee6a benchmark/summary.json
|
| 22 |
04b007131663760bf3e581e5a953be77044014e87efe1d2a6ca4b72ec0eac978 chat_template.jinja
|
| 23 |
bb33f3a52d787e15db7d516c6de36a7ddd2c675e7964d5b9309131c4e5d365f0 config.json
|
| 24 |
f5b7248f7781412e8b5d8f2fb3b0c309568e4e75baf55029b26a6fa985d9d608 distillation/README.md
|
|
|
|
| 36 |
d9b135cf5b0adf4bbfc321200395fbd6112e49fd1a027b30262bcff1c4c0b654 distillation/qwen35_distill/losses.py
|
| 37 |
00731ae7838317105f9fff960e3da210b2a55954feb37b3ac793bcfc81fdab2d distillation/qwen35_distill/schema.py
|
| 38 |
4dc432acb70e03848d2e5964e8550065490fda3857aac32fe3b79895ef72026f distillation/qwen35_distill/training.py
|
| 39 |
+
1b6e475b9d7d9af52bb8845de020303b1c6e89646d8e4f9be17dbe4ab2910417 docs/RELEASE_CONTRACT.md
|
| 40 |
a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d merges.txt
|
| 41 |
2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3 model.safetensors
|
| 42 |
77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049 models/semeval-propaganda/LICENSE
|
| 43 |
+
bf4ba7cc2514502a28f424507c41c1cfce5089c051ba7500cf46c1d13860722f models/semeval-propaganda/README.md
|
| 44 |
0bae347bb54f8089a5fa5450f8366b5c56eb3e598beaf9a87f4385363a402bcb models/semeval-propaganda/benchmark/final4l_aggregate.json
|
| 45 |
c1bd379680abcf50b47b7d23e4aafbc8b5e7f40a0b0e9700bde21a8ce49af803 models/semeval-propaganda/benchmark/seed41.json
|
| 46 |
a30e38433d008db4aa7e5fbfd477ab80a0daf50e970f8e7b04397efc1a47843a models/semeval-propaganda/benchmark/seed42.json
|
|
|
|
| 70 |
b9ce99b993c4f4563bc2521fc731bfd82f5e8149c141133599fbd8f982ba43d9 provenance/task_agnostic_stages/24to8.json
|
| 71 |
ec734855870758cbfef99031dac2ded85b54c496532e5c514a9a4f63287c3493 provenance/task_agnostic_stages/6to4.json
|
| 72 |
65e5383afbbcfe0b7dd78a5411c319ed6bf2586659fb1892aca9ecbd5dfd4993 provenance/task_agnostic_stages/8to6.json
|
| 73 |
+
207e1df5765c838b7d355a0c34cf5ae2c55f90b419cff0f4bbf80528a48fd939 release_manifest.json
|
| 74 |
7b48e7811e56e552fc4aef44189e1d7c9b1352c85d2b3f10ff0bdd1ea5cdd824 requirements.txt
|
| 75 |
06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523 tokenizer.json
|
| 76 |
5ab9bed0a4d27949672f65ba1141d6dd5b0514fb9091d3426506ecebb5d5e294 tokenizer_config.json
|
benchmark/BENCHMARK_CARD.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Complete frozen transfer benchmark
|
| 2 |
+
|
| 3 |
+
This report is generated from the nine frozen seed reports distributed in
|
| 4 |
+
`benchmark/reports/`. It evaluates a headless base through fresh-head
|
| 5 |
+
fine-tuning; it is not a zero-shot root-model score.
|
| 6 |
+
|
| 7 |
+
## Scope
|
| 8 |
+
|
| 9 |
+
- Derived task: article-level presence of 14 propaganda techniques.
|
| 10 |
+
- Test: 55 previously opened English articles / 434 windows.
|
| 11 |
+
- Seeds: 41, 42, 43.
|
| 12 |
+
- Error bars: three-seed sample SD, not confidence intervals.
|
| 13 |
+
- No multilingual downstream evaluation was performed.
|
| 14 |
+
|
| 15 |
+
## Overall metrics
|
| 16 |
+
|
| 17 |
+
| Arm | Macro-F1 | Micro-F1 | Exact match | Positive rate |
|
| 18 |
+
|---|---:|---:|---:|---:|
|
| 19 |
+
| Task-agnostic KD base | 0.58748 ± 0.01579 | 0.65295 ± 0.01088 | 0.03030 ± 0.02777 | 0.49870 ± 0.08248 |
|
| 20 |
+
| Structural copy, no task-free KD | 0.51809 ± 0.01631 | 0.59046 ± 0.01603 | 0.00000 ± 0.00000 | 0.64329 ± 0.06161 |
|
| 21 |
+
| Existing specialized 4L, separate lineage | 0.58757 ± 0.00717 | 0.64798 ± 0.00187 | 0.00606 ± 0.01050 | 0.58052 ± 0.02848 |
|
| 22 |
+
|
| 23 |
+
The observed mean same-seed Macro-F1 delta between the task-agnostic
|
| 24 |
+
base and structural control is `+0.06939`.
|
| 25 |
+
The specialized arm is a separate historical lineage and is reference-only.
|
| 26 |
+
|
| 27 |
+
## Task-agnostic base per-label metrics
|
| 28 |
+
|
| 29 |
+
| Label | Support | Precision | Recall | F1 |
|
| 30 |
+
|---|---:|---:|---:|---:|
|
| 31 |
+
| `Appeal_to_Authority` | 11 | 0.3959 | 0.5758 | 0.4657 |
|
| 32 |
+
| `Appeal_to_fear-prejudice` | 24 | 0.5540 | 0.7361 | 0.6278 |
|
| 33 |
+
| `Bandwagon,Reductio_ad_hitlerum` | 7 | 0.4852 | 0.8095 | 0.6007 |
|
| 34 |
+
| `Black-and-White_Fallacy` | 12 | 0.3668 | 0.4722 | 0.4054 |
|
| 35 |
+
| `Causal_Oversimplification` | 18 | 0.5429 | 0.7963 | 0.6447 |
|
| 36 |
+
| `Doubt` | 22 | 0.5224 | 0.9848 | 0.6802 |
|
| 37 |
+
| `Exaggeration,Minimisation` | 22 | 0.4621 | 0.7727 | 0.5727 |
|
| 38 |
+
| `Flag-Waving` | 17 | 0.5643 | 0.8235 | 0.6465 |
|
| 39 |
+
| `Loaded_Language` | 45 | 0.8649 | 0.9852 | 0.9209 |
|
| 40 |
+
| `Name_Calling,Labeling` | 33 | 0.7381 | 0.9091 | 0.8125 |
|
| 41 |
+
| `Repetition` | 23 | 0.5197 | 0.7536 | 0.6148 |
|
| 42 |
+
| `Slogans` | 14 | 0.5161 | 0.7381 | 0.5835 |
|
| 43 |
+
| `Thought-terminating_Cliches` | 7 | 0.3492 | 0.3333 | 0.3190 |
|
| 44 |
+
| `Whataboutism,Straw_Men,Red_Herring` | 10 | 0.2762 | 0.5000 | 0.3304 |
|
| 45 |
+
|
| 46 |
+
## Resource scope
|
| 47 |
+
|
| 48 |
+
Resource rows describe the fresh-head transfer checkpoints on the recorded
|
| 49 |
+
RTX 5070 Ti BF16 sliding-window protocol. They are not a headless-root CPU
|
| 50 |
+
or edge-device benchmark. See `resource_metrics.csv` for every seed.
|
| 51 |
+
|
| 52 |
+
## Files
|
| 53 |
+
|
| 54 |
+
- `summary.json`: aggregate and interpretation contract.
|
| 55 |
+
- `seed_metrics.csv`: overall metrics and thresholds by arm/seed.
|
| 56 |
+
- `arm_summary.csv`: means and sample SDs.
|
| 57 |
+
- `per_label_metrics.csv`: precision/recall/F1/support by arm/seed/label.
|
| 58 |
+
- `per_label_summary.csv`: per-label means and sample SDs.
|
| 59 |
+
- `resource_metrics.csv`: parameters, timing, and CUDA memory.
|
| 60 |
+
- `compression_ladder.csv`: 24L→8L→6L→4L size and KD evidence.
|
| 61 |
+
- `reports/`: normalized full frozen reports, including probabilities.
|
| 62 |
+
|
| 63 |
+
## Interpretation boundary
|
| 64 |
+
|
| 65 |
+
The opened test and prior model-selection history prevent confirmatory
|
| 66 |
+
inference. Three seeds do not represent 165 independent observations.
|
| 67 |
+
Upstream support for 201 languages/dialects motivates multilingual metadata,
|
| 68 |
+
but this four-layer root has no direct multilingual downstream score.
|
benchmark/arm_summary.csv
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
arm,metric,values,mean,sample_sd,n
|
| 2 |
+
task_agnostic_base,macro_f1,"[0.5883093293517536, 0.6028379036218096, 0.5712816197954101]",0.5874762842563245,0.01579462678670608,3
|
| 3 |
+
task_agnostic_base,micro_f1,"[0.6468085106382978, 0.6655172413793103, 0.6465256797583082]",0.6529504772586388,0.010884055708931178,3
|
| 4 |
+
task_agnostic_base,exact_match,"[0.05454545454545454, 0.03636363636363636, 0.0]",0.0303030303030303,0.027773186030035393,3
|
| 5 |
+
task_agnostic_base,predicted_positive_rate,"[0.5714285714285714, 0.4090909090909091, 0.5155844155844156]",0.4987012987012987,0.08247520160679181,3
|
| 6 |
+
structural_copy_control,macro_f1,"[0.5169424579262757, 0.5349405779540175, 0.5023778364496152]",0.5180869574433028,0.016311512588628545,3
|
| 7 |
+
structural_copy_control,micro_f1,"[0.5773955773955775, 0.6083445491251682, 0.585635359116022]",0.5904584952122559,0.01602830914762526,3
|
| 8 |
+
structural_copy_control,exact_match,"[0.0, 0.0, 0.0]",0.0,0.0,3
|
| 9 |
+
structural_copy_control,predicted_positive_rate,"[0.712987012987013, 0.6207792207792208, 0.5961038961038961]",0.6432900432900432,0.06160737456395461,3
|
| 10 |
+
existing_specialized_separate_lineage,macro_f1,"[0.5832848452288824, 0.5958405181160762, 0.5835785045218137]",0.5875679559555907,0.007165753447433348,3
|
| 11 |
+
existing_specialized_separate_lineage,micro_f1,"[0.6494252873563218, 0.6458616010854816, 0.6486486486486486]",0.647978512363484,0.0018739735696546695,3
|
| 12 |
+
existing_specialized_separate_lineage,exact_match,"[0.0, 0.0, 0.01818181818181818]",0.006060606060606061,0.01049727762162956,3
|
| 13 |
+
existing_specialized_separate_lineage,predicted_positive_rate,"[0.5597402597402598, 0.612987012987013, 0.5688311688311688]",0.5805194805194805,0.02848274311618351,3
|
benchmark/compression_ladder.csv
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name,layers,parameters,weight_bytes,weight_mib,weight_sha256,kd_stage,mean_total_loss,mean_interface_loss,mean_final_loss
|
| 2 |
+
24L upstream text teacher,24,752393024,1504825632,1435.1135559082031,30783da4651259c26b8a204821ea426b9526431e0c14da0d1c6fa284e7414e5e,,,,
|
| 3 |
+
8L task-free KD,8,420318144,840647880,801.7042922973633,086fbaf9a4838ecde56cd3541c9c3212b22d0c5ee745cf2c1c5e0bb31e0ec22c,24to8,0.17622653172702485,0.21562811763578793,0.16626427527853593
|
| 4 |
+
6L task-free KD,6,377207424,754423424,719.4742431640625,492749c20f77a42f6bc993f0927a82217b94f872b7eacf25612f7520c9bc28ce,8to6,0.03382104352249371,0.04486069045515251,0.031147060785087888
|
| 5 |
+
4L task-free KD root,4,334096704,668198976,637.2442016601562,2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3,6to4,0.047020394468859195,0.07048421185527332,0.043119939065377366
|
benchmark/per_label_metrics.csv
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
arm,seed,label,support,precision,recall,f1
|
| 2 |
+
task_agnostic_base,41,Appeal_to_Authority,11,0.3333333333333333,0.5454545454545454,0.41379310344827586
|
| 3 |
+
task_agnostic_base,41,Appeal_to_fear-prejudice,24,0.5757575757575758,0.7916666666666666,0.6666666666666667
|
| 4 |
+
task_agnostic_base,41,"Bandwagon,Reductio_ad_hitlerum",7,0.5,0.8571428571428571,0.631578947368421
|
| 5 |
+
task_agnostic_base,41,Black-and-White_Fallacy,12,0.3333333333333333,0.5833333333333334,0.4242424242424242
|
| 6 |
+
task_agnostic_base,41,Causal_Oversimplification,18,0.5,0.6666666666666666,0.5714285714285715
|
| 7 |
+
task_agnostic_base,41,Doubt,22,0.4782608695652174,1.0,0.6470588235294118
|
| 8 |
+
task_agnostic_base,41,"Exaggeration,Minimisation",22,0.4523809523809524,0.8636363636363636,0.59375
|
| 9 |
+
task_agnostic_base,41,Flag-Waving,17,0.4594594594594595,1.0,0.6296296296296297
|
| 10 |
+
task_agnostic_base,41,Loaded_Language,45,0.9183673469387755,1.0,0.9574468085106383
|
| 11 |
+
task_agnostic_base,41,"Name_Calling,Labeling",33,0.7333333333333333,1.0,0.846153846153846
|
| 12 |
+
task_agnostic_base,41,Repetition,23,0.5428571428571428,0.8260869565217391,0.6551724137931034
|
| 13 |
+
task_agnostic_base,41,Slogans,14,0.41379310344827586,0.8571428571428571,0.5581395348837208
|
| 14 |
+
task_agnostic_base,41,Thought-terminating_Cliches,7,0.21428571428571427,0.42857142857142855,0.2857142857142857
|
| 15 |
+
task_agnostic_base,41,"Whataboutism,Straw_Men,Red_Herring",10,0.22857142857142856,0.8,0.3555555555555555
|
| 16 |
+
task_agnostic_base,42,Appeal_to_Authority,11,0.5384615384615384,0.6363636363636364,0.5833333333333334
|
| 17 |
+
task_agnostic_base,42,Appeal_to_fear-prejudice,24,0.56,0.5833333333333334,0.5714285714285714
|
| 18 |
+
task_agnostic_base,42,"Bandwagon,Reductio_ad_hitlerum",7,0.5555555555555556,0.7142857142857143,0.6250000000000001
|
| 19 |
+
task_agnostic_base,42,Black-and-White_Fallacy,12,0.45454545454545453,0.4166666666666667,0.43478260869565216
|
| 20 |
+
task_agnostic_base,42,Causal_Oversimplification,18,0.5769230769230769,0.8333333333333334,0.6818181818181818
|
| 21 |
+
task_agnostic_base,42,Doubt,22,0.6,0.9545454545454546,0.7368421052631579
|
| 22 |
+
task_agnostic_base,42,"Exaggeration,Minimisation",22,0.48148148148148145,0.5909090909090909,0.5306122448979591
|
| 23 |
+
task_agnostic_base,42,Flag-Waving,17,0.7333333333333333,0.6470588235294118,0.6875
|
| 24 |
+
task_agnostic_base,42,Loaded_Language,45,0.8461538461538461,0.9777777777777777,0.9072164948453608
|
| 25 |
+
task_agnostic_base,42,"Name_Calling,Labeling",33,0.7428571428571429,0.7878787878787878,0.7647058823529412
|
| 26 |
+
task_agnostic_base,42,Repetition,23,0.5,0.7391304347826086,0.5964912280701754
|
| 27 |
+
task_agnostic_base,42,Slogans,14,0.75,0.6428571428571429,0.6923076923076924
|
| 28 |
+
task_agnostic_base,42,Thought-terminating_Cliches,7,0.3333333333333333,0.2857142857142857,0.30769230769230765
|
| 29 |
+
task_agnostic_base,42,"Whataboutism,Straw_Men,Red_Herring",10,0.26666666666666666,0.4,0.32
|
| 30 |
+
task_agnostic_base,43,Appeal_to_Authority,11,0.3157894736842105,0.5454545454545454,0.39999999999999997
|
| 31 |
+
task_agnostic_base,43,Appeal_to_fear-prejudice,24,0.5263157894736842,0.8333333333333334,0.6451612903225806
|
| 32 |
+
task_agnostic_base,43,"Bandwagon,Reductio_ad_hitlerum",7,0.4,0.8571428571428571,0.5454545454545455
|
| 33 |
+
task_agnostic_base,43,Black-and-White_Fallacy,12,0.3125,0.4166666666666667,0.35714285714285715
|
| 34 |
+
task_agnostic_base,43,Causal_Oversimplification,18,0.5517241379310345,0.8888888888888888,0.6808510638297872
|
| 35 |
+
task_agnostic_base,43,Doubt,22,0.4888888888888889,1.0,0.6567164179104478
|
| 36 |
+
task_agnostic_base,43,"Exaggeration,Minimisation",22,0.4523809523809524,0.8636363636363636,0.59375
|
| 37 |
+
task_agnostic_base,43,Flag-Waving,17,0.5,0.8235294117647058,0.6222222222222222
|
| 38 |
+
task_agnostic_base,43,Loaded_Language,45,0.8301886792452831,0.9777777777777777,0.8979591836734695
|
| 39 |
+
task_agnostic_base,43,"Name_Calling,Labeling",33,0.7380952380952381,0.9393939393939394,0.8266666666666667
|
| 40 |
+
task_agnostic_base,43,Repetition,23,0.5161290322580645,0.6956521739130435,0.5925925925925926
|
| 41 |
+
task_agnostic_base,43,Slogans,14,0.38461538461538464,0.7142857142857143,0.5
|
| 42 |
+
task_agnostic_base,43,Thought-terminating_Cliches,7,0.5,0.2857142857142857,0.36363636363636365
|
| 43 |
+
task_agnostic_base,43,"Whataboutism,Straw_Men,Red_Herring",10,0.3333333333333333,0.3,0.3157894736842105
|
| 44 |
+
structural_copy_control,41,Appeal_to_Authority,11,0.2962962962962963,0.7272727272727273,0.42105263157894735
|
| 45 |
+
structural_copy_control,41,Appeal_to_fear-prejudice,24,0.4666666666666667,0.875,0.608695652173913
|
| 46 |
+
structural_copy_control,41,"Bandwagon,Reductio_ad_hitlerum",7,0.2857142857142857,0.8571428571428571,0.42857142857142855
|
| 47 |
+
structural_copy_control,41,Black-and-White_Fallacy,12,0.2222222222222222,0.3333333333333333,0.26666666666666666
|
| 48 |
+
structural_copy_control,41,Causal_Oversimplification,18,0.43243243243243246,0.8888888888888888,0.5818181818181818
|
| 49 |
+
structural_copy_control,41,Doubt,22,0.4,1.0,0.5714285714285715
|
| 50 |
+
structural_copy_control,41,"Exaggeration,Minimisation",22,0.425531914893617,0.9090909090909091,0.5797101449275361
|
| 51 |
+
structural_copy_control,41,Flag-Waving,17,0.37777777777777777,1.0,0.5483870967741935
|
| 52 |
+
structural_copy_control,41,Loaded_Language,45,0.8333333333333334,1.0,0.9090909090909091
|
| 53 |
+
structural_copy_control,41,"Name_Calling,Labeling",33,0.6,1.0,0.7499999999999999
|
| 54 |
+
structural_copy_control,41,Repetition,23,0.46,1.0,0.6301369863013699
|
| 55 |
+
structural_copy_control,41,Slogans,14,0.2857142857142857,0.5714285714285714,0.38095238095238093
|
| 56 |
+
structural_copy_control,41,Thought-terminating_Cliches,7,0.125,0.5714285714285714,0.20512820512820512
|
| 57 |
+
structural_copy_control,41,"Whataboutism,Straw_Men,Red_Herring",10,0.22857142857142856,0.8,0.3555555555555555
|
| 58 |
+
structural_copy_control,42,Appeal_to_Authority,11,0.1875,0.2727272727272727,0.2222222222222222
|
| 59 |
+
structural_copy_control,42,Appeal_to_fear-prejudice,24,0.5384615384615384,0.875,0.6666666666666667
|
| 60 |
+
structural_copy_control,42,"Bandwagon,Reductio_ad_hitlerum",7,0.35,1.0,0.5185185185185185
|
| 61 |
+
structural_copy_control,42,Black-and-White_Fallacy,12,0.38461538461538464,0.8333333333333334,0.5263157894736842
|
| 62 |
+
structural_copy_control,42,Causal_Oversimplification,18,0.4473684210526316,0.9444444444444444,0.6071428571428572
|
| 63 |
+
structural_copy_control,42,Doubt,22,0.4583333333333333,1.0,0.6285714285714286
|
| 64 |
+
structural_copy_control,42,"Exaggeration,Minimisation",22,0.46511627906976744,0.9090909090909091,0.6153846153846153
|
| 65 |
+
structural_copy_control,42,Flag-Waving,17,0.43333333333333335,0.7647058823529411,0.5531914893617021
|
| 66 |
+
structural_copy_control,42,Loaded_Language,45,0.8490566037735849,1.0,0.9183673469387755
|
| 67 |
+
structural_copy_control,42,"Name_Calling,Labeling",33,0.6666666666666666,0.9696969696969697,0.7901234567901234
|
| 68 |
+
structural_copy_control,42,Repetition,23,0.43137254901960786,0.9565217391304348,0.5945945945945946
|
| 69 |
+
structural_copy_control,42,Slogans,14,0.3888888888888889,0.5,0.43750000000000006
|
| 70 |
+
structural_copy_control,42,Thought-terminating_Cliches,7,0.11764705882352941,0.2857142857142857,0.16666666666666666
|
| 71 |
+
structural_copy_control,42,"Whataboutism,Straw_Men,Red_Herring",10,0.16129032258064516,0.5,0.24390243902439024
|
| 72 |
+
structural_copy_control,43,Appeal_to_Authority,11,0.2962962962962963,0.7272727272727273,0.42105263157894735
|
| 73 |
+
structural_copy_control,43,Appeal_to_fear-prejudice,24,0.5263157894736842,0.8333333333333334,0.6451612903225806
|
| 74 |
+
structural_copy_control,43,"Bandwagon,Reductio_ad_hitlerum",7,0.22727272727272727,0.7142857142857143,0.3448275862068965
|
| 75 |
+
structural_copy_control,43,Black-and-White_Fallacy,12,0.3333333333333333,0.4166666666666667,0.3703703703703704
|
| 76 |
+
structural_copy_control,43,Causal_Oversimplification,18,0.4594594594594595,0.9444444444444444,0.6181818181818182
|
| 77 |
+
structural_copy_control,43,Doubt,22,0.4782608695652174,1.0,0.6470588235294118
|
| 78 |
+
structural_copy_control,43,"Exaggeration,Minimisation",22,0.4186046511627907,0.8181818181818182,0.5538461538461539
|
| 79 |
+
structural_copy_control,43,Flag-Waving,17,0.3333333333333333,0.6470588235294118,0.44
|
| 80 |
+
structural_copy_control,43,Loaded_Language,45,0.8653846153846154,1.0,0.9278350515463918
|
| 81 |
+
structural_copy_control,43,"Name_Calling,Labeling",33,0.6458333333333334,0.9393939393939394,0.7654320987654321
|
| 82 |
+
structural_copy_control,43,Repetition,23,0.45454545454545453,0.8695652173913043,0.5970149253731344
|
| 83 |
+
structural_copy_control,43,Slogans,14,0.3076923076923077,0.2857142857142857,0.29629629629629634
|
| 84 |
+
structural_copy_control,43,Thought-terminating_Cliches,7,0.1,0.2857142857142857,0.14814814814814817
|
| 85 |
+
structural_copy_control,43,"Whataboutism,Straw_Men,Red_Herring",10,0.19047619047619047,0.4,0.25806451612903225
|
| 86 |
+
existing_specialized_separate_lineage,41,Appeal_to_Authority,11,0.5,0.6363636363636364,0.56
|
| 87 |
+
existing_specialized_separate_lineage,41,Appeal_to_fear-prejudice,24,0.5,0.9166666666666666,0.6470588235294118
|
| 88 |
+
existing_specialized_separate_lineage,41,"Bandwagon,Reductio_ad_hitlerum",7,0.4444444444444444,0.5714285714285714,0.5
|
| 89 |
+
existing_specialized_separate_lineage,41,Black-and-White_Fallacy,12,0.4666666666666667,0.5833333333333334,0.5185185185185186
|
| 90 |
+
existing_specialized_separate_lineage,41,Causal_Oversimplification,18,0.46153846153846156,0.6666666666666666,0.5454545454545455
|
| 91 |
+
existing_specialized_separate_lineage,41,Doubt,22,0.5238095238095238,1.0,0.6875000000000001
|
| 92 |
+
existing_specialized_separate_lineage,41,"Exaggeration,Minimisation",22,0.4418604651162791,0.8636363636363636,0.5846153846153846
|
| 93 |
+
existing_specialized_separate_lineage,41,Flag-Waving,17,0.53125,1.0,0.6938775510204082
|
| 94 |
+
existing_specialized_separate_lineage,41,Loaded_Language,45,0.8490566037735849,1.0,0.9183673469387755
|
| 95 |
+
existing_specialized_separate_lineage,41,"Name_Calling,Labeling",33,0.717391304347826,1.0,0.8354430379746834
|
| 96 |
+
existing_specialized_separate_lineage,41,Repetition,23,0.46938775510204084,1.0,0.6388888888888888
|
| 97 |
+
existing_specialized_separate_lineage,41,Slogans,14,0.5,0.5,0.5
|
| 98 |
+
existing_specialized_separate_lineage,41,Thought-terminating_Cliches,7,0.14285714285714285,0.5714285714285714,0.2285714285714286
|
| 99 |
+
existing_specialized_separate_lineage,41,"Whataboutism,Straw_Men,Red_Herring",10,0.25,0.4,0.3076923076923077
|
| 100 |
+
existing_specialized_separate_lineage,42,Appeal_to_Authority,11,0.3225806451612903,0.9090909090909091,0.4761904761904761
|
| 101 |
+
existing_specialized_separate_lineage,42,Appeal_to_fear-prejudice,24,0.5405405405405406,0.8333333333333334,0.6557377049180328
|
| 102 |
+
existing_specialized_separate_lineage,42,"Bandwagon,Reductio_ad_hitlerum",7,0.42857142857142855,0.8571428571428571,0.5714285714285714
|
| 103 |
+
existing_specialized_separate_lineage,42,Black-and-White_Fallacy,12,0.47058823529411764,0.6666666666666666,0.5517241379310345
|
| 104 |
+
existing_specialized_separate_lineage,42,Causal_Oversimplification,18,0.4444444444444444,0.8888888888888888,0.5925925925925926
|
| 105 |
+
existing_specialized_separate_lineage,42,Doubt,22,0.5116279069767442,1.0,0.676923076923077
|
| 106 |
+
existing_specialized_separate_lineage,42,"Exaggeration,Minimisation",22,0.4375,0.9545454545454546,0.6
|
| 107 |
+
existing_specialized_separate_lineage,42,Flag-Waving,17,0.6,0.8823529411764706,0.7142857142857143
|
| 108 |
+
existing_specialized_separate_lineage,42,Loaded_Language,45,0.8181818181818182,1.0,0.9
|
| 109 |
+
existing_specialized_separate_lineage,42,"Name_Calling,Labeling",33,0.6808510638297872,0.9696969696969697,0.7999999999999999
|
| 110 |
+
existing_specialized_separate_lineage,42,Repetition,23,0.4888888888888889,0.9565217391304348,0.6470588235294117
|
| 111 |
+
existing_specialized_separate_lineage,42,Slogans,14,0.39285714285714285,0.7857142857142857,0.5238095238095237
|
| 112 |
+
existing_specialized_separate_lineage,42,Thought-terminating_Cliches,7,0.2,0.8571428571428571,0.32432432432432434
|
| 113 |
+
existing_specialized_separate_lineage,42,"Whataboutism,Straw_Men,Red_Herring",10,0.25,0.4,0.3076923076923077
|
| 114 |
+
existing_specialized_separate_lineage,43,Appeal_to_Authority,11,0.3125,0.45454545454545453,0.3703703703703703
|
| 115 |
+
existing_specialized_separate_lineage,43,Appeal_to_fear-prejudice,24,0.5365853658536586,0.9166666666666666,0.676923076923077
|
| 116 |
+
existing_specialized_separate_lineage,43,"Bandwagon,Reductio_ad_hitlerum",7,0.46153846153846156,0.8571428571428571,0.6
|
| 117 |
+
existing_specialized_separate_lineage,43,Black-and-White_Fallacy,12,0.46153846153846156,0.5,0.48000000000000004
|
| 118 |
+
existing_specialized_separate_lineage,43,Causal_Oversimplification,18,0.4358974358974359,0.9444444444444444,0.5964912280701755
|
| 119 |
+
existing_specialized_separate_lineage,43,Doubt,22,0.4782608695652174,1.0,0.6470588235294118
|
| 120 |
+
existing_specialized_separate_lineage,43,"Exaggeration,Minimisation",22,0.4583333333333333,1.0,0.6285714285714286
|
| 121 |
+
existing_specialized_separate_lineage,43,Flag-Waving,17,0.5416666666666666,0.7647058823529411,0.6341463414634146
|
| 122 |
+
existing_specialized_separate_lineage,43,Loaded_Language,45,0.8490566037735849,1.0,0.9183673469387755
|
| 123 |
+
existing_specialized_separate_lineage,43,"Name_Calling,Labeling",33,0.6888888888888889,0.9393939393939394,0.7948717948717948
|
| 124 |
+
existing_specialized_separate_lineage,43,Repetition,23,0.47619047619047616,0.8695652173913043,0.6153846153846153
|
| 125 |
+
existing_specialized_separate_lineage,43,Slogans,14,0.4074074074074074,0.7857142857142857,0.5365853658536585
|
| 126 |
+
existing_specialized_separate_lineage,43,Thought-terminating_Cliches,7,0.21052631578947367,0.5714285714285714,0.3076923076923077
|
| 127 |
+
existing_specialized_separate_lineage,43,"Whataboutism,Straw_Men,Red_Herring",10,0.3333333333333333,0.4,0.3636363636363636
|
benchmark/per_label_summary.csv
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
arm,label,support,precision_mean,precision_sample_sd,recall_mean,recall_sample_sd,f1_mean,f1_sample_sd
|
| 2 |
+
task_agnostic_base,Appeal_to_Authority,11,0.3958614484930274,0.1238064457010119,0.5757575757575757,0.05248638810814781,0.4657088122605364,0.1020990126979069
|
| 3 |
+
task_agnostic_base,Appeal_to_fear-prejudice,24,0.5540244550770866,0.02525673990402998,0.7361111111111112,0.13393959390267993,0.6277521761392729,0.04994879016183189
|
| 4 |
+
task_agnostic_base,"Bandwagon,Reductio_ad_hitlerum",7,0.4851851851851852,0.07882887649552875,0.8095238095238095,0.08247860988423221,0.6006778309409889,0.04793776258918466
|
| 5 |
+
task_agnostic_base,Black-and-White_Fallacy,12,0.3667929292929293,0.07670649392711647,0.47222222222222227,0.09622504486493764,0.4053892966936445,0.04211369218048746
|
| 6 |
+
task_agnostic_base,Causal_Oversimplification,18,0.5428824049513704,0.039216350192122156,0.7962962962962963,0.11564811108145183,0.6446992723588468,0.0634561308339175
|
| 7 |
+
task_agnostic_base,Doubt,22,0.5223832528180354,0.06742780050556177,0.9848484848484849,0.026243194054073875,0.6802057822343391,0.04928561754686331
|
| 8 |
+
task_agnostic_base,"Exaggeration,Minimisation",22,0.4620811287477954,0.016801198309751,0.7727272727272727,0.15745916432444337,0.5727040816326531,0.03645259990419198
|
| 9 |
+
task_agnostic_base,Flag-Waving,17,0.5642642642642642,0.1478145677656334,0.8235294117647058,0.1764705882352941,0.6464506172839506,0.035742219952758225
|
| 10 |
+
task_agnostic_base,Loaded_Language,45,0.8649032907793016,0.046984312428427724,0.9851851851851852,0.012830005981991702,0.9208741623431562,0.03200926878737141
|
| 11 |
+
task_agnostic_base,"Name_Calling,Labeling",33,0.7380952380952381,0.0047619047619048005,0.9090909090909091,0.10925912955951485,0.8125087983911513,0.04252971464057227
|
| 12 |
+
task_agnostic_base,Repetition,23,0.5196620583717357,0.021645909050194662,0.7536231884057971,0.06641414050660639,0.6147520781519571,0.03505927109416221
|
| 13 |
+
task_agnostic_base,Slogans,14,0.5161361626878869,0.20305677930281646,0.7380952380952381,0.10910894511799614,0.5834824090638044,0.09862686289220693
|
| 14 |
+
task_agnostic_base,Thought-terminating_Cliches,7,0.3492063492063492,0.143516994603374,0.3333333333333333,0.08247860988423225,0.319014319014319,0.04017590711117064
|
| 15 |
+
task_agnostic_base,"Whataboutism,Straw_Men,Red_Herring",10,0.2761904761904762,0.05302632726504782,0.5,0.2645751311064591,0.330448343079922,0.021845164720973014
|
| 16 |
+
structural_copy_control,Appeal_to_Authority,11,0.26003086419753085,0.06281357095350094,0.5757575757575758,0.262431940540739,0.35477582846003897,0.11479479036518875
|
| 17 |
+
structural_copy_control,Appeal_to_fear-prejudice,24,0.5104813315339631,0.038427508996734555,0.8611111111111112,0.024056261216234387,0.6401745363877202,0.029305467347150463
|
| 18 |
+
structural_copy_control,"Bandwagon,Reductio_ad_hitlerum",7,0.2876623376623376,0.061386823077618426,0.8571428571428571,0.14285714285714285,0.4306391777656145,0.08686392624022733
|
| 19 |
+
structural_copy_control,Black-and-White_Fallacy,12,0.31339031339031337,0.08301311842925053,0.5277777777777778,0.2678791878053599,0.38778427550357375,0.13069755090515112
|
| 20 |
+
structural_copy_control,Causal_Oversimplification,18,0.4464201043148412,0.013538446216471536,0.9259259259259259,0.03207501495497923,0.6023809523809524,0.018643640071557464
|
| 21 |
+
structural_copy_control,Doubt,22,0.4455314009661836,0.04067072694857813,1.0,0.0,0.615686274509804,0.039427200694482675
|
| 22 |
+
structural_copy_control,"Exaggeration,Minimisation",22,0.4364176150420584,0.025093958106097894,0.8787878787878788,0.05248638810814775,0.5829803047194351,0.030899288356345502
|
| 23 |
+
structural_copy_control,Flag-Waving,17,0.3814814814814815,0.05010277503136551,0.803921568627451,0.17970885078258195,0.5138595287119652,0.06400931989326974
|
| 24 |
+
structural_copy_control,Loaded_Language,45,0.8492581841638446,0.016026591848331686,1.0,0.0,0.9184311025253588,0.009372233868132662
|
| 25 |
+
structural_copy_control,"Name_Calling,Labeling",33,0.6375,0.03410563654946854,0.9696969696969697,0.030303030303030276,0.7685185185185185,0.02023900779105559
|
| 26 |
+
structural_copy_control,Repetition,23,0.44863933452168747,0.01520014564765896,0.9420289855072463,0.06641414050660639,0.607248835423033,0.019858627551810427
|
| 27 |
+
structural_copy_control,Slogans,14,0.3274318274318274,0.05434598572773684,0.4523809523809524,0.1486904285332952,0.3715828924162258,0.07106660236343577
|
| 28 |
+
structural_copy_control,Thought-terminating_Cliches,7,0.1142156862745098,0.012848374923608973,0.38095238095238093,0.1649572197684645,0.17331433998100665,0.029065880233243847
|
| 29 |
+
structural_copy_control,"Whataboutism,Straw_Men,Red_Herring",10,0.19344598054275472,0.03373872471827226,0.5666666666666667,0.2081665999466133,0.28584083690299267,0.06078854830133676
|
| 30 |
+
existing_specialized_separate_lineage,Appeal_to_Authority,11,0.37836021505376344,0.10546365636105155,0.6666666666666666,0.2287828616748712,0.46885361552028215,0.09502747629890008
|
| 31 |
+
existing_specialized_separate_lineage,Appeal_to_fear-prejudice,24,0.525708635464733,0.02235198659615819,0.8888888888888888,0.048112522432468774,0.6599065351235072,0.015362381542320084
|
| 32 |
+
existing_specialized_separate_lineage,"Bandwagon,Reductio_ad_hitlerum",7,0.44485144485144484,0.016487284575115283,0.7619047619047619,0.1649572197684645,0.5571428571428572,0.051507875363771265
|
| 33 |
+
existing_specialized_separate_lineage,Black-and-White_Fallacy,12,0.4662644544997486,0.004538274146363688,0.5833333333333334,0.08333333333333331,0.516747552149851,0.03589484965697345
|
| 34 |
+
existing_specialized_separate_lineage,Causal_Oversimplification,18,0.4472934472934473,0.013055771210700413,0.8333333333333333,0.1469861839480328,0.5781794553724379,0.028407563015813122
|
| 35 |
+
existing_specialized_separate_lineage,Doubt,22,0.5045661001171619,0.023581174969843577,1.0,0.0,0.6704939668174963,0.020973132363659372
|
| 36 |
+
existing_specialized_separate_lineage,"Exaggeration,Minimisation",22,0.44589793281653745,0.010987847993006481,0.9393939393939394,0.06943296507508849,0.6043956043956044,0.022305256187015846
|
| 37 |
+
existing_specialized_separate_lineage,Flag-Waving,17,0.5576388888888889,0.03705367103320583,0.8823529411764706,0.11764705882352944,0.680769868923179,0.04164658769406446
|
| 38 |
+
existing_specialized_separate_lineage,Loaded_Language,45,0.8387650085763294,0.01782556577257849,1.0,0.0,0.9122448979591837,0.01060439269940129
|
| 39 |
+
existing_specialized_separate_lineage,"Name_Calling,Labeling",33,0.6957104190221673,0.019201490809269257,0.9696969696969697,0.030303030303030276,0.8101049442821594,0.022092733328802537
|
| 40 |
+
existing_specialized_separate_lineage,Repetition,23,0.4781557067271353,0.009897987322673213,0.9420289855072463,0.06641414050660639,0.6337774426009719,0.016444118333485254
|
| 41 |
+
existing_specialized_separate_lineage,Slogans,14,0.4334215167548501,0.05811581861282867,0.6904761904761905,0.1649572197684645,0.5201316298877274,0.018567913895656114
|
| 42 |
+
existing_specialized_separate_lineage,Thought-terminating_Cliches,7,0.18446115288220552,0.03641251252864963,0.6666666666666666,0.1649572197684645,0.2868626868626869,0.0511620864429468
|
| 43 |
+
existing_specialized_separate_lineage,"Whataboutism,Straw_Men,Red_Herring",10,0.2777777777777778,0.0481125224324688,0.4,0.0,0.32634032634032634,0.03229931575886014
|
benchmark/reports/existing_specialized_separate_lineage-seed41.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/reports/existing_specialized_separate_lineage-seed42.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/reports/existing_specialized_separate_lineage-seed43.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/reports/structural_copy_control-seed41.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/reports/structural_copy_control-seed42.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/reports/structural_copy_control-seed43.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/reports/task_agnostic_base-seed41.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/reports/task_agnostic_base-seed42.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/reports/task_agnostic_base-seed43.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/resource_metrics.csv
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
arm,seed,parameters,trainable_parameters,peak_cuda_allocated_bytes,peak_cuda_reserved_bytes,total_wall_seconds,test_wall_seconds,test_seconds_per_article,device,dtype
|
| 2 |
+
task_agnostic_base,41,334111040,334111040,3380975616,3519021056,1583.0297977919981,4.490653285000008,0.08164824154545468,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
| 3 |
+
task_agnostic_base,42,334111040,334111040,3380975616,3519021056,1588.617994428001,4.329529417009326,0.07871871667289683,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
| 4 |
+
task_agnostic_base,43,334111040,334111040,3380975616,3510632448,1603.4310500120046,4.595363147003809,0.08355205721825107,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
| 5 |
+
structural_copy_control,41,334111040,334111040,3380975616,3519021056,1575.5905925639963,4.332684432010865,0.07877608058201573,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
| 6 |
+
structural_copy_control,42,334111040,334111040,3380975616,3519021056,1623.5639530190092,4.569082793022972,0.08307423260041767,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
| 7 |
+
structural_copy_control,43,334111040,334111040,3380975616,3510632448,1629.195114912989,4.4899811029899865,0.08163602005436339,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
| 8 |
+
existing_specialized_separate_lineage,41,210903360,210903360,2143655936,2281701376,1634.488557530014,4.6637764379847795,0.0847959352360869,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
| 9 |
+
existing_specialized_separate_lineage,42,210903360,210903360,2143655936,2281701376,1652.9397718419787,4.728947311989032,0.08598086021798239,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
| 10 |
+
existing_specialized_separate_lineage,43,210903360,210903360,2143655936,2273312768,1643.488202479988,4.704143202980049,0.08552987641781908,NVIDIA GeForce RTX 5070 Ti,bfloat16
|
benchmark/seed_metrics.csv
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
arm,seed,threshold,macro_f1,micro_f1,exact_match,predicted_positive_rate,tp,fp,fn,test_articles,test_windows
|
| 2 |
+
task_agnostic_base,41,0.15,0.5883093293517536,0.6468085106382978,0.05454545454545454,0.5714285714285714,228,212,37,55,434
|
| 3 |
+
task_agnostic_base,42,0.3,0.6028379036218096,0.6655172413793103,0.03636363636363636,0.4090909090909091,193,122,72,55,434
|
| 4 |
+
task_agnostic_base,43,0.1,0.5712816197954101,0.6465256797583082,0.0,0.5155844155844156,214,183,51,55,434
|
| 5 |
+
structural_copy_control,41,0.05,0.5169424579262757,0.5773955773955775,0.0,0.712987012987013,235,314,30,55,434
|
| 6 |
+
structural_copy_control,42,0.2,0.5349405779540175,0.6083445491251682,0.0,0.6207792207792208,226,252,39,55,434
|
| 7 |
+
structural_copy_control,43,0.2,0.5023778364496152,0.585635359116022,0.0,0.5961038961038961,212,247,53,55,434
|
| 8 |
+
existing_specialized_separate_lineage,41,0.1,0.5832848452288824,0.6494252873563218,0.0,0.5597402597402598,226,205,39,55,434
|
| 9 |
+
existing_specialized_separate_lineage,42,0.05,0.5958405181160762,0.6458616010854816,0.0,0.612987012987013,238,234,27,55,434
|
| 10 |
+
existing_specialized_separate_lineage,43,0.05,0.5835785045218137,0.6486486486486486,0.01818181818181818,0.5688311688311688,228,210,37,55,434
|
benchmark/semeval-transfer-summary.json
CHANGED
|
@@ -1,47 +1,891 @@
|
|
| 1 |
{
|
| 2 |
-
"schema": "standalone4l-transfer-
|
| 3 |
-
"
|
| 4 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
"seeds": [
|
| 6 |
41,
|
| 7 |
42,
|
| 8 |
43
|
| 9 |
],
|
| 10 |
-
"
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
"mean": 0.5874762842563245,
|
| 17 |
-
"sample_sd": 0.01579462678670608
|
| 18 |
},
|
| 19 |
-
"
|
| 20 |
-
"
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
],
|
| 25 |
-
"
|
| 26 |
-
"sample_sd": 0.016311512588628545
|
| 27 |
},
|
| 28 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
0.07136687142547793,
|
| 30 |
0.06789732566779205,
|
| 31 |
0.06890378334579494
|
| 32 |
],
|
| 33 |
-
"
|
| 34 |
"publication_gate_passed": true,
|
| 35 |
-
"
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"schema": "standalone4l-complete-transfer-benchmark-v2",
|
| 3 |
+
"status": "complete_from_frozen_reports",
|
| 4 |
+
"claim_scope": "Exploratory fresh-head transfer evidence on one English task and a previously opened 55-article SemEval-derived test split.",
|
| 5 |
+
"headless_root_requires_adaptation": true,
|
| 6 |
+
"test_previously_opened": true,
|
| 7 |
+
"test_articles": 55,
|
| 8 |
+
"test_windows": 434,
|
| 9 |
"seeds": [
|
| 10 |
41,
|
| 11 |
42,
|
| 12 |
43
|
| 13 |
],
|
| 14 |
+
"three_seed_sd_is_not_confidence_interval": true,
|
| 15 |
+
"multilingual_downstream_evaluation_performed": false,
|
| 16 |
+
"upstream_multilingual_metadata": {
|
| 17 |
+
"source": "Qwen/Qwen3.5-0.8B official model card",
|
| 18 |
+
"languages_and_dialects_claimed_upstream": 201,
|
| 19 |
+
"scope": "inherited tokenizer and architecture coverage only"
|
|
|
|
|
|
|
| 20 |
},
|
| 21 |
+
"protocol": {
|
| 22 |
+
"epochs": 5,
|
| 23 |
+
"primary_epoch": 5,
|
| 24 |
+
"early_stopping": false,
|
| 25 |
+
"max_length": 256,
|
| 26 |
+
"stride": 128,
|
| 27 |
+
"window_target": "span-aligned technique labels",
|
| 28 |
+
"article_aggregation": "labelwise maximum probability",
|
| 29 |
+
"micro_batch_size": 1,
|
| 30 |
+
"gradient_accumulation": 32,
|
| 31 |
+
"effective_batch_size": 32,
|
| 32 |
+
"learning_rate": 2e-05,
|
| 33 |
+
"weight_decay": 0.01,
|
| 34 |
+
"gradient_clip": 1.0,
|
| 35 |
+
"optimizer": "AdamW",
|
| 36 |
+
"dtype": "bfloat16",
|
| 37 |
+
"thresholds": [
|
| 38 |
+
0.05,
|
| 39 |
+
0.1,
|
| 40 |
+
0.15,
|
| 41 |
+
0.2,
|
| 42 |
+
0.25,
|
| 43 |
+
0.3,
|
| 44 |
+
0.35,
|
| 45 |
+
0.4,
|
| 46 |
+
0.45,
|
| 47 |
+
0.5,
|
| 48 |
+
0.55,
|
| 49 |
+
0.6,
|
| 50 |
+
0.65,
|
| 51 |
+
0.7,
|
| 52 |
+
0.75,
|
| 53 |
+
0.8,
|
| 54 |
+
0.85,
|
| 55 |
+
0.9
|
| 56 |
],
|
| 57 |
+
"threshold_selection": "epoch-5 calibration macro-F1, then micro-F1"
|
|
|
|
| 58 |
},
|
| 59 |
+
"arms": {
|
| 60 |
+
"task_agnostic_base": {
|
| 61 |
+
"display": "Task-agnostic KD base",
|
| 62 |
+
"lineage": "root base plus a fresh 14-label transfer head",
|
| 63 |
+
"metrics": {
|
| 64 |
+
"macro_f1": {
|
| 65 |
+
"values": [
|
| 66 |
+
0.5883093293517536,
|
| 67 |
+
0.6028379036218096,
|
| 68 |
+
0.5712816197954101
|
| 69 |
+
],
|
| 70 |
+
"mean": 0.5874762842563245,
|
| 71 |
+
"sample_sd": 0.01579462678670608,
|
| 72 |
+
"n": 3
|
| 73 |
+
},
|
| 74 |
+
"micro_f1": {
|
| 75 |
+
"values": [
|
| 76 |
+
0.6468085106382978,
|
| 77 |
+
0.6655172413793103,
|
| 78 |
+
0.6465256797583082
|
| 79 |
+
],
|
| 80 |
+
"mean": 0.6529504772586388,
|
| 81 |
+
"sample_sd": 0.010884055708931178,
|
| 82 |
+
"n": 3
|
| 83 |
+
},
|
| 84 |
+
"exact_match": {
|
| 85 |
+
"values": [
|
| 86 |
+
0.05454545454545454,
|
| 87 |
+
0.03636363636363636,
|
| 88 |
+
0.0
|
| 89 |
+
],
|
| 90 |
+
"mean": 0.0303030303030303,
|
| 91 |
+
"sample_sd": 0.027773186030035393,
|
| 92 |
+
"n": 3
|
| 93 |
+
},
|
| 94 |
+
"predicted_positive_rate": {
|
| 95 |
+
"values": [
|
| 96 |
+
0.5714285714285714,
|
| 97 |
+
0.4090909090909091,
|
| 98 |
+
0.5155844155844156
|
| 99 |
+
],
|
| 100 |
+
"mean": 0.4987012987012987,
|
| 101 |
+
"sample_sd": 0.08247520160679181,
|
| 102 |
+
"n": 3
|
| 103 |
+
}
|
| 104 |
+
},
|
| 105 |
+
"resources": {
|
| 106 |
+
"parameters": {
|
| 107 |
+
"values": [
|
| 108 |
+
334111040.0,
|
| 109 |
+
334111040.0,
|
| 110 |
+
334111040.0
|
| 111 |
+
],
|
| 112 |
+
"mean": 334111040.0,
|
| 113 |
+
"sample_sd": 0.0,
|
| 114 |
+
"n": 3
|
| 115 |
+
},
|
| 116 |
+
"peak_cuda_allocated_bytes": {
|
| 117 |
+
"values": [
|
| 118 |
+
3380975616.0,
|
| 119 |
+
3380975616.0,
|
| 120 |
+
3380975616.0
|
| 121 |
+
],
|
| 122 |
+
"mean": 3380975616.0,
|
| 123 |
+
"sample_sd": 0.0,
|
| 124 |
+
"n": 3
|
| 125 |
+
},
|
| 126 |
+
"peak_cuda_reserved_bytes": {
|
| 127 |
+
"values": [
|
| 128 |
+
3519021056.0,
|
| 129 |
+
3519021056.0,
|
| 130 |
+
3510632448.0
|
| 131 |
+
],
|
| 132 |
+
"mean": 3516224853.3333335,
|
| 133 |
+
"sample_sd": 4843165.086926248,
|
| 134 |
+
"n": 3
|
| 135 |
+
},
|
| 136 |
+
"total_wall_seconds": {
|
| 137 |
+
"values": [
|
| 138 |
+
1583.0297977919981,
|
| 139 |
+
1588.617994428001,
|
| 140 |
+
1603.4310500120046
|
| 141 |
+
],
|
| 142 |
+
"mean": 1591.692947410668,
|
| 143 |
+
"sample_sd": 10.54249851412311,
|
| 144 |
+
"n": 3
|
| 145 |
+
},
|
| 146 |
+
"test_seconds_per_article": {
|
| 147 |
+
"values": [
|
| 148 |
+
0.08164824154545468,
|
| 149 |
+
0.07871871667289683,
|
| 150 |
+
0.08355205721825107
|
| 151 |
+
],
|
| 152 |
+
"mean": 0.08130633847886752,
|
| 153 |
+
"sample_sd": 0.0024347419754560395,
|
| 154 |
+
"n": 3
|
| 155 |
+
}
|
| 156 |
+
},
|
| 157 |
+
"thresholds": [
|
| 158 |
+
0.15,
|
| 159 |
+
0.3,
|
| 160 |
+
0.1
|
| 161 |
+
]
|
| 162 |
+
},
|
| 163 |
+
"structural_copy_control": {
|
| 164 |
+
"display": "Structural copy, no task-free KD",
|
| 165 |
+
"lineage": "same four-layer structure without task-free KD",
|
| 166 |
+
"metrics": {
|
| 167 |
+
"macro_f1": {
|
| 168 |
+
"values": [
|
| 169 |
+
0.5169424579262757,
|
| 170 |
+
0.5349405779540175,
|
| 171 |
+
0.5023778364496152
|
| 172 |
+
],
|
| 173 |
+
"mean": 0.5180869574433028,
|
| 174 |
+
"sample_sd": 0.016311512588628545,
|
| 175 |
+
"n": 3
|
| 176 |
+
},
|
| 177 |
+
"micro_f1": {
|
| 178 |
+
"values": [
|
| 179 |
+
0.5773955773955775,
|
| 180 |
+
0.6083445491251682,
|
| 181 |
+
0.585635359116022
|
| 182 |
+
],
|
| 183 |
+
"mean": 0.5904584952122559,
|
| 184 |
+
"sample_sd": 0.01602830914762526,
|
| 185 |
+
"n": 3
|
| 186 |
+
},
|
| 187 |
+
"exact_match": {
|
| 188 |
+
"values": [
|
| 189 |
+
0.0,
|
| 190 |
+
0.0,
|
| 191 |
+
0.0
|
| 192 |
+
],
|
| 193 |
+
"mean": 0.0,
|
| 194 |
+
"sample_sd": 0.0,
|
| 195 |
+
"n": 3
|
| 196 |
+
},
|
| 197 |
+
"predicted_positive_rate": {
|
| 198 |
+
"values": [
|
| 199 |
+
0.712987012987013,
|
| 200 |
+
0.6207792207792208,
|
| 201 |
+
0.5961038961038961
|
| 202 |
+
],
|
| 203 |
+
"mean": 0.6432900432900432,
|
| 204 |
+
"sample_sd": 0.06160737456395461,
|
| 205 |
+
"n": 3
|
| 206 |
+
}
|
| 207 |
+
},
|
| 208 |
+
"resources": {
|
| 209 |
+
"parameters": {
|
| 210 |
+
"values": [
|
| 211 |
+
334111040.0,
|
| 212 |
+
334111040.0,
|
| 213 |
+
334111040.0
|
| 214 |
+
],
|
| 215 |
+
"mean": 334111040.0,
|
| 216 |
+
"sample_sd": 0.0,
|
| 217 |
+
"n": 3
|
| 218 |
+
},
|
| 219 |
+
"peak_cuda_allocated_bytes": {
|
| 220 |
+
"values": [
|
| 221 |
+
3380975616.0,
|
| 222 |
+
3380975616.0,
|
| 223 |
+
3380975616.0
|
| 224 |
+
],
|
| 225 |
+
"mean": 3380975616.0,
|
| 226 |
+
"sample_sd": 0.0,
|
| 227 |
+
"n": 3
|
| 228 |
+
},
|
| 229 |
+
"peak_cuda_reserved_bytes": {
|
| 230 |
+
"values": [
|
| 231 |
+
3519021056.0,
|
| 232 |
+
3519021056.0,
|
| 233 |
+
3510632448.0
|
| 234 |
+
],
|
| 235 |
+
"mean": 3516224853.3333335,
|
| 236 |
+
"sample_sd": 4843165.086926248,
|
| 237 |
+
"n": 3
|
| 238 |
+
},
|
| 239 |
+
"total_wall_seconds": {
|
| 240 |
+
"values": [
|
| 241 |
+
1575.5905925639963,
|
| 242 |
+
1623.5639530190092,
|
| 243 |
+
1629.195114912989
|
| 244 |
+
],
|
| 245 |
+
"mean": 1609.4498868319981,
|
| 246 |
+
"sample_sd": 29.457874198835384,
|
| 247 |
+
"n": 3
|
| 248 |
+
},
|
| 249 |
+
"test_seconds_per_article": {
|
| 250 |
+
"values": [
|
| 251 |
+
0.07877608058201573,
|
| 252 |
+
0.08307423260041767,
|
| 253 |
+
0.08163602005436339
|
| 254 |
+
],
|
| 255 |
+
"mean": 0.08116211107893226,
|
| 256 |
+
"sample_sd": 0.002187914527825257,
|
| 257 |
+
"n": 3
|
| 258 |
+
}
|
| 259 |
+
},
|
| 260 |
+
"thresholds": [
|
| 261 |
+
0.05,
|
| 262 |
+
0.2,
|
| 263 |
+
0.2
|
| 264 |
+
]
|
| 265 |
+
},
|
| 266 |
+
"existing_specialized_separate_lineage": {
|
| 267 |
+
"display": "Existing specialized 4L, separate lineage",
|
| 268 |
+
"lineage": "historical task-specific shrink/distillation lineage",
|
| 269 |
+
"metrics": {
|
| 270 |
+
"macro_f1": {
|
| 271 |
+
"values": [
|
| 272 |
+
0.5832848452288824,
|
| 273 |
+
0.5958405181160762,
|
| 274 |
+
0.5835785045218137
|
| 275 |
+
],
|
| 276 |
+
"mean": 0.5875679559555907,
|
| 277 |
+
"sample_sd": 0.007165753447433348,
|
| 278 |
+
"n": 3
|
| 279 |
+
},
|
| 280 |
+
"micro_f1": {
|
| 281 |
+
"values": [
|
| 282 |
+
0.6494252873563218,
|
| 283 |
+
0.6458616010854816,
|
| 284 |
+
0.6486486486486486
|
| 285 |
+
],
|
| 286 |
+
"mean": 0.647978512363484,
|
| 287 |
+
"sample_sd": 0.0018739735696546695,
|
| 288 |
+
"n": 3
|
| 289 |
+
},
|
| 290 |
+
"exact_match": {
|
| 291 |
+
"values": [
|
| 292 |
+
0.0,
|
| 293 |
+
0.0,
|
| 294 |
+
0.01818181818181818
|
| 295 |
+
],
|
| 296 |
+
"mean": 0.006060606060606061,
|
| 297 |
+
"sample_sd": 0.01049727762162956,
|
| 298 |
+
"n": 3
|
| 299 |
+
},
|
| 300 |
+
"predicted_positive_rate": {
|
| 301 |
+
"values": [
|
| 302 |
+
0.5597402597402598,
|
| 303 |
+
0.612987012987013,
|
| 304 |
+
0.5688311688311688
|
| 305 |
+
],
|
| 306 |
+
"mean": 0.5805194805194805,
|
| 307 |
+
"sample_sd": 0.02848274311618351,
|
| 308 |
+
"n": 3
|
| 309 |
+
}
|
| 310 |
+
},
|
| 311 |
+
"resources": {
|
| 312 |
+
"parameters": {
|
| 313 |
+
"values": [
|
| 314 |
+
210903360.0,
|
| 315 |
+
210903360.0,
|
| 316 |
+
210903360.0
|
| 317 |
+
],
|
| 318 |
+
"mean": 210903360.0,
|
| 319 |
+
"sample_sd": 0.0,
|
| 320 |
+
"n": 3
|
| 321 |
+
},
|
| 322 |
+
"peak_cuda_allocated_bytes": {
|
| 323 |
+
"values": [
|
| 324 |
+
2143655936.0,
|
| 325 |
+
2143655936.0,
|
| 326 |
+
2143655936.0
|
| 327 |
+
],
|
| 328 |
+
"mean": 2143655936.0,
|
| 329 |
+
"sample_sd": 0.0,
|
| 330 |
+
"n": 3
|
| 331 |
+
},
|
| 332 |
+
"peak_cuda_reserved_bytes": {
|
| 333 |
+
"values": [
|
| 334 |
+
2281701376.0,
|
| 335 |
+
2281701376.0,
|
| 336 |
+
2273312768.0
|
| 337 |
+
],
|
| 338 |
+
"mean": 2278905173.3333335,
|
| 339 |
+
"sample_sd": 4843165.086926248,
|
| 340 |
+
"n": 3
|
| 341 |
+
},
|
| 342 |
+
"total_wall_seconds": {
|
| 343 |
+
"values": [
|
| 344 |
+
1634.488557530014,
|
| 345 |
+
1652.9397718419787,
|
| 346 |
+
1643.488202479988
|
| 347 |
+
],
|
| 348 |
+
"mean": 1643.6388439506602,
|
| 349 |
+
"sample_sd": 9.226529522850198,
|
| 350 |
+
"n": 3
|
| 351 |
+
},
|
| 352 |
+
"test_seconds_per_article": {
|
| 353 |
+
"values": [
|
| 354 |
+
0.0847959352360869,
|
| 355 |
+
0.08598086021798239,
|
| 356 |
+
0.08552987641781908
|
| 357 |
+
],
|
| 358 |
+
"mean": 0.08543555729062946,
|
| 359 |
+
"sample_sd": 0.0005980667826383295,
|
| 360 |
+
"n": 3
|
| 361 |
+
}
|
| 362 |
+
},
|
| 363 |
+
"thresholds": [
|
| 364 |
+
0.1,
|
| 365 |
+
0.05,
|
| 366 |
+
0.05
|
| 367 |
+
]
|
| 368 |
+
}
|
| 369 |
+
},
|
| 370 |
+
"same_seed_macro_f1_deltas_base_minus_control": [
|
| 371 |
0.07136687142547793,
|
| 372 |
0.06789732566779205,
|
| 373 |
0.06890378334579494
|
| 374 |
],
|
| 375 |
+
"mean_macro_f1_delta_base_minus_control": 0.06938932681302164,
|
| 376 |
"publication_gate_passed": true,
|
| 377 |
+
"per_label_summary": [
|
| 378 |
+
{
|
| 379 |
+
"arm": "task_agnostic_base",
|
| 380 |
+
"label": "Appeal_to_Authority",
|
| 381 |
+
"support": 11,
|
| 382 |
+
"precision_mean": 0.3958614484930274,
|
| 383 |
+
"precision_sample_sd": 0.1238064457010119,
|
| 384 |
+
"recall_mean": 0.5757575757575757,
|
| 385 |
+
"recall_sample_sd": 0.05248638810814781,
|
| 386 |
+
"f1_mean": 0.4657088122605364,
|
| 387 |
+
"f1_sample_sd": 0.1020990126979069
|
| 388 |
+
},
|
| 389 |
+
{
|
| 390 |
+
"arm": "task_agnostic_base",
|
| 391 |
+
"label": "Appeal_to_fear-prejudice",
|
| 392 |
+
"support": 24,
|
| 393 |
+
"precision_mean": 0.5540244550770866,
|
| 394 |
+
"precision_sample_sd": 0.02525673990402998,
|
| 395 |
+
"recall_mean": 0.7361111111111112,
|
| 396 |
+
"recall_sample_sd": 0.13393959390267993,
|
| 397 |
+
"f1_mean": 0.6277521761392729,
|
| 398 |
+
"f1_sample_sd": 0.04994879016183189
|
| 399 |
+
},
|
| 400 |
+
{
|
| 401 |
+
"arm": "task_agnostic_base",
|
| 402 |
+
"label": "Bandwagon,Reductio_ad_hitlerum",
|
| 403 |
+
"support": 7,
|
| 404 |
+
"precision_mean": 0.4851851851851852,
|
| 405 |
+
"precision_sample_sd": 0.07882887649552875,
|
| 406 |
+
"recall_mean": 0.8095238095238095,
|
| 407 |
+
"recall_sample_sd": 0.08247860988423221,
|
| 408 |
+
"f1_mean": 0.6006778309409889,
|
| 409 |
+
"f1_sample_sd": 0.04793776258918466
|
| 410 |
+
},
|
| 411 |
+
{
|
| 412 |
+
"arm": "task_agnostic_base",
|
| 413 |
+
"label": "Black-and-White_Fallacy",
|
| 414 |
+
"support": 12,
|
| 415 |
+
"precision_mean": 0.3667929292929293,
|
| 416 |
+
"precision_sample_sd": 0.07670649392711647,
|
| 417 |
+
"recall_mean": 0.47222222222222227,
|
| 418 |
+
"recall_sample_sd": 0.09622504486493764,
|
| 419 |
+
"f1_mean": 0.4053892966936445,
|
| 420 |
+
"f1_sample_sd": 0.04211369218048746
|
| 421 |
+
},
|
| 422 |
+
{
|
| 423 |
+
"arm": "task_agnostic_base",
|
| 424 |
+
"label": "Causal_Oversimplification",
|
| 425 |
+
"support": 18,
|
| 426 |
+
"precision_mean": 0.5428824049513704,
|
| 427 |
+
"precision_sample_sd": 0.039216350192122156,
|
| 428 |
+
"recall_mean": 0.7962962962962963,
|
| 429 |
+
"recall_sample_sd": 0.11564811108145183,
|
| 430 |
+
"f1_mean": 0.6446992723588468,
|
| 431 |
+
"f1_sample_sd": 0.0634561308339175
|
| 432 |
+
},
|
| 433 |
+
{
|
| 434 |
+
"arm": "task_agnostic_base",
|
| 435 |
+
"label": "Doubt",
|
| 436 |
+
"support": 22,
|
| 437 |
+
"precision_mean": 0.5223832528180354,
|
| 438 |
+
"precision_sample_sd": 0.06742780050556177,
|
| 439 |
+
"recall_mean": 0.9848484848484849,
|
| 440 |
+
"recall_sample_sd": 0.026243194054073875,
|
| 441 |
+
"f1_mean": 0.6802057822343391,
|
| 442 |
+
"f1_sample_sd": 0.04928561754686331
|
| 443 |
+
},
|
| 444 |
+
{
|
| 445 |
+
"arm": "task_agnostic_base",
|
| 446 |
+
"label": "Exaggeration,Minimisation",
|
| 447 |
+
"support": 22,
|
| 448 |
+
"precision_mean": 0.4620811287477954,
|
| 449 |
+
"precision_sample_sd": 0.016801198309751,
|
| 450 |
+
"recall_mean": 0.7727272727272727,
|
| 451 |
+
"recall_sample_sd": 0.15745916432444337,
|
| 452 |
+
"f1_mean": 0.5727040816326531,
|
| 453 |
+
"f1_sample_sd": 0.03645259990419198
|
| 454 |
+
},
|
| 455 |
+
{
|
| 456 |
+
"arm": "task_agnostic_base",
|
| 457 |
+
"label": "Flag-Waving",
|
| 458 |
+
"support": 17,
|
| 459 |
+
"precision_mean": 0.5642642642642642,
|
| 460 |
+
"precision_sample_sd": 0.1478145677656334,
|
| 461 |
+
"recall_mean": 0.8235294117647058,
|
| 462 |
+
"recall_sample_sd": 0.1764705882352941,
|
| 463 |
+
"f1_mean": 0.6464506172839506,
|
| 464 |
+
"f1_sample_sd": 0.035742219952758225
|
| 465 |
+
},
|
| 466 |
+
{
|
| 467 |
+
"arm": "task_agnostic_base",
|
| 468 |
+
"label": "Loaded_Language",
|
| 469 |
+
"support": 45,
|
| 470 |
+
"precision_mean": 0.8649032907793016,
|
| 471 |
+
"precision_sample_sd": 0.046984312428427724,
|
| 472 |
+
"recall_mean": 0.9851851851851852,
|
| 473 |
+
"recall_sample_sd": 0.012830005981991702,
|
| 474 |
+
"f1_mean": 0.9208741623431562,
|
| 475 |
+
"f1_sample_sd": 0.03200926878737141
|
| 476 |
+
},
|
| 477 |
+
{
|
| 478 |
+
"arm": "task_agnostic_base",
|
| 479 |
+
"label": "Name_Calling,Labeling",
|
| 480 |
+
"support": 33,
|
| 481 |
+
"precision_mean": 0.7380952380952381,
|
| 482 |
+
"precision_sample_sd": 0.0047619047619048005,
|
| 483 |
+
"recall_mean": 0.9090909090909091,
|
| 484 |
+
"recall_sample_sd": 0.10925912955951485,
|
| 485 |
+
"f1_mean": 0.8125087983911513,
|
| 486 |
+
"f1_sample_sd": 0.04252971464057227
|
| 487 |
+
},
|
| 488 |
+
{
|
| 489 |
+
"arm": "task_agnostic_base",
|
| 490 |
+
"label": "Repetition",
|
| 491 |
+
"support": 23,
|
| 492 |
+
"precision_mean": 0.5196620583717357,
|
| 493 |
+
"precision_sample_sd": 0.021645909050194662,
|
| 494 |
+
"recall_mean": 0.7536231884057971,
|
| 495 |
+
"recall_sample_sd": 0.06641414050660639,
|
| 496 |
+
"f1_mean": 0.6147520781519571,
|
| 497 |
+
"f1_sample_sd": 0.03505927109416221
|
| 498 |
+
},
|
| 499 |
+
{
|
| 500 |
+
"arm": "task_agnostic_base",
|
| 501 |
+
"label": "Slogans",
|
| 502 |
+
"support": 14,
|
| 503 |
+
"precision_mean": 0.5161361626878869,
|
| 504 |
+
"precision_sample_sd": 0.20305677930281646,
|
| 505 |
+
"recall_mean": 0.7380952380952381,
|
| 506 |
+
"recall_sample_sd": 0.10910894511799614,
|
| 507 |
+
"f1_mean": 0.5834824090638044,
|
| 508 |
+
"f1_sample_sd": 0.09862686289220693
|
| 509 |
+
},
|
| 510 |
+
{
|
| 511 |
+
"arm": "task_agnostic_base",
|
| 512 |
+
"label": "Thought-terminating_Cliches",
|
| 513 |
+
"support": 7,
|
| 514 |
+
"precision_mean": 0.3492063492063492,
|
| 515 |
+
"precision_sample_sd": 0.143516994603374,
|
| 516 |
+
"recall_mean": 0.3333333333333333,
|
| 517 |
+
"recall_sample_sd": 0.08247860988423225,
|
| 518 |
+
"f1_mean": 0.319014319014319,
|
| 519 |
+
"f1_sample_sd": 0.04017590711117064
|
| 520 |
+
},
|
| 521 |
+
{
|
| 522 |
+
"arm": "task_agnostic_base",
|
| 523 |
+
"label": "Whataboutism,Straw_Men,Red_Herring",
|
| 524 |
+
"support": 10,
|
| 525 |
+
"precision_mean": 0.2761904761904762,
|
| 526 |
+
"precision_sample_sd": 0.05302632726504782,
|
| 527 |
+
"recall_mean": 0.5,
|
| 528 |
+
"recall_sample_sd": 0.2645751311064591,
|
| 529 |
+
"f1_mean": 0.330448343079922,
|
| 530 |
+
"f1_sample_sd": 0.021845164720973014
|
| 531 |
+
},
|
| 532 |
+
{
|
| 533 |
+
"arm": "structural_copy_control",
|
| 534 |
+
"label": "Appeal_to_Authority",
|
| 535 |
+
"support": 11,
|
| 536 |
+
"precision_mean": 0.26003086419753085,
|
| 537 |
+
"precision_sample_sd": 0.06281357095350094,
|
| 538 |
+
"recall_mean": 0.5757575757575758,
|
| 539 |
+
"recall_sample_sd": 0.262431940540739,
|
| 540 |
+
"f1_mean": 0.35477582846003897,
|
| 541 |
+
"f1_sample_sd": 0.11479479036518875
|
| 542 |
+
},
|
| 543 |
+
{
|
| 544 |
+
"arm": "structural_copy_control",
|
| 545 |
+
"label": "Appeal_to_fear-prejudice",
|
| 546 |
+
"support": 24,
|
| 547 |
+
"precision_mean": 0.5104813315339631,
|
| 548 |
+
"precision_sample_sd": 0.038427508996734555,
|
| 549 |
+
"recall_mean": 0.8611111111111112,
|
| 550 |
+
"recall_sample_sd": 0.024056261216234387,
|
| 551 |
+
"f1_mean": 0.6401745363877202,
|
| 552 |
+
"f1_sample_sd": 0.029305467347150463
|
| 553 |
+
},
|
| 554 |
+
{
|
| 555 |
+
"arm": "structural_copy_control",
|
| 556 |
+
"label": "Bandwagon,Reductio_ad_hitlerum",
|
| 557 |
+
"support": 7,
|
| 558 |
+
"precision_mean": 0.2876623376623376,
|
| 559 |
+
"precision_sample_sd": 0.061386823077618426,
|
| 560 |
+
"recall_mean": 0.8571428571428571,
|
| 561 |
+
"recall_sample_sd": 0.14285714285714285,
|
| 562 |
+
"f1_mean": 0.4306391777656145,
|
| 563 |
+
"f1_sample_sd": 0.08686392624022733
|
| 564 |
+
},
|
| 565 |
+
{
|
| 566 |
+
"arm": "structural_copy_control",
|
| 567 |
+
"label": "Black-and-White_Fallacy",
|
| 568 |
+
"support": 12,
|
| 569 |
+
"precision_mean": 0.31339031339031337,
|
| 570 |
+
"precision_sample_sd": 0.08301311842925053,
|
| 571 |
+
"recall_mean": 0.5277777777777778,
|
| 572 |
+
"recall_sample_sd": 0.2678791878053599,
|
| 573 |
+
"f1_mean": 0.38778427550357375,
|
| 574 |
+
"f1_sample_sd": 0.13069755090515112
|
| 575 |
+
},
|
| 576 |
+
{
|
| 577 |
+
"arm": "structural_copy_control",
|
| 578 |
+
"label": "Causal_Oversimplification",
|
| 579 |
+
"support": 18,
|
| 580 |
+
"precision_mean": 0.4464201043148412,
|
| 581 |
+
"precision_sample_sd": 0.013538446216471536,
|
| 582 |
+
"recall_mean": 0.9259259259259259,
|
| 583 |
+
"recall_sample_sd": 0.03207501495497923,
|
| 584 |
+
"f1_mean": 0.6023809523809524,
|
| 585 |
+
"f1_sample_sd": 0.018643640071557464
|
| 586 |
+
},
|
| 587 |
+
{
|
| 588 |
+
"arm": "structural_copy_control",
|
| 589 |
+
"label": "Doubt",
|
| 590 |
+
"support": 22,
|
| 591 |
+
"precision_mean": 0.4455314009661836,
|
| 592 |
+
"precision_sample_sd": 0.04067072694857813,
|
| 593 |
+
"recall_mean": 1.0,
|
| 594 |
+
"recall_sample_sd": 0.0,
|
| 595 |
+
"f1_mean": 0.615686274509804,
|
| 596 |
+
"f1_sample_sd": 0.039427200694482675
|
| 597 |
+
},
|
| 598 |
+
{
|
| 599 |
+
"arm": "structural_copy_control",
|
| 600 |
+
"label": "Exaggeration,Minimisation",
|
| 601 |
+
"support": 22,
|
| 602 |
+
"precision_mean": 0.4364176150420584,
|
| 603 |
+
"precision_sample_sd": 0.025093958106097894,
|
| 604 |
+
"recall_mean": 0.8787878787878788,
|
| 605 |
+
"recall_sample_sd": 0.05248638810814775,
|
| 606 |
+
"f1_mean": 0.5829803047194351,
|
| 607 |
+
"f1_sample_sd": 0.030899288356345502
|
| 608 |
+
},
|
| 609 |
+
{
|
| 610 |
+
"arm": "structural_copy_control",
|
| 611 |
+
"label": "Flag-Waving",
|
| 612 |
+
"support": 17,
|
| 613 |
+
"precision_mean": 0.3814814814814815,
|
| 614 |
+
"precision_sample_sd": 0.05010277503136551,
|
| 615 |
+
"recall_mean": 0.803921568627451,
|
| 616 |
+
"recall_sample_sd": 0.17970885078258195,
|
| 617 |
+
"f1_mean": 0.5138595287119652,
|
| 618 |
+
"f1_sample_sd": 0.06400931989326974
|
| 619 |
+
},
|
| 620 |
+
{
|
| 621 |
+
"arm": "structural_copy_control",
|
| 622 |
+
"label": "Loaded_Language",
|
| 623 |
+
"support": 45,
|
| 624 |
+
"precision_mean": 0.8492581841638446,
|
| 625 |
+
"precision_sample_sd": 0.016026591848331686,
|
| 626 |
+
"recall_mean": 1.0,
|
| 627 |
+
"recall_sample_sd": 0.0,
|
| 628 |
+
"f1_mean": 0.9184311025253588,
|
| 629 |
+
"f1_sample_sd": 0.009372233868132662
|
| 630 |
+
},
|
| 631 |
+
{
|
| 632 |
+
"arm": "structural_copy_control",
|
| 633 |
+
"label": "Name_Calling,Labeling",
|
| 634 |
+
"support": 33,
|
| 635 |
+
"precision_mean": 0.6375,
|
| 636 |
+
"precision_sample_sd": 0.03410563654946854,
|
| 637 |
+
"recall_mean": 0.9696969696969697,
|
| 638 |
+
"recall_sample_sd": 0.030303030303030276,
|
| 639 |
+
"f1_mean": 0.7685185185185185,
|
| 640 |
+
"f1_sample_sd": 0.02023900779105559
|
| 641 |
+
},
|
| 642 |
+
{
|
| 643 |
+
"arm": "structural_copy_control",
|
| 644 |
+
"label": "Repetition",
|
| 645 |
+
"support": 23,
|
| 646 |
+
"precision_mean": 0.44863933452168747,
|
| 647 |
+
"precision_sample_sd": 0.01520014564765896,
|
| 648 |
+
"recall_mean": 0.9420289855072463,
|
| 649 |
+
"recall_sample_sd": 0.06641414050660639,
|
| 650 |
+
"f1_mean": 0.607248835423033,
|
| 651 |
+
"f1_sample_sd": 0.019858627551810427
|
| 652 |
+
},
|
| 653 |
+
{
|
| 654 |
+
"arm": "structural_copy_control",
|
| 655 |
+
"label": "Slogans",
|
| 656 |
+
"support": 14,
|
| 657 |
+
"precision_mean": 0.3274318274318274,
|
| 658 |
+
"precision_sample_sd": 0.05434598572773684,
|
| 659 |
+
"recall_mean": 0.4523809523809524,
|
| 660 |
+
"recall_sample_sd": 0.1486904285332952,
|
| 661 |
+
"f1_mean": 0.3715828924162258,
|
| 662 |
+
"f1_sample_sd": 0.07106660236343577
|
| 663 |
+
},
|
| 664 |
+
{
|
| 665 |
+
"arm": "structural_copy_control",
|
| 666 |
+
"label": "Thought-terminating_Cliches",
|
| 667 |
+
"support": 7,
|
| 668 |
+
"precision_mean": 0.1142156862745098,
|
| 669 |
+
"precision_sample_sd": 0.012848374923608973,
|
| 670 |
+
"recall_mean": 0.38095238095238093,
|
| 671 |
+
"recall_sample_sd": 0.1649572197684645,
|
| 672 |
+
"f1_mean": 0.17331433998100665,
|
| 673 |
+
"f1_sample_sd": 0.029065880233243847
|
| 674 |
+
},
|
| 675 |
+
{
|
| 676 |
+
"arm": "structural_copy_control",
|
| 677 |
+
"label": "Whataboutism,Straw_Men,Red_Herring",
|
| 678 |
+
"support": 10,
|
| 679 |
+
"precision_mean": 0.19344598054275472,
|
| 680 |
+
"precision_sample_sd": 0.03373872471827226,
|
| 681 |
+
"recall_mean": 0.5666666666666667,
|
| 682 |
+
"recall_sample_sd": 0.2081665999466133,
|
| 683 |
+
"f1_mean": 0.28584083690299267,
|
| 684 |
+
"f1_sample_sd": 0.06078854830133676
|
| 685 |
+
},
|
| 686 |
+
{
|
| 687 |
+
"arm": "existing_specialized_separate_lineage",
|
| 688 |
+
"label": "Appeal_to_Authority",
|
| 689 |
+
"support": 11,
|
| 690 |
+
"precision_mean": 0.37836021505376344,
|
| 691 |
+
"precision_sample_sd": 0.10546365636105155,
|
| 692 |
+
"recall_mean": 0.6666666666666666,
|
| 693 |
+
"recall_sample_sd": 0.2287828616748712,
|
| 694 |
+
"f1_mean": 0.46885361552028215,
|
| 695 |
+
"f1_sample_sd": 0.09502747629890008
|
| 696 |
+
},
|
| 697 |
+
{
|
| 698 |
+
"arm": "existing_specialized_separate_lineage",
|
| 699 |
+
"label": "Appeal_to_fear-prejudice",
|
| 700 |
+
"support": 24,
|
| 701 |
+
"precision_mean": 0.525708635464733,
|
| 702 |
+
"precision_sample_sd": 0.02235198659615819,
|
| 703 |
+
"recall_mean": 0.8888888888888888,
|
| 704 |
+
"recall_sample_sd": 0.048112522432468774,
|
| 705 |
+
"f1_mean": 0.6599065351235072,
|
| 706 |
+
"f1_sample_sd": 0.015362381542320084
|
| 707 |
+
},
|
| 708 |
+
{
|
| 709 |
+
"arm": "existing_specialized_separate_lineage",
|
| 710 |
+
"label": "Bandwagon,Reductio_ad_hitlerum",
|
| 711 |
+
"support": 7,
|
| 712 |
+
"precision_mean": 0.44485144485144484,
|
| 713 |
+
"precision_sample_sd": 0.016487284575115283,
|
| 714 |
+
"recall_mean": 0.7619047619047619,
|
| 715 |
+
"recall_sample_sd": 0.1649572197684645,
|
| 716 |
+
"f1_mean": 0.5571428571428572,
|
| 717 |
+
"f1_sample_sd": 0.051507875363771265
|
| 718 |
+
},
|
| 719 |
+
{
|
| 720 |
+
"arm": "existing_specialized_separate_lineage",
|
| 721 |
+
"label": "Black-and-White_Fallacy",
|
| 722 |
+
"support": 12,
|
| 723 |
+
"precision_mean": 0.4662644544997486,
|
| 724 |
+
"precision_sample_sd": 0.004538274146363688,
|
| 725 |
+
"recall_mean": 0.5833333333333334,
|
| 726 |
+
"recall_sample_sd": 0.08333333333333331,
|
| 727 |
+
"f1_mean": 0.516747552149851,
|
| 728 |
+
"f1_sample_sd": 0.03589484965697345
|
| 729 |
+
},
|
| 730 |
+
{
|
| 731 |
+
"arm": "existing_specialized_separate_lineage",
|
| 732 |
+
"label": "Causal_Oversimplification",
|
| 733 |
+
"support": 18,
|
| 734 |
+
"precision_mean": 0.4472934472934473,
|
| 735 |
+
"precision_sample_sd": 0.013055771210700413,
|
| 736 |
+
"recall_mean": 0.8333333333333333,
|
| 737 |
+
"recall_sample_sd": 0.1469861839480328,
|
| 738 |
+
"f1_mean": 0.5781794553724379,
|
| 739 |
+
"f1_sample_sd": 0.028407563015813122
|
| 740 |
+
},
|
| 741 |
+
{
|
| 742 |
+
"arm": "existing_specialized_separate_lineage",
|
| 743 |
+
"label": "Doubt",
|
| 744 |
+
"support": 22,
|
| 745 |
+
"precision_mean": 0.5045661001171619,
|
| 746 |
+
"precision_sample_sd": 0.023581174969843577,
|
| 747 |
+
"recall_mean": 1.0,
|
| 748 |
+
"recall_sample_sd": 0.0,
|
| 749 |
+
"f1_mean": 0.6704939668174963,
|
| 750 |
+
"f1_sample_sd": 0.020973132363659372
|
| 751 |
+
},
|
| 752 |
+
{
|
| 753 |
+
"arm": "existing_specialized_separate_lineage",
|
| 754 |
+
"label": "Exaggeration,Minimisation",
|
| 755 |
+
"support": 22,
|
| 756 |
+
"precision_mean": 0.44589793281653745,
|
| 757 |
+
"precision_sample_sd": 0.010987847993006481,
|
| 758 |
+
"recall_mean": 0.9393939393939394,
|
| 759 |
+
"recall_sample_sd": 0.06943296507508849,
|
| 760 |
+
"f1_mean": 0.6043956043956044,
|
| 761 |
+
"f1_sample_sd": 0.022305256187015846
|
| 762 |
+
},
|
| 763 |
+
{
|
| 764 |
+
"arm": "existing_specialized_separate_lineage",
|
| 765 |
+
"label": "Flag-Waving",
|
| 766 |
+
"support": 17,
|
| 767 |
+
"precision_mean": 0.5576388888888889,
|
| 768 |
+
"precision_sample_sd": 0.03705367103320583,
|
| 769 |
+
"recall_mean": 0.8823529411764706,
|
| 770 |
+
"recall_sample_sd": 0.11764705882352944,
|
| 771 |
+
"f1_mean": 0.680769868923179,
|
| 772 |
+
"f1_sample_sd": 0.04164658769406446
|
| 773 |
+
},
|
| 774 |
+
{
|
| 775 |
+
"arm": "existing_specialized_separate_lineage",
|
| 776 |
+
"label": "Loaded_Language",
|
| 777 |
+
"support": 45,
|
| 778 |
+
"precision_mean": 0.8387650085763294,
|
| 779 |
+
"precision_sample_sd": 0.01782556577257849,
|
| 780 |
+
"recall_mean": 1.0,
|
| 781 |
+
"recall_sample_sd": 0.0,
|
| 782 |
+
"f1_mean": 0.9122448979591837,
|
| 783 |
+
"f1_sample_sd": 0.01060439269940129
|
| 784 |
+
},
|
| 785 |
+
{
|
| 786 |
+
"arm": "existing_specialized_separate_lineage",
|
| 787 |
+
"label": "Name_Calling,Labeling",
|
| 788 |
+
"support": 33,
|
| 789 |
+
"precision_mean": 0.6957104190221673,
|
| 790 |
+
"precision_sample_sd": 0.019201490809269257,
|
| 791 |
+
"recall_mean": 0.9696969696969697,
|
| 792 |
+
"recall_sample_sd": 0.030303030303030276,
|
| 793 |
+
"f1_mean": 0.8101049442821594,
|
| 794 |
+
"f1_sample_sd": 0.022092733328802537
|
| 795 |
+
},
|
| 796 |
+
{
|
| 797 |
+
"arm": "existing_specialized_separate_lineage",
|
| 798 |
+
"label": "Repetition",
|
| 799 |
+
"support": 23,
|
| 800 |
+
"precision_mean": 0.4781557067271353,
|
| 801 |
+
"precision_sample_sd": 0.009897987322673213,
|
| 802 |
+
"recall_mean": 0.9420289855072463,
|
| 803 |
+
"recall_sample_sd": 0.06641414050660639,
|
| 804 |
+
"f1_mean": 0.6337774426009719,
|
| 805 |
+
"f1_sample_sd": 0.016444118333485254
|
| 806 |
+
},
|
| 807 |
+
{
|
| 808 |
+
"arm": "existing_specialized_separate_lineage",
|
| 809 |
+
"label": "Slogans",
|
| 810 |
+
"support": 14,
|
| 811 |
+
"precision_mean": 0.4334215167548501,
|
| 812 |
+
"precision_sample_sd": 0.05811581861282867,
|
| 813 |
+
"recall_mean": 0.6904761904761905,
|
| 814 |
+
"recall_sample_sd": 0.1649572197684645,
|
| 815 |
+
"f1_mean": 0.5201316298877274,
|
| 816 |
+
"f1_sample_sd": 0.018567913895656114
|
| 817 |
+
},
|
| 818 |
+
{
|
| 819 |
+
"arm": "existing_specialized_separate_lineage",
|
| 820 |
+
"label": "Thought-terminating_Cliches",
|
| 821 |
+
"support": 7,
|
| 822 |
+
"precision_mean": 0.18446115288220552,
|
| 823 |
+
"precision_sample_sd": 0.03641251252864963,
|
| 824 |
+
"recall_mean": 0.6666666666666666,
|
| 825 |
+
"recall_sample_sd": 0.1649572197684645,
|
| 826 |
+
"f1_mean": 0.2868626868626869,
|
| 827 |
+
"f1_sample_sd": 0.0511620864429468
|
| 828 |
+
},
|
| 829 |
+
{
|
| 830 |
+
"arm": "existing_specialized_separate_lineage",
|
| 831 |
+
"label": "Whataboutism,Straw_Men,Red_Herring",
|
| 832 |
+
"support": 10,
|
| 833 |
+
"precision_mean": 0.2777777777777778,
|
| 834 |
+
"precision_sample_sd": 0.0481125224324688,
|
| 835 |
+
"recall_mean": 0.4,
|
| 836 |
+
"recall_sample_sd": 0.0,
|
| 837 |
+
"f1_mean": 0.32634032634032634,
|
| 838 |
+
"f1_sample_sd": 0.03229931575886014
|
| 839 |
+
}
|
| 840 |
+
],
|
| 841 |
+
"compression_ladder": [
|
| 842 |
+
{
|
| 843 |
+
"name": "24L upstream text teacher",
|
| 844 |
+
"layers": 24,
|
| 845 |
+
"parameters": 752393024,
|
| 846 |
+
"weight_bytes": 1504825632,
|
| 847 |
+
"weight_mib": 1435.1135559082031,
|
| 848 |
+
"weight_sha256": "30783da4651259c26b8a204821ea426b9526431e0c14da0d1c6fa284e7414e5e",
|
| 849 |
+
"kd_stage": "",
|
| 850 |
+
"mean_total_loss": "",
|
| 851 |
+
"mean_interface_loss": "",
|
| 852 |
+
"mean_final_loss": ""
|
| 853 |
+
},
|
| 854 |
+
{
|
| 855 |
+
"name": "8L task-free KD",
|
| 856 |
+
"layers": 8,
|
| 857 |
+
"parameters": 420318144,
|
| 858 |
+
"weight_bytes": 840647880,
|
| 859 |
+
"weight_mib": 801.7042922973633,
|
| 860 |
+
"weight_sha256": "086fbaf9a4838ecde56cd3541c9c3212b22d0c5ee745cf2c1c5e0bb31e0ec22c",
|
| 861 |
+
"kd_stage": "24to8",
|
| 862 |
+
"mean_total_loss": 0.17622653172702485,
|
| 863 |
+
"mean_interface_loss": 0.21562811763578793,
|
| 864 |
+
"mean_final_loss": 0.16626427527853593
|
| 865 |
+
},
|
| 866 |
+
{
|
| 867 |
+
"name": "6L task-free KD",
|
| 868 |
+
"layers": 6,
|
| 869 |
+
"parameters": 377207424,
|
| 870 |
+
"weight_bytes": 754423424,
|
| 871 |
+
"weight_mib": 719.4742431640625,
|
| 872 |
+
"weight_sha256": "492749c20f77a42f6bc993f0927a82217b94f872b7eacf25612f7520c9bc28ce",
|
| 873 |
+
"kd_stage": "8to6",
|
| 874 |
+
"mean_total_loss": 0.03382104352249371,
|
| 875 |
+
"mean_interface_loss": 0.04486069045515251,
|
| 876 |
+
"mean_final_loss": 0.031147060785087888
|
| 877 |
+
},
|
| 878 |
+
{
|
| 879 |
+
"name": "4L task-free KD root",
|
| 880 |
+
"layers": 4,
|
| 881 |
+
"parameters": 334096704,
|
| 882 |
+
"weight_bytes": 668198976,
|
| 883 |
+
"weight_mib": 637.2442016601562,
|
| 884 |
+
"weight_sha256": "2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3",
|
| 885 |
+
"kd_stage": "6to4",
|
| 886 |
+
"mean_total_loss": 0.047020394468859195,
|
| 887 |
+
"mean_interface_loss": 0.07048421185527332,
|
| 888 |
+
"mean_final_loss": 0.043119939065377366
|
| 889 |
+
}
|
| 890 |
+
]
|
| 891 |
}
|
benchmark/summary.json
ADDED
|
@@ -0,0 +1,891 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema": "standalone4l-complete-transfer-benchmark-v2",
|
| 3 |
+
"status": "complete_from_frozen_reports",
|
| 4 |
+
"claim_scope": "Exploratory fresh-head transfer evidence on one English task and a previously opened 55-article SemEval-derived test split.",
|
| 5 |
+
"headless_root_requires_adaptation": true,
|
| 6 |
+
"test_previously_opened": true,
|
| 7 |
+
"test_articles": 55,
|
| 8 |
+
"test_windows": 434,
|
| 9 |
+
"seeds": [
|
| 10 |
+
41,
|
| 11 |
+
42,
|
| 12 |
+
43
|
| 13 |
+
],
|
| 14 |
+
"three_seed_sd_is_not_confidence_interval": true,
|
| 15 |
+
"multilingual_downstream_evaluation_performed": false,
|
| 16 |
+
"upstream_multilingual_metadata": {
|
| 17 |
+
"source": "Qwen/Qwen3.5-0.8B official model card",
|
| 18 |
+
"languages_and_dialects_claimed_upstream": 201,
|
| 19 |
+
"scope": "inherited tokenizer and architecture coverage only"
|
| 20 |
+
},
|
| 21 |
+
"protocol": {
|
| 22 |
+
"epochs": 5,
|
| 23 |
+
"primary_epoch": 5,
|
| 24 |
+
"early_stopping": false,
|
| 25 |
+
"max_length": 256,
|
| 26 |
+
"stride": 128,
|
| 27 |
+
"window_target": "span-aligned technique labels",
|
| 28 |
+
"article_aggregation": "labelwise maximum probability",
|
| 29 |
+
"micro_batch_size": 1,
|
| 30 |
+
"gradient_accumulation": 32,
|
| 31 |
+
"effective_batch_size": 32,
|
| 32 |
+
"learning_rate": 2e-05,
|
| 33 |
+
"weight_decay": 0.01,
|
| 34 |
+
"gradient_clip": 1.0,
|
| 35 |
+
"optimizer": "AdamW",
|
| 36 |
+
"dtype": "bfloat16",
|
| 37 |
+
"thresholds": [
|
| 38 |
+
0.05,
|
| 39 |
+
0.1,
|
| 40 |
+
0.15,
|
| 41 |
+
0.2,
|
| 42 |
+
0.25,
|
| 43 |
+
0.3,
|
| 44 |
+
0.35,
|
| 45 |
+
0.4,
|
| 46 |
+
0.45,
|
| 47 |
+
0.5,
|
| 48 |
+
0.55,
|
| 49 |
+
0.6,
|
| 50 |
+
0.65,
|
| 51 |
+
0.7,
|
| 52 |
+
0.75,
|
| 53 |
+
0.8,
|
| 54 |
+
0.85,
|
| 55 |
+
0.9
|
| 56 |
+
],
|
| 57 |
+
"threshold_selection": "epoch-5 calibration macro-F1, then micro-F1"
|
| 58 |
+
},
|
| 59 |
+
"arms": {
|
| 60 |
+
"task_agnostic_base": {
|
| 61 |
+
"display": "Task-agnostic KD base",
|
| 62 |
+
"lineage": "root base plus a fresh 14-label transfer head",
|
| 63 |
+
"metrics": {
|
| 64 |
+
"macro_f1": {
|
| 65 |
+
"values": [
|
| 66 |
+
0.5883093293517536,
|
| 67 |
+
0.6028379036218096,
|
| 68 |
+
0.5712816197954101
|
| 69 |
+
],
|
| 70 |
+
"mean": 0.5874762842563245,
|
| 71 |
+
"sample_sd": 0.01579462678670608,
|
| 72 |
+
"n": 3
|
| 73 |
+
},
|
| 74 |
+
"micro_f1": {
|
| 75 |
+
"values": [
|
| 76 |
+
0.6468085106382978,
|
| 77 |
+
0.6655172413793103,
|
| 78 |
+
0.6465256797583082
|
| 79 |
+
],
|
| 80 |
+
"mean": 0.6529504772586388,
|
| 81 |
+
"sample_sd": 0.010884055708931178,
|
| 82 |
+
"n": 3
|
| 83 |
+
},
|
| 84 |
+
"exact_match": {
|
| 85 |
+
"values": [
|
| 86 |
+
0.05454545454545454,
|
| 87 |
+
0.03636363636363636,
|
| 88 |
+
0.0
|
| 89 |
+
],
|
| 90 |
+
"mean": 0.0303030303030303,
|
| 91 |
+
"sample_sd": 0.027773186030035393,
|
| 92 |
+
"n": 3
|
| 93 |
+
},
|
| 94 |
+
"predicted_positive_rate": {
|
| 95 |
+
"values": [
|
| 96 |
+
0.5714285714285714,
|
| 97 |
+
0.4090909090909091,
|
| 98 |
+
0.5155844155844156
|
| 99 |
+
],
|
| 100 |
+
"mean": 0.4987012987012987,
|
| 101 |
+
"sample_sd": 0.08247520160679181,
|
| 102 |
+
"n": 3
|
| 103 |
+
}
|
| 104 |
+
},
|
| 105 |
+
"resources": {
|
| 106 |
+
"parameters": {
|
| 107 |
+
"values": [
|
| 108 |
+
334111040.0,
|
| 109 |
+
334111040.0,
|
| 110 |
+
334111040.0
|
| 111 |
+
],
|
| 112 |
+
"mean": 334111040.0,
|
| 113 |
+
"sample_sd": 0.0,
|
| 114 |
+
"n": 3
|
| 115 |
+
},
|
| 116 |
+
"peak_cuda_allocated_bytes": {
|
| 117 |
+
"values": [
|
| 118 |
+
3380975616.0,
|
| 119 |
+
3380975616.0,
|
| 120 |
+
3380975616.0
|
| 121 |
+
],
|
| 122 |
+
"mean": 3380975616.0,
|
| 123 |
+
"sample_sd": 0.0,
|
| 124 |
+
"n": 3
|
| 125 |
+
},
|
| 126 |
+
"peak_cuda_reserved_bytes": {
|
| 127 |
+
"values": [
|
| 128 |
+
3519021056.0,
|
| 129 |
+
3519021056.0,
|
| 130 |
+
3510632448.0
|
| 131 |
+
],
|
| 132 |
+
"mean": 3516224853.3333335,
|
| 133 |
+
"sample_sd": 4843165.086926248,
|
| 134 |
+
"n": 3
|
| 135 |
+
},
|
| 136 |
+
"total_wall_seconds": {
|
| 137 |
+
"values": [
|
| 138 |
+
1583.0297977919981,
|
| 139 |
+
1588.617994428001,
|
| 140 |
+
1603.4310500120046
|
| 141 |
+
],
|
| 142 |
+
"mean": 1591.692947410668,
|
| 143 |
+
"sample_sd": 10.54249851412311,
|
| 144 |
+
"n": 3
|
| 145 |
+
},
|
| 146 |
+
"test_seconds_per_article": {
|
| 147 |
+
"values": [
|
| 148 |
+
0.08164824154545468,
|
| 149 |
+
0.07871871667289683,
|
| 150 |
+
0.08355205721825107
|
| 151 |
+
],
|
| 152 |
+
"mean": 0.08130633847886752,
|
| 153 |
+
"sample_sd": 0.0024347419754560395,
|
| 154 |
+
"n": 3
|
| 155 |
+
}
|
| 156 |
+
},
|
| 157 |
+
"thresholds": [
|
| 158 |
+
0.15,
|
| 159 |
+
0.3,
|
| 160 |
+
0.1
|
| 161 |
+
]
|
| 162 |
+
},
|
| 163 |
+
"structural_copy_control": {
|
| 164 |
+
"display": "Structural copy, no task-free KD",
|
| 165 |
+
"lineage": "same four-layer structure without task-free KD",
|
| 166 |
+
"metrics": {
|
| 167 |
+
"macro_f1": {
|
| 168 |
+
"values": [
|
| 169 |
+
0.5169424579262757,
|
| 170 |
+
0.5349405779540175,
|
| 171 |
+
0.5023778364496152
|
| 172 |
+
],
|
| 173 |
+
"mean": 0.5180869574433028,
|
| 174 |
+
"sample_sd": 0.016311512588628545,
|
| 175 |
+
"n": 3
|
| 176 |
+
},
|
| 177 |
+
"micro_f1": {
|
| 178 |
+
"values": [
|
| 179 |
+
0.5773955773955775,
|
| 180 |
+
0.6083445491251682,
|
| 181 |
+
0.585635359116022
|
| 182 |
+
],
|
| 183 |
+
"mean": 0.5904584952122559,
|
| 184 |
+
"sample_sd": 0.01602830914762526,
|
| 185 |
+
"n": 3
|
| 186 |
+
},
|
| 187 |
+
"exact_match": {
|
| 188 |
+
"values": [
|
| 189 |
+
0.0,
|
| 190 |
+
0.0,
|
| 191 |
+
0.0
|
| 192 |
+
],
|
| 193 |
+
"mean": 0.0,
|
| 194 |
+
"sample_sd": 0.0,
|
| 195 |
+
"n": 3
|
| 196 |
+
},
|
| 197 |
+
"predicted_positive_rate": {
|
| 198 |
+
"values": [
|
| 199 |
+
0.712987012987013,
|
| 200 |
+
0.6207792207792208,
|
| 201 |
+
0.5961038961038961
|
| 202 |
+
],
|
| 203 |
+
"mean": 0.6432900432900432,
|
| 204 |
+
"sample_sd": 0.06160737456395461,
|
| 205 |
+
"n": 3
|
| 206 |
+
}
|
| 207 |
+
},
|
| 208 |
+
"resources": {
|
| 209 |
+
"parameters": {
|
| 210 |
+
"values": [
|
| 211 |
+
334111040.0,
|
| 212 |
+
334111040.0,
|
| 213 |
+
334111040.0
|
| 214 |
+
],
|
| 215 |
+
"mean": 334111040.0,
|
| 216 |
+
"sample_sd": 0.0,
|
| 217 |
+
"n": 3
|
| 218 |
+
},
|
| 219 |
+
"peak_cuda_allocated_bytes": {
|
| 220 |
+
"values": [
|
| 221 |
+
3380975616.0,
|
| 222 |
+
3380975616.0,
|
| 223 |
+
3380975616.0
|
| 224 |
+
],
|
| 225 |
+
"mean": 3380975616.0,
|
| 226 |
+
"sample_sd": 0.0,
|
| 227 |
+
"n": 3
|
| 228 |
+
},
|
| 229 |
+
"peak_cuda_reserved_bytes": {
|
| 230 |
+
"values": [
|
| 231 |
+
3519021056.0,
|
| 232 |
+
3519021056.0,
|
| 233 |
+
3510632448.0
|
| 234 |
+
],
|
| 235 |
+
"mean": 3516224853.3333335,
|
| 236 |
+
"sample_sd": 4843165.086926248,
|
| 237 |
+
"n": 3
|
| 238 |
+
},
|
| 239 |
+
"total_wall_seconds": {
|
| 240 |
+
"values": [
|
| 241 |
+
1575.5905925639963,
|
| 242 |
+
1623.5639530190092,
|
| 243 |
+
1629.195114912989
|
| 244 |
+
],
|
| 245 |
+
"mean": 1609.4498868319981,
|
| 246 |
+
"sample_sd": 29.457874198835384,
|
| 247 |
+
"n": 3
|
| 248 |
+
},
|
| 249 |
+
"test_seconds_per_article": {
|
| 250 |
+
"values": [
|
| 251 |
+
0.07877608058201573,
|
| 252 |
+
0.08307423260041767,
|
| 253 |
+
0.08163602005436339
|
| 254 |
+
],
|
| 255 |
+
"mean": 0.08116211107893226,
|
| 256 |
+
"sample_sd": 0.002187914527825257,
|
| 257 |
+
"n": 3
|
| 258 |
+
}
|
| 259 |
+
},
|
| 260 |
+
"thresholds": [
|
| 261 |
+
0.05,
|
| 262 |
+
0.2,
|
| 263 |
+
0.2
|
| 264 |
+
]
|
| 265 |
+
},
|
| 266 |
+
"existing_specialized_separate_lineage": {
|
| 267 |
+
"display": "Existing specialized 4L, separate lineage",
|
| 268 |
+
"lineage": "historical task-specific shrink/distillation lineage",
|
| 269 |
+
"metrics": {
|
| 270 |
+
"macro_f1": {
|
| 271 |
+
"values": [
|
| 272 |
+
0.5832848452288824,
|
| 273 |
+
0.5958405181160762,
|
| 274 |
+
0.5835785045218137
|
| 275 |
+
],
|
| 276 |
+
"mean": 0.5875679559555907,
|
| 277 |
+
"sample_sd": 0.007165753447433348,
|
| 278 |
+
"n": 3
|
| 279 |
+
},
|
| 280 |
+
"micro_f1": {
|
| 281 |
+
"values": [
|
| 282 |
+
0.6494252873563218,
|
| 283 |
+
0.6458616010854816,
|
| 284 |
+
0.6486486486486486
|
| 285 |
+
],
|
| 286 |
+
"mean": 0.647978512363484,
|
| 287 |
+
"sample_sd": 0.0018739735696546695,
|
| 288 |
+
"n": 3
|
| 289 |
+
},
|
| 290 |
+
"exact_match": {
|
| 291 |
+
"values": [
|
| 292 |
+
0.0,
|
| 293 |
+
0.0,
|
| 294 |
+
0.01818181818181818
|
| 295 |
+
],
|
| 296 |
+
"mean": 0.006060606060606061,
|
| 297 |
+
"sample_sd": 0.01049727762162956,
|
| 298 |
+
"n": 3
|
| 299 |
+
},
|
| 300 |
+
"predicted_positive_rate": {
|
| 301 |
+
"values": [
|
| 302 |
+
0.5597402597402598,
|
| 303 |
+
0.612987012987013,
|
| 304 |
+
0.5688311688311688
|
| 305 |
+
],
|
| 306 |
+
"mean": 0.5805194805194805,
|
| 307 |
+
"sample_sd": 0.02848274311618351,
|
| 308 |
+
"n": 3
|
| 309 |
+
}
|
| 310 |
+
},
|
| 311 |
+
"resources": {
|
| 312 |
+
"parameters": {
|
| 313 |
+
"values": [
|
| 314 |
+
210903360.0,
|
| 315 |
+
210903360.0,
|
| 316 |
+
210903360.0
|
| 317 |
+
],
|
| 318 |
+
"mean": 210903360.0,
|
| 319 |
+
"sample_sd": 0.0,
|
| 320 |
+
"n": 3
|
| 321 |
+
},
|
| 322 |
+
"peak_cuda_allocated_bytes": {
|
| 323 |
+
"values": [
|
| 324 |
+
2143655936.0,
|
| 325 |
+
2143655936.0,
|
| 326 |
+
2143655936.0
|
| 327 |
+
],
|
| 328 |
+
"mean": 2143655936.0,
|
| 329 |
+
"sample_sd": 0.0,
|
| 330 |
+
"n": 3
|
| 331 |
+
},
|
| 332 |
+
"peak_cuda_reserved_bytes": {
|
| 333 |
+
"values": [
|
| 334 |
+
2281701376.0,
|
| 335 |
+
2281701376.0,
|
| 336 |
+
2273312768.0
|
| 337 |
+
],
|
| 338 |
+
"mean": 2278905173.3333335,
|
| 339 |
+
"sample_sd": 4843165.086926248,
|
| 340 |
+
"n": 3
|
| 341 |
+
},
|
| 342 |
+
"total_wall_seconds": {
|
| 343 |
+
"values": [
|
| 344 |
+
1634.488557530014,
|
| 345 |
+
1652.9397718419787,
|
| 346 |
+
1643.488202479988
|
| 347 |
+
],
|
| 348 |
+
"mean": 1643.6388439506602,
|
| 349 |
+
"sample_sd": 9.226529522850198,
|
| 350 |
+
"n": 3
|
| 351 |
+
},
|
| 352 |
+
"test_seconds_per_article": {
|
| 353 |
+
"values": [
|
| 354 |
+
0.0847959352360869,
|
| 355 |
+
0.08598086021798239,
|
| 356 |
+
0.08552987641781908
|
| 357 |
+
],
|
| 358 |
+
"mean": 0.08543555729062946,
|
| 359 |
+
"sample_sd": 0.0005980667826383295,
|
| 360 |
+
"n": 3
|
| 361 |
+
}
|
| 362 |
+
},
|
| 363 |
+
"thresholds": [
|
| 364 |
+
0.1,
|
| 365 |
+
0.05,
|
| 366 |
+
0.05
|
| 367 |
+
]
|
| 368 |
+
}
|
| 369 |
+
},
|
| 370 |
+
"same_seed_macro_f1_deltas_base_minus_control": [
|
| 371 |
+
0.07136687142547793,
|
| 372 |
+
0.06789732566779205,
|
| 373 |
+
0.06890378334579494
|
| 374 |
+
],
|
| 375 |
+
"mean_macro_f1_delta_base_minus_control": 0.06938932681302164,
|
| 376 |
+
"publication_gate_passed": true,
|
| 377 |
+
"per_label_summary": [
|
| 378 |
+
{
|
| 379 |
+
"arm": "task_agnostic_base",
|
| 380 |
+
"label": "Appeal_to_Authority",
|
| 381 |
+
"support": 11,
|
| 382 |
+
"precision_mean": 0.3958614484930274,
|
| 383 |
+
"precision_sample_sd": 0.1238064457010119,
|
| 384 |
+
"recall_mean": 0.5757575757575757,
|
| 385 |
+
"recall_sample_sd": 0.05248638810814781,
|
| 386 |
+
"f1_mean": 0.4657088122605364,
|
| 387 |
+
"f1_sample_sd": 0.1020990126979069
|
| 388 |
+
},
|
| 389 |
+
{
|
| 390 |
+
"arm": "task_agnostic_base",
|
| 391 |
+
"label": "Appeal_to_fear-prejudice",
|
| 392 |
+
"support": 24,
|
| 393 |
+
"precision_mean": 0.5540244550770866,
|
| 394 |
+
"precision_sample_sd": 0.02525673990402998,
|
| 395 |
+
"recall_mean": 0.7361111111111112,
|
| 396 |
+
"recall_sample_sd": 0.13393959390267993,
|
| 397 |
+
"f1_mean": 0.6277521761392729,
|
| 398 |
+
"f1_sample_sd": 0.04994879016183189
|
| 399 |
+
},
|
| 400 |
+
{
|
| 401 |
+
"arm": "task_agnostic_base",
|
| 402 |
+
"label": "Bandwagon,Reductio_ad_hitlerum",
|
| 403 |
+
"support": 7,
|
| 404 |
+
"precision_mean": 0.4851851851851852,
|
| 405 |
+
"precision_sample_sd": 0.07882887649552875,
|
| 406 |
+
"recall_mean": 0.8095238095238095,
|
| 407 |
+
"recall_sample_sd": 0.08247860988423221,
|
| 408 |
+
"f1_mean": 0.6006778309409889,
|
| 409 |
+
"f1_sample_sd": 0.04793776258918466
|
| 410 |
+
},
|
| 411 |
+
{
|
| 412 |
+
"arm": "task_agnostic_base",
|
| 413 |
+
"label": "Black-and-White_Fallacy",
|
| 414 |
+
"support": 12,
|
| 415 |
+
"precision_mean": 0.3667929292929293,
|
| 416 |
+
"precision_sample_sd": 0.07670649392711647,
|
| 417 |
+
"recall_mean": 0.47222222222222227,
|
| 418 |
+
"recall_sample_sd": 0.09622504486493764,
|
| 419 |
+
"f1_mean": 0.4053892966936445,
|
| 420 |
+
"f1_sample_sd": 0.04211369218048746
|
| 421 |
+
},
|
| 422 |
+
{
|
| 423 |
+
"arm": "task_agnostic_base",
|
| 424 |
+
"label": "Causal_Oversimplification",
|
| 425 |
+
"support": 18,
|
| 426 |
+
"precision_mean": 0.5428824049513704,
|
| 427 |
+
"precision_sample_sd": 0.039216350192122156,
|
| 428 |
+
"recall_mean": 0.7962962962962963,
|
| 429 |
+
"recall_sample_sd": 0.11564811108145183,
|
| 430 |
+
"f1_mean": 0.6446992723588468,
|
| 431 |
+
"f1_sample_sd": 0.0634561308339175
|
| 432 |
+
},
|
| 433 |
+
{
|
| 434 |
+
"arm": "task_agnostic_base",
|
| 435 |
+
"label": "Doubt",
|
| 436 |
+
"support": 22,
|
| 437 |
+
"precision_mean": 0.5223832528180354,
|
| 438 |
+
"precision_sample_sd": 0.06742780050556177,
|
| 439 |
+
"recall_mean": 0.9848484848484849,
|
| 440 |
+
"recall_sample_sd": 0.026243194054073875,
|
| 441 |
+
"f1_mean": 0.6802057822343391,
|
| 442 |
+
"f1_sample_sd": 0.04928561754686331
|
| 443 |
+
},
|
| 444 |
+
{
|
| 445 |
+
"arm": "task_agnostic_base",
|
| 446 |
+
"label": "Exaggeration,Minimisation",
|
| 447 |
+
"support": 22,
|
| 448 |
+
"precision_mean": 0.4620811287477954,
|
| 449 |
+
"precision_sample_sd": 0.016801198309751,
|
| 450 |
+
"recall_mean": 0.7727272727272727,
|
| 451 |
+
"recall_sample_sd": 0.15745916432444337,
|
| 452 |
+
"f1_mean": 0.5727040816326531,
|
| 453 |
+
"f1_sample_sd": 0.03645259990419198
|
| 454 |
+
},
|
| 455 |
+
{
|
| 456 |
+
"arm": "task_agnostic_base",
|
| 457 |
+
"label": "Flag-Waving",
|
| 458 |
+
"support": 17,
|
| 459 |
+
"precision_mean": 0.5642642642642642,
|
| 460 |
+
"precision_sample_sd": 0.1478145677656334,
|
| 461 |
+
"recall_mean": 0.8235294117647058,
|
| 462 |
+
"recall_sample_sd": 0.1764705882352941,
|
| 463 |
+
"f1_mean": 0.6464506172839506,
|
| 464 |
+
"f1_sample_sd": 0.035742219952758225
|
| 465 |
+
},
|
| 466 |
+
{
|
| 467 |
+
"arm": "task_agnostic_base",
|
| 468 |
+
"label": "Loaded_Language",
|
| 469 |
+
"support": 45,
|
| 470 |
+
"precision_mean": 0.8649032907793016,
|
| 471 |
+
"precision_sample_sd": 0.046984312428427724,
|
| 472 |
+
"recall_mean": 0.9851851851851852,
|
| 473 |
+
"recall_sample_sd": 0.012830005981991702,
|
| 474 |
+
"f1_mean": 0.9208741623431562,
|
| 475 |
+
"f1_sample_sd": 0.03200926878737141
|
| 476 |
+
},
|
| 477 |
+
{
|
| 478 |
+
"arm": "task_agnostic_base",
|
| 479 |
+
"label": "Name_Calling,Labeling",
|
| 480 |
+
"support": 33,
|
| 481 |
+
"precision_mean": 0.7380952380952381,
|
| 482 |
+
"precision_sample_sd": 0.0047619047619048005,
|
| 483 |
+
"recall_mean": 0.9090909090909091,
|
| 484 |
+
"recall_sample_sd": 0.10925912955951485,
|
| 485 |
+
"f1_mean": 0.8125087983911513,
|
| 486 |
+
"f1_sample_sd": 0.04252971464057227
|
| 487 |
+
},
|
| 488 |
+
{
|
| 489 |
+
"arm": "task_agnostic_base",
|
| 490 |
+
"label": "Repetition",
|
| 491 |
+
"support": 23,
|
| 492 |
+
"precision_mean": 0.5196620583717357,
|
| 493 |
+
"precision_sample_sd": 0.021645909050194662,
|
| 494 |
+
"recall_mean": 0.7536231884057971,
|
| 495 |
+
"recall_sample_sd": 0.06641414050660639,
|
| 496 |
+
"f1_mean": 0.6147520781519571,
|
| 497 |
+
"f1_sample_sd": 0.03505927109416221
|
| 498 |
+
},
|
| 499 |
+
{
|
| 500 |
+
"arm": "task_agnostic_base",
|
| 501 |
+
"label": "Slogans",
|
| 502 |
+
"support": 14,
|
| 503 |
+
"precision_mean": 0.5161361626878869,
|
| 504 |
+
"precision_sample_sd": 0.20305677930281646,
|
| 505 |
+
"recall_mean": 0.7380952380952381,
|
| 506 |
+
"recall_sample_sd": 0.10910894511799614,
|
| 507 |
+
"f1_mean": 0.5834824090638044,
|
| 508 |
+
"f1_sample_sd": 0.09862686289220693
|
| 509 |
+
},
|
| 510 |
+
{
|
| 511 |
+
"arm": "task_agnostic_base",
|
| 512 |
+
"label": "Thought-terminating_Cliches",
|
| 513 |
+
"support": 7,
|
| 514 |
+
"precision_mean": 0.3492063492063492,
|
| 515 |
+
"precision_sample_sd": 0.143516994603374,
|
| 516 |
+
"recall_mean": 0.3333333333333333,
|
| 517 |
+
"recall_sample_sd": 0.08247860988423225,
|
| 518 |
+
"f1_mean": 0.319014319014319,
|
| 519 |
+
"f1_sample_sd": 0.04017590711117064
|
| 520 |
+
},
|
| 521 |
+
{
|
| 522 |
+
"arm": "task_agnostic_base",
|
| 523 |
+
"label": "Whataboutism,Straw_Men,Red_Herring",
|
| 524 |
+
"support": 10,
|
| 525 |
+
"precision_mean": 0.2761904761904762,
|
| 526 |
+
"precision_sample_sd": 0.05302632726504782,
|
| 527 |
+
"recall_mean": 0.5,
|
| 528 |
+
"recall_sample_sd": 0.2645751311064591,
|
| 529 |
+
"f1_mean": 0.330448343079922,
|
| 530 |
+
"f1_sample_sd": 0.021845164720973014
|
| 531 |
+
},
|
| 532 |
+
{
|
| 533 |
+
"arm": "structural_copy_control",
|
| 534 |
+
"label": "Appeal_to_Authority",
|
| 535 |
+
"support": 11,
|
| 536 |
+
"precision_mean": 0.26003086419753085,
|
| 537 |
+
"precision_sample_sd": 0.06281357095350094,
|
| 538 |
+
"recall_mean": 0.5757575757575758,
|
| 539 |
+
"recall_sample_sd": 0.262431940540739,
|
| 540 |
+
"f1_mean": 0.35477582846003897,
|
| 541 |
+
"f1_sample_sd": 0.11479479036518875
|
| 542 |
+
},
|
| 543 |
+
{
|
| 544 |
+
"arm": "structural_copy_control",
|
| 545 |
+
"label": "Appeal_to_fear-prejudice",
|
| 546 |
+
"support": 24,
|
| 547 |
+
"precision_mean": 0.5104813315339631,
|
| 548 |
+
"precision_sample_sd": 0.038427508996734555,
|
| 549 |
+
"recall_mean": 0.8611111111111112,
|
| 550 |
+
"recall_sample_sd": 0.024056261216234387,
|
| 551 |
+
"f1_mean": 0.6401745363877202,
|
| 552 |
+
"f1_sample_sd": 0.029305467347150463
|
| 553 |
+
},
|
| 554 |
+
{
|
| 555 |
+
"arm": "structural_copy_control",
|
| 556 |
+
"label": "Bandwagon,Reductio_ad_hitlerum",
|
| 557 |
+
"support": 7,
|
| 558 |
+
"precision_mean": 0.2876623376623376,
|
| 559 |
+
"precision_sample_sd": 0.061386823077618426,
|
| 560 |
+
"recall_mean": 0.8571428571428571,
|
| 561 |
+
"recall_sample_sd": 0.14285714285714285,
|
| 562 |
+
"f1_mean": 0.4306391777656145,
|
| 563 |
+
"f1_sample_sd": 0.08686392624022733
|
| 564 |
+
},
|
| 565 |
+
{
|
| 566 |
+
"arm": "structural_copy_control",
|
| 567 |
+
"label": "Black-and-White_Fallacy",
|
| 568 |
+
"support": 12,
|
| 569 |
+
"precision_mean": 0.31339031339031337,
|
| 570 |
+
"precision_sample_sd": 0.08301311842925053,
|
| 571 |
+
"recall_mean": 0.5277777777777778,
|
| 572 |
+
"recall_sample_sd": 0.2678791878053599,
|
| 573 |
+
"f1_mean": 0.38778427550357375,
|
| 574 |
+
"f1_sample_sd": 0.13069755090515112
|
| 575 |
+
},
|
| 576 |
+
{
|
| 577 |
+
"arm": "structural_copy_control",
|
| 578 |
+
"label": "Causal_Oversimplification",
|
| 579 |
+
"support": 18,
|
| 580 |
+
"precision_mean": 0.4464201043148412,
|
| 581 |
+
"precision_sample_sd": 0.013538446216471536,
|
| 582 |
+
"recall_mean": 0.9259259259259259,
|
| 583 |
+
"recall_sample_sd": 0.03207501495497923,
|
| 584 |
+
"f1_mean": 0.6023809523809524,
|
| 585 |
+
"f1_sample_sd": 0.018643640071557464
|
| 586 |
+
},
|
| 587 |
+
{
|
| 588 |
+
"arm": "structural_copy_control",
|
| 589 |
+
"label": "Doubt",
|
| 590 |
+
"support": 22,
|
| 591 |
+
"precision_mean": 0.4455314009661836,
|
| 592 |
+
"precision_sample_sd": 0.04067072694857813,
|
| 593 |
+
"recall_mean": 1.0,
|
| 594 |
+
"recall_sample_sd": 0.0,
|
| 595 |
+
"f1_mean": 0.615686274509804,
|
| 596 |
+
"f1_sample_sd": 0.039427200694482675
|
| 597 |
+
},
|
| 598 |
+
{
|
| 599 |
+
"arm": "structural_copy_control",
|
| 600 |
+
"label": "Exaggeration,Minimisation",
|
| 601 |
+
"support": 22,
|
| 602 |
+
"precision_mean": 0.4364176150420584,
|
| 603 |
+
"precision_sample_sd": 0.025093958106097894,
|
| 604 |
+
"recall_mean": 0.8787878787878788,
|
| 605 |
+
"recall_sample_sd": 0.05248638810814775,
|
| 606 |
+
"f1_mean": 0.5829803047194351,
|
| 607 |
+
"f1_sample_sd": 0.030899288356345502
|
| 608 |
+
},
|
| 609 |
+
{
|
| 610 |
+
"arm": "structural_copy_control",
|
| 611 |
+
"label": "Flag-Waving",
|
| 612 |
+
"support": 17,
|
| 613 |
+
"precision_mean": 0.3814814814814815,
|
| 614 |
+
"precision_sample_sd": 0.05010277503136551,
|
| 615 |
+
"recall_mean": 0.803921568627451,
|
| 616 |
+
"recall_sample_sd": 0.17970885078258195,
|
| 617 |
+
"f1_mean": 0.5138595287119652,
|
| 618 |
+
"f1_sample_sd": 0.06400931989326974
|
| 619 |
+
},
|
| 620 |
+
{
|
| 621 |
+
"arm": "structural_copy_control",
|
| 622 |
+
"label": "Loaded_Language",
|
| 623 |
+
"support": 45,
|
| 624 |
+
"precision_mean": 0.8492581841638446,
|
| 625 |
+
"precision_sample_sd": 0.016026591848331686,
|
| 626 |
+
"recall_mean": 1.0,
|
| 627 |
+
"recall_sample_sd": 0.0,
|
| 628 |
+
"f1_mean": 0.9184311025253588,
|
| 629 |
+
"f1_sample_sd": 0.009372233868132662
|
| 630 |
+
},
|
| 631 |
+
{
|
| 632 |
+
"arm": "structural_copy_control",
|
| 633 |
+
"label": "Name_Calling,Labeling",
|
| 634 |
+
"support": 33,
|
| 635 |
+
"precision_mean": 0.6375,
|
| 636 |
+
"precision_sample_sd": 0.03410563654946854,
|
| 637 |
+
"recall_mean": 0.9696969696969697,
|
| 638 |
+
"recall_sample_sd": 0.030303030303030276,
|
| 639 |
+
"f1_mean": 0.7685185185185185,
|
| 640 |
+
"f1_sample_sd": 0.02023900779105559
|
| 641 |
+
},
|
| 642 |
+
{
|
| 643 |
+
"arm": "structural_copy_control",
|
| 644 |
+
"label": "Repetition",
|
| 645 |
+
"support": 23,
|
| 646 |
+
"precision_mean": 0.44863933452168747,
|
| 647 |
+
"precision_sample_sd": 0.01520014564765896,
|
| 648 |
+
"recall_mean": 0.9420289855072463,
|
| 649 |
+
"recall_sample_sd": 0.06641414050660639,
|
| 650 |
+
"f1_mean": 0.607248835423033,
|
| 651 |
+
"f1_sample_sd": 0.019858627551810427
|
| 652 |
+
},
|
| 653 |
+
{
|
| 654 |
+
"arm": "structural_copy_control",
|
| 655 |
+
"label": "Slogans",
|
| 656 |
+
"support": 14,
|
| 657 |
+
"precision_mean": 0.3274318274318274,
|
| 658 |
+
"precision_sample_sd": 0.05434598572773684,
|
| 659 |
+
"recall_mean": 0.4523809523809524,
|
| 660 |
+
"recall_sample_sd": 0.1486904285332952,
|
| 661 |
+
"f1_mean": 0.3715828924162258,
|
| 662 |
+
"f1_sample_sd": 0.07106660236343577
|
| 663 |
+
},
|
| 664 |
+
{
|
| 665 |
+
"arm": "structural_copy_control",
|
| 666 |
+
"label": "Thought-terminating_Cliches",
|
| 667 |
+
"support": 7,
|
| 668 |
+
"precision_mean": 0.1142156862745098,
|
| 669 |
+
"precision_sample_sd": 0.012848374923608973,
|
| 670 |
+
"recall_mean": 0.38095238095238093,
|
| 671 |
+
"recall_sample_sd": 0.1649572197684645,
|
| 672 |
+
"f1_mean": 0.17331433998100665,
|
| 673 |
+
"f1_sample_sd": 0.029065880233243847
|
| 674 |
+
},
|
| 675 |
+
{
|
| 676 |
+
"arm": "structural_copy_control",
|
| 677 |
+
"label": "Whataboutism,Straw_Men,Red_Herring",
|
| 678 |
+
"support": 10,
|
| 679 |
+
"precision_mean": 0.19344598054275472,
|
| 680 |
+
"precision_sample_sd": 0.03373872471827226,
|
| 681 |
+
"recall_mean": 0.5666666666666667,
|
| 682 |
+
"recall_sample_sd": 0.2081665999466133,
|
| 683 |
+
"f1_mean": 0.28584083690299267,
|
| 684 |
+
"f1_sample_sd": 0.06078854830133676
|
| 685 |
+
},
|
| 686 |
+
{
|
| 687 |
+
"arm": "existing_specialized_separate_lineage",
|
| 688 |
+
"label": "Appeal_to_Authority",
|
| 689 |
+
"support": 11,
|
| 690 |
+
"precision_mean": 0.37836021505376344,
|
| 691 |
+
"precision_sample_sd": 0.10546365636105155,
|
| 692 |
+
"recall_mean": 0.6666666666666666,
|
| 693 |
+
"recall_sample_sd": 0.2287828616748712,
|
| 694 |
+
"f1_mean": 0.46885361552028215,
|
| 695 |
+
"f1_sample_sd": 0.09502747629890008
|
| 696 |
+
},
|
| 697 |
+
{
|
| 698 |
+
"arm": "existing_specialized_separate_lineage",
|
| 699 |
+
"label": "Appeal_to_fear-prejudice",
|
| 700 |
+
"support": 24,
|
| 701 |
+
"precision_mean": 0.525708635464733,
|
| 702 |
+
"precision_sample_sd": 0.02235198659615819,
|
| 703 |
+
"recall_mean": 0.8888888888888888,
|
| 704 |
+
"recall_sample_sd": 0.048112522432468774,
|
| 705 |
+
"f1_mean": 0.6599065351235072,
|
| 706 |
+
"f1_sample_sd": 0.015362381542320084
|
| 707 |
+
},
|
| 708 |
+
{
|
| 709 |
+
"arm": "existing_specialized_separate_lineage",
|
| 710 |
+
"label": "Bandwagon,Reductio_ad_hitlerum",
|
| 711 |
+
"support": 7,
|
| 712 |
+
"precision_mean": 0.44485144485144484,
|
| 713 |
+
"precision_sample_sd": 0.016487284575115283,
|
| 714 |
+
"recall_mean": 0.7619047619047619,
|
| 715 |
+
"recall_sample_sd": 0.1649572197684645,
|
| 716 |
+
"f1_mean": 0.5571428571428572,
|
| 717 |
+
"f1_sample_sd": 0.051507875363771265
|
| 718 |
+
},
|
| 719 |
+
{
|
| 720 |
+
"arm": "existing_specialized_separate_lineage",
|
| 721 |
+
"label": "Black-and-White_Fallacy",
|
| 722 |
+
"support": 12,
|
| 723 |
+
"precision_mean": 0.4662644544997486,
|
| 724 |
+
"precision_sample_sd": 0.004538274146363688,
|
| 725 |
+
"recall_mean": 0.5833333333333334,
|
| 726 |
+
"recall_sample_sd": 0.08333333333333331,
|
| 727 |
+
"f1_mean": 0.516747552149851,
|
| 728 |
+
"f1_sample_sd": 0.03589484965697345
|
| 729 |
+
},
|
| 730 |
+
{
|
| 731 |
+
"arm": "existing_specialized_separate_lineage",
|
| 732 |
+
"label": "Causal_Oversimplification",
|
| 733 |
+
"support": 18,
|
| 734 |
+
"precision_mean": 0.4472934472934473,
|
| 735 |
+
"precision_sample_sd": 0.013055771210700413,
|
| 736 |
+
"recall_mean": 0.8333333333333333,
|
| 737 |
+
"recall_sample_sd": 0.1469861839480328,
|
| 738 |
+
"f1_mean": 0.5781794553724379,
|
| 739 |
+
"f1_sample_sd": 0.028407563015813122
|
| 740 |
+
},
|
| 741 |
+
{
|
| 742 |
+
"arm": "existing_specialized_separate_lineage",
|
| 743 |
+
"label": "Doubt",
|
| 744 |
+
"support": 22,
|
| 745 |
+
"precision_mean": 0.5045661001171619,
|
| 746 |
+
"precision_sample_sd": 0.023581174969843577,
|
| 747 |
+
"recall_mean": 1.0,
|
| 748 |
+
"recall_sample_sd": 0.0,
|
| 749 |
+
"f1_mean": 0.6704939668174963,
|
| 750 |
+
"f1_sample_sd": 0.020973132363659372
|
| 751 |
+
},
|
| 752 |
+
{
|
| 753 |
+
"arm": "existing_specialized_separate_lineage",
|
| 754 |
+
"label": "Exaggeration,Minimisation",
|
| 755 |
+
"support": 22,
|
| 756 |
+
"precision_mean": 0.44589793281653745,
|
| 757 |
+
"precision_sample_sd": 0.010987847993006481,
|
| 758 |
+
"recall_mean": 0.9393939393939394,
|
| 759 |
+
"recall_sample_sd": 0.06943296507508849,
|
| 760 |
+
"f1_mean": 0.6043956043956044,
|
| 761 |
+
"f1_sample_sd": 0.022305256187015846
|
| 762 |
+
},
|
| 763 |
+
{
|
| 764 |
+
"arm": "existing_specialized_separate_lineage",
|
| 765 |
+
"label": "Flag-Waving",
|
| 766 |
+
"support": 17,
|
| 767 |
+
"precision_mean": 0.5576388888888889,
|
| 768 |
+
"precision_sample_sd": 0.03705367103320583,
|
| 769 |
+
"recall_mean": 0.8823529411764706,
|
| 770 |
+
"recall_sample_sd": 0.11764705882352944,
|
| 771 |
+
"f1_mean": 0.680769868923179,
|
| 772 |
+
"f1_sample_sd": 0.04164658769406446
|
| 773 |
+
},
|
| 774 |
+
{
|
| 775 |
+
"arm": "existing_specialized_separate_lineage",
|
| 776 |
+
"label": "Loaded_Language",
|
| 777 |
+
"support": 45,
|
| 778 |
+
"precision_mean": 0.8387650085763294,
|
| 779 |
+
"precision_sample_sd": 0.01782556577257849,
|
| 780 |
+
"recall_mean": 1.0,
|
| 781 |
+
"recall_sample_sd": 0.0,
|
| 782 |
+
"f1_mean": 0.9122448979591837,
|
| 783 |
+
"f1_sample_sd": 0.01060439269940129
|
| 784 |
+
},
|
| 785 |
+
{
|
| 786 |
+
"arm": "existing_specialized_separate_lineage",
|
| 787 |
+
"label": "Name_Calling,Labeling",
|
| 788 |
+
"support": 33,
|
| 789 |
+
"precision_mean": 0.6957104190221673,
|
| 790 |
+
"precision_sample_sd": 0.019201490809269257,
|
| 791 |
+
"recall_mean": 0.9696969696969697,
|
| 792 |
+
"recall_sample_sd": 0.030303030303030276,
|
| 793 |
+
"f1_mean": 0.8101049442821594,
|
| 794 |
+
"f1_sample_sd": 0.022092733328802537
|
| 795 |
+
},
|
| 796 |
+
{
|
| 797 |
+
"arm": "existing_specialized_separate_lineage",
|
| 798 |
+
"label": "Repetition",
|
| 799 |
+
"support": 23,
|
| 800 |
+
"precision_mean": 0.4781557067271353,
|
| 801 |
+
"precision_sample_sd": 0.009897987322673213,
|
| 802 |
+
"recall_mean": 0.9420289855072463,
|
| 803 |
+
"recall_sample_sd": 0.06641414050660639,
|
| 804 |
+
"f1_mean": 0.6337774426009719,
|
| 805 |
+
"f1_sample_sd": 0.016444118333485254
|
| 806 |
+
},
|
| 807 |
+
{
|
| 808 |
+
"arm": "existing_specialized_separate_lineage",
|
| 809 |
+
"label": "Slogans",
|
| 810 |
+
"support": 14,
|
| 811 |
+
"precision_mean": 0.4334215167548501,
|
| 812 |
+
"precision_sample_sd": 0.05811581861282867,
|
| 813 |
+
"recall_mean": 0.6904761904761905,
|
| 814 |
+
"recall_sample_sd": 0.1649572197684645,
|
| 815 |
+
"f1_mean": 0.5201316298877274,
|
| 816 |
+
"f1_sample_sd": 0.018567913895656114
|
| 817 |
+
},
|
| 818 |
+
{
|
| 819 |
+
"arm": "existing_specialized_separate_lineage",
|
| 820 |
+
"label": "Thought-terminating_Cliches",
|
| 821 |
+
"support": 7,
|
| 822 |
+
"precision_mean": 0.18446115288220552,
|
| 823 |
+
"precision_sample_sd": 0.03641251252864963,
|
| 824 |
+
"recall_mean": 0.6666666666666666,
|
| 825 |
+
"recall_sample_sd": 0.1649572197684645,
|
| 826 |
+
"f1_mean": 0.2868626868626869,
|
| 827 |
+
"f1_sample_sd": 0.0511620864429468
|
| 828 |
+
},
|
| 829 |
+
{
|
| 830 |
+
"arm": "existing_specialized_separate_lineage",
|
| 831 |
+
"label": "Whataboutism,Straw_Men,Red_Herring",
|
| 832 |
+
"support": 10,
|
| 833 |
+
"precision_mean": 0.2777777777777778,
|
| 834 |
+
"precision_sample_sd": 0.0481125224324688,
|
| 835 |
+
"recall_mean": 0.4,
|
| 836 |
+
"recall_sample_sd": 0.0,
|
| 837 |
+
"f1_mean": 0.32634032634032634,
|
| 838 |
+
"f1_sample_sd": 0.03229931575886014
|
| 839 |
+
}
|
| 840 |
+
],
|
| 841 |
+
"compression_ladder": [
|
| 842 |
+
{
|
| 843 |
+
"name": "24L upstream text teacher",
|
| 844 |
+
"layers": 24,
|
| 845 |
+
"parameters": 752393024,
|
| 846 |
+
"weight_bytes": 1504825632,
|
| 847 |
+
"weight_mib": 1435.1135559082031,
|
| 848 |
+
"weight_sha256": "30783da4651259c26b8a204821ea426b9526431e0c14da0d1c6fa284e7414e5e",
|
| 849 |
+
"kd_stage": "",
|
| 850 |
+
"mean_total_loss": "",
|
| 851 |
+
"mean_interface_loss": "",
|
| 852 |
+
"mean_final_loss": ""
|
| 853 |
+
},
|
| 854 |
+
{
|
| 855 |
+
"name": "8L task-free KD",
|
| 856 |
+
"layers": 8,
|
| 857 |
+
"parameters": 420318144,
|
| 858 |
+
"weight_bytes": 840647880,
|
| 859 |
+
"weight_mib": 801.7042922973633,
|
| 860 |
+
"weight_sha256": "086fbaf9a4838ecde56cd3541c9c3212b22d0c5ee745cf2c1c5e0bb31e0ec22c",
|
| 861 |
+
"kd_stage": "24to8",
|
| 862 |
+
"mean_total_loss": 0.17622653172702485,
|
| 863 |
+
"mean_interface_loss": 0.21562811763578793,
|
| 864 |
+
"mean_final_loss": 0.16626427527853593
|
| 865 |
+
},
|
| 866 |
+
{
|
| 867 |
+
"name": "6L task-free KD",
|
| 868 |
+
"layers": 6,
|
| 869 |
+
"parameters": 377207424,
|
| 870 |
+
"weight_bytes": 754423424,
|
| 871 |
+
"weight_mib": 719.4742431640625,
|
| 872 |
+
"weight_sha256": "492749c20f77a42f6bc993f0927a82217b94f872b7eacf25612f7520c9bc28ce",
|
| 873 |
+
"kd_stage": "8to6",
|
| 874 |
+
"mean_total_loss": 0.03382104352249371,
|
| 875 |
+
"mean_interface_loss": 0.04486069045515251,
|
| 876 |
+
"mean_final_loss": 0.031147060785087888
|
| 877 |
+
},
|
| 878 |
+
{
|
| 879 |
+
"name": "4L task-free KD root",
|
| 880 |
+
"layers": 4,
|
| 881 |
+
"parameters": 334096704,
|
| 882 |
+
"weight_bytes": 668198976,
|
| 883 |
+
"weight_mib": 637.2442016601562,
|
| 884 |
+
"weight_sha256": "2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3",
|
| 885 |
+
"kd_stage": "6to4",
|
| 886 |
+
"mean_total_loss": 0.047020394468859195,
|
| 887 |
+
"mean_interface_loss": 0.07048421185527332,
|
| 888 |
+
"mean_final_loss": 0.043119939065377366
|
| 889 |
+
}
|
| 890 |
+
]
|
| 891 |
+
}
|
docs/RELEASE_CONTRACT.md
CHANGED
|
@@ -14,14 +14,13 @@
|
|
| 14 |
SemEval examples, labels, logits, thresholds, or evidence objectives.
|
| 15 |
2. `models/semeval-propaganda/`: the existing three-seed task-specific
|
| 16 |
classifier copied byte-for-byte as a separate historical lineage. It is not
|
| 17 |
-
described as a fine-tuned child of the
|
| 18 |
3. `distillation/`: the reusable 24L→8L→6L→4L platform for unlabeled
|
| 19 |
-
representation distillation and user-owned single-label or multilabel
|
| 20 |
-
classification data.
|
| 21 |
|
| 22 |
## Base training contract
|
| 23 |
|
| 24 |
-
- Source:
|
| 25 |
- Structural defaults:
|
| 26 |
- 24→8: `0,4,6,11,13,16,20,23`
|
| 27 |
- 8→6: `0,1,3,4,6,7`
|
|
@@ -31,43 +30,81 @@
|
|
| 31 |
- Objective: aligned hidden-boundary and final-representation distillation.
|
| 32 |
- One epoch per stage, seed 41, maximum length 128, batch size 1, gradient
|
| 33 |
accumulation 8, learning rate 2e-5, BF16 CUDA.
|
|
|
|
| 34 |
- No classification head is published at the root.
|
| 35 |
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
silently selecting one.
|
| 40 |
|
| 41 |
-
##
|
| 42 |
|
| 43 |
-
The
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
exploratory rather than confirmatory.
|
| 47 |
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
|
| 58 |
## Release gates
|
| 59 |
|
| 60 |
- Root `model.safetensors` contains no `score.weight`.
|
| 61 |
-
- Root loads with `AutoModel`
|
| 62 |
-
|
| 63 |
- Root weight SHA-256 is
|
| 64 |
`2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3`.
|
| 65 |
- Base manifests say `labels_read=false` and `semeval_used=false`.
|
| 66 |
-
-
|
|
|
|
|
|
|
| 67 |
- The platform validates label order and ID/group split leakage, never
|
| 68 |
overwrites output directories, hashes inputs/outputs, verifies immutable
|
| 69 |
teachers, and reloads fresh checkpoints.
|
| 70 |
- The release contains no SemEval or WikiText source data.
|
| 71 |
- The old specialized repository revision and inventory are checked before and
|
| 72 |
-
after
|
| 73 |
-
|
|
|
|
| 14 |
SemEval examples, labels, logits, thresholds, or evidence objectives.
|
| 15 |
2. `models/semeval-propaganda/`: the existing three-seed task-specific
|
| 16 |
classifier copied byte-for-byte as a separate historical lineage. It is not
|
| 17 |
+
described as a fine-tuned child of the root base.
|
| 18 |
3. `distillation/`: the reusable 24L→8L→6L→4L platform for unlabeled
|
| 19 |
+
representation distillation and user-owned single-label or multilabel data.
|
|
|
|
| 20 |
|
| 21 |
## Base training contract
|
| 22 |
|
| 23 |
+
- Source: text backbone and full tokenizer of `Qwen/Qwen3.5-0.8B`.
|
| 24 |
- Structural defaults:
|
| 25 |
- 24→8: `0,4,6,11,13,16,20,23`
|
| 26 |
- 8→6: `0,1,3,4,6,7`
|
|
|
|
| 30 |
- Objective: aligned hidden-boundary and final-representation distillation.
|
| 31 |
- One epoch per stage, seed 41, maximum length 128, batch size 1, gradient
|
| 32 |
accumulation 8, learning rate 2e-5, BF16 CUDA.
|
| 33 |
+
- Root artifact: 334,096,704 parameters and 668,198,976 BF16 weight bytes.
|
| 34 |
- No classification head is published at the root.
|
| 35 |
|
| 36 |
+
WikiText source text is not redistributed. Its dataset page's metadata and
|
| 37 |
+
prose currently disagree on the license version, so users must review the
|
| 38 |
+
upstream terms directly.
|
|
|
|
| 39 |
|
| 40 |
+
## Benchmark evidence contract
|
| 41 |
|
| 42 |
+
The headless root is evaluated through a fresh-head transfer probe, not by
|
| 43 |
+
pretending it is already a classifier. Both the task-agnostic base and the
|
| 44 |
+
structural-copy control use the same frozen protocol:
|
|
|
|
| 45 |
|
| 46 |
+
- SemEval-2020 Task 11 annotations converted to article-level 14-label targets;
|
| 47 |
+
- train/calibration/test article counts 260/56/55;
|
| 48 |
+
- seeds 41/42/43; five epochs; no early stopping;
|
| 49 |
+
- 256-token windows, stride 128, label-wise maximum article aggregation;
|
| 50 |
+
- micro batch 1, accumulation 32, AdamW, LR 2e-5, BF16;
|
| 51 |
+
- one global threshold selected on epoch-5 calibration Macro-F1 then Micro-F1;
|
| 52 |
+
- 55 previously opened test articles and 434 test windows.
|
| 53 |
|
| 54 |
+
| Initialization | Macro-F1 mean ± SD | Micro-F1 mean ± SD | Exact match mean ± SD |
|
| 55 |
+
|---|---:|---:|---:|
|
| 56 |
+
| Task-agnostic KD base | 0.58748 ± 0.01579 | 0.65295 ± 0.01088 | 0.03030 ± 0.02777 |
|
| 57 |
+
| Structural copy, no task-free KD | 0.51809 ± 0.01631 | 0.59046 ± 0.01603 | 0.00000 ± 0.00000 |
|
| 58 |
+
|
| 59 |
+
The observed same-seed mean Macro-F1 difference is +0.06939. All values and
|
| 60 |
+
sample SDs must be derived from the six frozen full reports at build time; the
|
| 61 |
+
card, JSON, and CSV tables must numerically close against those reports.
|
| 62 |
+
|
| 63 |
+
The existing specialized 4L lineage may appear only as a clearly labeled
|
| 64 |
+
reference because it uses a separate task-specific shrink/distillation history
|
| 65 |
+
and reduced vocabulary. It is not evidence that the task-free base produced
|
| 66 |
+
that checkpoint.
|
| 67 |
+
|
| 68 |
+
The report bundle must include:
|
| 69 |
+
|
| 70 |
+
- per-arm, per-seed Macro-F1, Micro-F1, exact match, positive rate, threshold;
|
| 71 |
+
- three-seed means and sample SDs;
|
| 72 |
+
- per-label precision, recall, F1, and fixed test support;
|
| 73 |
+
- parameters, CUDA allocation/reservation, total wall time, and test timing;
|
| 74 |
+
- compression-ladder depth, parameter count, BF16 size, hashes, and KD losses;
|
| 75 |
+
- normalized full frozen reports with article-level probability vectors.
|
| 76 |
+
|
| 77 |
+
The test is already opened, per-label supports are small, and three seeds are
|
| 78 |
+
not a confidence interval. No confirmatory, significance, universal
|
| 79 |
+
classification, or population-generalization claim is permitted.
|
| 80 |
+
|
| 81 |
+
## Multilingual metadata contract
|
| 82 |
+
|
| 83 |
+
The root keeps the full upstream tokenizer. The official Qwen3.5 card states
|
| 84 |
+
support for 201 languages and dialects and reports upstream multilingual
|
| 85 |
+
benchmarks. Therefore `language: multilingual` and a `multilingual` tag are
|
| 86 |
+
permitted as inherited input/architecture metadata.
|
| 87 |
+
|
| 88 |
+
This release's task-free KD corpus and only downstream probe are English.
|
| 89 |
+
Accordingly, the card must state prominently that the four-layer root has no
|
| 90 |
+
direct multilingual downstream evaluation and that tokenizer coverage is not a
|
| 91 |
+
multilingual classification-quality result. It may not claim verified quality
|
| 92 |
+
across 201 languages.
|
| 93 |
|
| 94 |
## Release gates
|
| 95 |
|
| 96 |
- Root `model.safetensors` contains no `score.weight`.
|
| 97 |
+
- Root loads with `AutoModel`; a temporary head can be attached with
|
| 98 |
+
`AutoModelForSequenceClassification`.
|
| 99 |
- Root weight SHA-256 is
|
| 100 |
`2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3`.
|
| 101 |
- Base manifests say `labels_read=false` and `semeval_used=false`.
|
| 102 |
+
- Root metadata contains `multilingual` and contains no propaganda task tag.
|
| 103 |
+
- Benchmark CSV/JSON/card values close against all frozen reports.
|
| 104 |
+
- Nested specialized checkpoint hashes equal the existing release.
|
| 105 |
- The platform validates label order and ID/group split leakage, never
|
| 106 |
overwrites output directories, hashes inputs/outputs, verifies immutable
|
| 107 |
teachers, and reloads fresh checkpoints.
|
| 108 |
- The release contains no SemEval or WikiText source data.
|
| 109 |
- The old specialized repository revision and inventory are checked before and
|
| 110 |
+
after updating only the new repository.
|
|
|
models/semeval-propaganda/README.md
CHANGED
|
@@ -23,5 +23,7 @@ model = AutoModelForSequenceClassification.from_pretrained(
|
|
| 23 |
```
|
| 24 |
|
| 25 |
The files below are byte-identical copies of the current release bundle. The
|
| 26 |
-
original release manifest
|
| 27 |
-
|
|
|
|
|
|
|
|
|
| 23 |
```
|
| 24 |
|
| 25 |
The files below are byte-identical copies of the current release bundle. The
|
| 26 |
+
original release manifest, locked thresholds, three full reports, and aggregate
|
| 27 |
+
are included for provenance. The repository-level `benchmark/` directory also
|
| 28 |
+
places this lineage beside the task-agnostic transfer probe as a clearly marked
|
| 29 |
+
reference. No SemEval source article or annotation file is redistributed.
|
models/semeval-propaganda/seeds/seed42/tokenizer.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
models/semeval-propaganda/seeds/seed43/tokenizer.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
models/semeval-propaganda/tokenizer.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
release_manifest.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"schema": "qwen35-unified-classification-base-release-
|
| 3 |
"status": "complete",
|
| 4 |
"repo_id": "mp-juuuns/qwen35-standalone4l-classification-base",
|
| 5 |
"old_repo_preserved": {
|
|
@@ -8,6 +8,8 @@
|
|
| 8 |
},
|
| 9 |
"root_role": "task_agnostic_headless_classification_backbone",
|
| 10 |
"root_weight_sha256": "2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3",
|
|
|
|
|
|
|
| 11 |
"nested_specialized_is_separate_lineage": true,
|
| 12 |
"source_data_redistributed": false,
|
| 13 |
"files_before_manifest": [
|
|
@@ -28,57 +30,134 @@
|
|
| 28 |
{
|
| 29 |
"path": "README.md",
|
| 30 |
"role": "root model card",
|
| 31 |
-
"bytes":
|
| 32 |
-
"sha256": "
|
| 33 |
"source": "docs/huggingface/UNIFIED_BASE_RELEASE_MODEL_CARD.md"
|
| 34 |
},
|
| 35 |
{
|
| 36 |
-
"path": "benchmark/
|
| 37 |
-
"role": "
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
"bytes": 115612,
|
| 39 |
"sha256": "816c3249d3ec281ef73d34dc3ab7576a48415d250c32977eb94ca37f73b0b810",
|
| 40 |
"source": "generated"
|
| 41 |
},
|
| 42 |
{
|
| 43 |
-
"path": "benchmark/reports/
|
| 44 |
-
"role": "normalized
|
| 45 |
"bytes": 115225,
|
| 46 |
"sha256": "0b19eabe1e31d0012c754db3b31173f8a6fcdee30973f1a1bebd3a370d11064c",
|
| 47 |
"source": "generated"
|
| 48 |
},
|
| 49 |
{
|
| 50 |
-
"path": "benchmark/reports/
|
| 51 |
-
"role": "normalized
|
| 52 |
"bytes": 115744,
|
| 53 |
"sha256": "6dde34173462eac50141ff082c18194f3233f8c06853532d9b12756562e69e2e",
|
| 54 |
"source": "generated"
|
| 55 |
},
|
| 56 |
{
|
| 57 |
-
"path": "benchmark/reports/
|
| 58 |
-
"role": "normalized
|
| 59 |
"bytes": 115885,
|
| 60 |
"sha256": "6b3a2e0372a83e1e71c2eca9596efc8cd1a3e39c28e25e3373f195431b5858a4",
|
| 61 |
"source": "generated"
|
| 62 |
},
|
| 63 |
{
|
| 64 |
-
"path": "benchmark/reports/
|
| 65 |
-
"role": "normalized
|
| 66 |
"bytes": 115674,
|
| 67 |
"sha256": "42dc788983716be0e9847aad87ff79c7e7297884f2635a8d098639be77326639",
|
| 68 |
"source": "generated"
|
| 69 |
},
|
| 70 |
{
|
| 71 |
-
"path": "benchmark/reports/
|
| 72 |
-
"role": "normalized
|
| 73 |
"bytes": 115062,
|
| 74 |
"sha256": "7ee52dc94a152a2f767dcff171f3c9cc1072fabd3ebc7b14db53617b7b1d33d5",
|
| 75 |
"source": "generated"
|
| 76 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 77 |
{
|
| 78 |
"path": "benchmark/semeval-transfer-summary.json",
|
| 79 |
-
"role": "
|
| 80 |
-
"bytes":
|
| 81 |
-
"sha256": "
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
"source": "generated"
|
| 83 |
},
|
| 84 |
{
|
|
@@ -203,8 +282,8 @@
|
|
| 203 |
{
|
| 204 |
"path": "docs/RELEASE_CONTRACT.md",
|
| 205 |
"role": "release contract",
|
| 206 |
-
"bytes":
|
| 207 |
-
"sha256": "
|
| 208 |
"source": "docs/huggingface/UNIFIED_BASE_RELEASE_CONTRACT.md"
|
| 209 |
},
|
| 210 |
{
|
|
@@ -231,8 +310,8 @@
|
|
| 231 |
{
|
| 232 |
"path": "models/semeval-propaganda/README.md",
|
| 233 |
"role": "specialized lineage card",
|
| 234 |
-
"bytes":
|
| 235 |
-
"sha256": "
|
| 236 |
"source": "generated"
|
| 237 |
},
|
| 238 |
{
|
|
|
|
| 1 |
{
|
| 2 |
+
"schema": "qwen35-unified-classification-base-release-v2",
|
| 3 |
"status": "complete",
|
| 4 |
"repo_id": "mp-juuuns/qwen35-standalone4l-classification-base",
|
| 5 |
"old_repo_preserved": {
|
|
|
|
| 8 |
},
|
| 9 |
"root_role": "task_agnostic_headless_classification_backbone",
|
| 10 |
"root_weight_sha256": "2732c616772fe320cdea228ab4554981418b1b2bf615c4183fb1ac8e6e2168d3",
|
| 11 |
+
"root_multilingual_metadata_scope": "inherited tokenizer/architecture coverage; no direct multilingual downstream evaluation",
|
| 12 |
+
"benchmark_schema": "standalone4l-complete-transfer-benchmark-v2",
|
| 13 |
"nested_specialized_is_separate_lineage": true,
|
| 14 |
"source_data_redistributed": false,
|
| 15 |
"files_before_manifest": [
|
|
|
|
| 30 |
{
|
| 31 |
"path": "README.md",
|
| 32 |
"role": "root model card",
|
| 33 |
+
"bytes": 12909,
|
| 34 |
+
"sha256": "5fcf007de6b6e828711c874310de1e042f1dada6989c70930ac0881285a9dc61",
|
| 35 |
"source": "docs/huggingface/UNIFIED_BASE_RELEASE_MODEL_CARD.md"
|
| 36 |
},
|
| 37 |
{
|
| 38 |
+
"path": "benchmark/BENCHMARK_CARD.md",
|
| 39 |
+
"role": "generated benchmark report",
|
| 40 |
+
"bytes": 3173,
|
| 41 |
+
"sha256": "bf4073de38c3905effd2dcce6f38894f325c8a0a79bedfb2779f0545c86b9c9f",
|
| 42 |
+
"source": "generated"
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"path": "benchmark/arm_summary.csv",
|
| 46 |
+
"role": "three-seed metric summary",
|
| 47 |
+
"bytes": 1655,
|
| 48 |
+
"sha256": "e01bd7ac59a3212e44b77e5fcfbc7c41ec91d74c998306d2438209d6ef1a3aa7",
|
| 49 |
+
"source": "generated"
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"path": "benchmark/compression_ladder.csv",
|
| 53 |
+
"role": "24L to 4L compression evidence",
|
| 54 |
+
"bytes": 827,
|
| 55 |
+
"sha256": "2c544c3907c4f592d724f658850b8c6b9634845cb5b50a66ebd2ea6c5d48ce0b",
|
| 56 |
+
"source": "generated"
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"path": "benchmark/per_label_metrics.csv",
|
| 60 |
+
"role": "per-label metrics by arm and seed",
|
| 61 |
+
"bytes": 12963,
|
| 62 |
+
"sha256": "ffc21be32b2106c7bf1df3854097a975ac370da01ac1b732eb5c2ba9075ddeda",
|
| 63 |
+
"source": "generated"
|
| 64 |
+
},
|
| 65 |
+
{
|
| 66 |
+
"path": "benchmark/per_label_summary.csv",
|
| 67 |
+
"role": "per-label three-seed summary",
|
| 68 |
+
"bytes": 7009,
|
| 69 |
+
"sha256": "c8e3876abf6cfe960c9332d3bc9d1fbde2f3213f3111351acf83b21bbb184049",
|
| 70 |
+
"source": "generated"
|
| 71 |
+
},
|
| 72 |
+
{
|
| 73 |
+
"path": "benchmark/reports/existing_specialized_separate_lineage-seed41.json",
|
| 74 |
+
"role": "normalized frozen benchmark report",
|
| 75 |
+
"bytes": 115793,
|
| 76 |
+
"sha256": "c1bd379680abcf50b47b7d23e4aafbc8b5e7f40a0b0e9700bde21a8ce49af803",
|
| 77 |
+
"source": "generated"
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
"path": "benchmark/reports/existing_specialized_separate_lineage-seed42.json",
|
| 81 |
+
"role": "normalized frozen benchmark report",
|
| 82 |
+
"bytes": 115780,
|
| 83 |
+
"sha256": "a30e38433d008db4aa7e5fbfd477ab80a0daf50e970f8e7b04397efc1a47843a",
|
| 84 |
+
"source": "generated"
|
| 85 |
+
},
|
| 86 |
+
{
|
| 87 |
+
"path": "benchmark/reports/existing_specialized_separate_lineage-seed43.json",
|
| 88 |
+
"role": "normalized frozen benchmark report",
|
| 89 |
+
"bytes": 115985,
|
| 90 |
+
"sha256": "96466290d09d27b8bc044a9f9a95765ce12b4d7b02dfb0123e436e7535623353",
|
| 91 |
+
"source": "generated"
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"path": "benchmark/reports/structural_copy_control-seed41.json",
|
| 95 |
+
"role": "normalized frozen benchmark report",
|
| 96 |
"bytes": 115612,
|
| 97 |
"sha256": "816c3249d3ec281ef73d34dc3ab7576a48415d250c32977eb94ca37f73b0b810",
|
| 98 |
"source": "generated"
|
| 99 |
},
|
| 100 |
{
|
| 101 |
+
"path": "benchmark/reports/structural_copy_control-seed42.json",
|
| 102 |
+
"role": "normalized frozen benchmark report",
|
| 103 |
"bytes": 115225,
|
| 104 |
"sha256": "0b19eabe1e31d0012c754db3b31173f8a6fcdee30973f1a1bebd3a370d11064c",
|
| 105 |
"source": "generated"
|
| 106 |
},
|
| 107 |
{
|
| 108 |
+
"path": "benchmark/reports/structural_copy_control-seed43.json",
|
| 109 |
+
"role": "normalized frozen benchmark report",
|
| 110 |
"bytes": 115744,
|
| 111 |
"sha256": "6dde34173462eac50141ff082c18194f3233f8c06853532d9b12756562e69e2e",
|
| 112 |
"source": "generated"
|
| 113 |
},
|
| 114 |
{
|
| 115 |
+
"path": "benchmark/reports/task_agnostic_base-seed41.json",
|
| 116 |
+
"role": "normalized frozen benchmark report",
|
| 117 |
"bytes": 115885,
|
| 118 |
"sha256": "6b3a2e0372a83e1e71c2eca9596efc8cd1a3e39c28e25e3373f195431b5858a4",
|
| 119 |
"source": "generated"
|
| 120 |
},
|
| 121 |
{
|
| 122 |
+
"path": "benchmark/reports/task_agnostic_base-seed42.json",
|
| 123 |
+
"role": "normalized frozen benchmark report",
|
| 124 |
"bytes": 115674,
|
| 125 |
"sha256": "42dc788983716be0e9847aad87ff79c7e7297884f2635a8d098639be77326639",
|
| 126 |
"source": "generated"
|
| 127 |
},
|
| 128 |
{
|
| 129 |
+
"path": "benchmark/reports/task_agnostic_base-seed43.json",
|
| 130 |
+
"role": "normalized frozen benchmark report",
|
| 131 |
"bytes": 115062,
|
| 132 |
"sha256": "7ee52dc94a152a2f767dcff171f3c9cc1072fabd3ebc7b14db53617b7b1d33d5",
|
| 133 |
"source": "generated"
|
| 134 |
},
|
| 135 |
+
{
|
| 136 |
+
"path": "benchmark/resource_metrics.csv",
|
| 137 |
+
"role": "resource measurements by arm and seed",
|
| 138 |
+
"bytes": 1649,
|
| 139 |
+
"sha256": "bf5cefdec3b2f3646114bd18bbf8e5e39721bde768ba968d73f22e96fc9ca8cf",
|
| 140 |
+
"source": "generated"
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
"path": "benchmark/seed_metrics.csv",
|
| 144 |
+
"role": "overall metrics by arm and seed",
|
| 145 |
+
"bytes": 1175,
|
| 146 |
+
"sha256": "e0b08a958d25243158a03dff76ca153d2ded76cc3f1771e4046a0068688ad6a8",
|
| 147 |
+
"source": "generated"
|
| 148 |
+
},
|
| 149 |
{
|
| 150 |
"path": "benchmark/semeval-transfer-summary.json",
|
| 151 |
+
"role": "compatibility alias for complete transfer summary",
|
| 152 |
+
"bytes": 26913,
|
| 153 |
+
"sha256": "3d2c9fcf9c6cc7da1a885968b70a42eaa38659d317c9d3290a2b3066f3f9ee6a",
|
| 154 |
+
"source": "generated"
|
| 155 |
+
},
|
| 156 |
+
{
|
| 157 |
+
"path": "benchmark/summary.json",
|
| 158 |
+
"role": "complete machine-readable transfer benchmark",
|
| 159 |
+
"bytes": 26913,
|
| 160 |
+
"sha256": "3d2c9fcf9c6cc7da1a885968b70a42eaa38659d317c9d3290a2b3066f3f9ee6a",
|
| 161 |
"source": "generated"
|
| 162 |
},
|
| 163 |
{
|
|
|
|
| 282 |
{
|
| 283 |
"path": "docs/RELEASE_CONTRACT.md",
|
| 284 |
"role": "release contract",
|
| 285 |
+
"bytes": 5381,
|
| 286 |
+
"sha256": "1b6e475b9d7d9af52bb8845de020303b1c6e89646d8e4f9be17dbe4ab2910417",
|
| 287 |
"source": "docs/huggingface/UNIFIED_BASE_RELEASE_CONTRACT.md"
|
| 288 |
},
|
| 289 |
{
|
|
|
|
| 310 |
{
|
| 311 |
"path": "models/semeval-propaganda/README.md",
|
| 312 |
"role": "specialized lineage card",
|
| 313 |
+
"bytes": 1403,
|
| 314 |
+
"sha256": "bf4ba7cc2514502a28f424507c41c1cfce5089c051ba7500cf46c1d13860722f",
|
| 315 |
"source": "generated"
|
| 316 |
},
|
| 317 |
{
|