Promote DeepSeek-teacher MiniLM classifier

Promotes verified release branch deepseek-teacher-20260719 (21af04487144d7be1f34a3b78a7305d94b7e3c6f) to main. Previous main remains available at c4dd45e8c984122c85d4cc1599f69618888b0a7a.

README.md

@@ -1,76 +1,67 @@
 ---
-language: en
-license: agpl-3.0
+language:
+  - en
 library_name: transformers
-tags:
-  - text-classification
-  - scientific-papers
-  - oecd-fields-of-science
-  - multitask
-base_model: allenai/specter2_base
+license: apache-2.0
 pipeline_tag: text-classification
+tags:
+  - bibr
+  - OECD
+  - scientific-paper-classification
+  - MiniLM
 ---
 
-# bibr paper classifier
-
-A multitask classifier that predicts, from a scientific paper's **title + abstract**:
+# bibr OECD/paper-type classifier — DeepSeek teacher, 2026-07-19
 
-- **OECD field of science, level 1** (`oecd_domain`) — 6 domains
-- **OECD field of science, level 2** (`oecd_subdomain`) — 36 subdomains
-- **paper type** — empirical, review, meta-analysis, case-study, commentary, corrigendum, erratum, retraction
-
-It is the field/type classification component of [bibr](https://bibr.org), a scientific-paper
-metadata extraction pipeline. In bibr it replaces a per-paper LLM classification call: a shared
-SPECTER2 encoder with three linear heads runs locally at zero marginal cost and is substantially
-more accurate than the zero-shot LLM it supersedes.
+This is the DeepSeek-v4-Flash-teacher MiniLM candidate for bibr's multitask
+scientific-paper classifier. It predicts OECD Level 1, OECD Level 2, and paper
+type from title plus abstract.
 
 ## Architecture
 
-`allenai/specter2_base` encoder → mean-pooled last hidden state → three linear heads
-(`l1_head`, `l2_head`, `paper_type_head`). Input text is `"{title} [SEP] {abstract}"`.
-
-## Results (held-out test)
-
-Measured on the deployment condition — title + a clean abstract, as provided at inference:
+- Encoder: `sentence-transformers/all-MiniLM-L6-v2`
+- Input template: `title_abstract_v1`
+- Maximum input length: 256 tokens
+- Heads: OECD L1, OECD L2, paper type
+- Model SHA-256:
+  `7a46c595cf3bb8e1eff39303786e8b7d16c95ea87f9f06b12d4a753f0be1e580`
 
-| task | macro-F1 | micro-F1 |
-|------|---------:|---------:|
-| OECD L1 (`oecd_domain`) | 0.742 | 0.768 |
-| OECD L2 (`oecd_subdomain`) | 0.461 | 0.599 |
-| paper type | 0.936 | 0.934 |
+## Training provenance
 
-paper_type per class: retraction 0.99, erratum 0.98, case-study 0.94, meta-analysis 0.93,
-commentary 0.92, empirical 0.90, review 0.89. `corrigendum` is not learnable from the available
-data (n=2) and is handled by a deterministic title guard in bibr rather than this model.
+- Training corpus rows: 122,363
+- Deterministic split: 91,771 train / 12,237 validation / 18,355 test
+- DeepSeek teacher changed 16,220 L1 labels relative to the matched OpenAlex
+  baseline.
+- Teacher-labelled rows are training supervision, not evaluation gold.
 
-For comparison, the zero-shot LLM this model replaces scores L1 macro **0.56** (title-only) / 0.35
-(title+abstract) against the same labels.
+## Evaluation
 
-### Confidence gating
+On the frozen 1,500-row Phase-A panel:
 
-The heads are softmax-scored; `paper_type` logits are temperature-scaled
-(`paper_type_temperature` in `inference_config.json`, Guo et al. 2017). Recommended emission policy:
-emit `oecd_domain` and `paper_type` always, and emit `oecd_subdomain` only above a confidence
-threshold (else null). Illustrative L2 precision/coverage: thr 0.5 → 66% coverage @ 0.60 precision;
-thr 0.8 → 36% coverage @ 0.74 precision.
+| Model | Accuracy | Macro-F1 |
+|---|---:|---:|
+| Matched OpenAlex MiniLM baseline | 0.7333 | 0.7280 |
+| This checkpoint | **0.8120** | **0.8096** |
+| Previously shipped SPECTER2 checkpoint | 0.6887 | 0.6874 |
 
-## Training data & labels
+The matched MiniLM gain was +0.0787 accuracy and +0.0816 macro-F1. The paired
+correctness table contained 158 teacher-only correct rows and 40 baseline-only
+correct rows; exact McNemar p-value was `8.687e-18`.
 
-- **OECD labels** are derived from the OpenAlex `primary_topic` taxonomy (CC0), mapped to the OECD
-  Fields of Science hierarchy. They are **not** LLM-generated. Title/abstract pairs whose abstract
-  was crossed from an unrelated work (a known OpenAlex data artifact) were detected with an
-  embedding-cosine consistency filter and trained title-only to avoid poisoning the input.
-- **paper_type labels** for the rarer classes come from MEDLINE/PubMed PublicationType metadata
-  (public domain), with a smaller set of LLM-adjudicated labels for the ambiguous majority.
+The Phase-A reference is explicitly
+`provisional_unadjudicated_codex_panel`. It is not human-adjudicated gold, so
+these figures support candidate selection but not a final scientific-quality
+claim.
 
-## Intended use & limitations
+## Bundle contract
 
-Designed for English scientific papers with a title and abstract. OECD field assignment is
-inherently ambiguous for interdisciplinary work; L2 in particular should be treated as a
-confidence-gated hint, not ground truth. Not suitable as a sole basis for high-stakes
-categorization.
+The bibr loader requires:
 
-## License
+- `model.safetensors`
+- `tokenizer.json`
+- `tokenizer_config.json`
+- `label_maps.json`
+- `inference_config.json`
 
-Released under AGPL-3.0, matching the bibr project. Training-data sources (OpenAlex, PubMed) carry
-their own open licenses noted above.
+`test_metrics.json`, `test_l1_diagnostics.json`, and `phaseA_metrics.json`
+provide training and provisional-panel diagnostics.

inference_config.json

@@ -1,9 +1,8 @@
 {
-  "encoder_name": "allenai/specter2_base",
+  "encoder_name": "sentence-transformers/all-MiniLM-L6-v2",
   "input_template": "title_abstract_v1",
-  "max_length": 384,
+  "max_length": 256,
   "model_kind": "paper_classifier_oecd_multitask_transformer",
-  "paper_type_temperature": 1.233031831988202,
   "tasks": [
     "oecd_l1",
     "oecd_l2",

label_maps.json

@@ -8,7 +8,6 @@
     "Social Sciences"
   ],
   "l2_classes": [
-    "Agricultural Biotechnology",
     "Agriculture, Forestry, and Fisheries",
     "Animal and Dairy Science",
     "Arts (arts, history of arts, performing arts, music)",
@@ -23,11 +22,9 @@
     "Economics and Business",
     "Education",
     "Electrical Engineering, Electronic Engineering, Information Engineering",
-    "Environmental Biotechnology",
     "Environmental Engineering",
     "Health Sciences",
     "History and Archaeology",
-    "Industrial Biotechnology",
     "Languages and Literature",
     "Law",
     "Materials Engineering",
@@ -36,7 +33,6 @@
     "Media and Communications",
     "Medical Biotechnology",
     "Medical Engineering",
-    "Nano-technology",
     "Philosophy, Ethics and Religion",
     "Physical Sciences",
     "Political Science",
@@ -48,7 +44,6 @@
   "paper_type_classes": [
     "case-study",
     "commentary",
-    "corrigendum",
     "empirical",
     "erratum",
     "meta-analysis",

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f38cf907fde11972314785bd8b6cd7fb484d5238fa435e48ad30e137016309c9
-size 439852128
+oid sha256:7a46c595cf3bb8e1eff39303786e8b7d16c95ea87f9f06b12d4a753f0be1e580
+size 90934820

phaseA_metrics.json

@@ -0,0 +1,168 @@
+{
+  "accuracy_all_lower_bound": 0.812,
+  "accuracy_scored": 0.812,
+  "classification_report": {
+    "Agricultural and Veterinary Sciences": {
+      "f1-score": 0.7692307692307693,
+      "precision": 0.8072916666666666,
+      "recall": 0.7345971563981043,
+      "support": 211.0
+    },
+    "Engineering and Technology": {
+      "f1-score": 0.8514851485148515,
+      "precision": 0.8704453441295547,
+      "recall": 0.8333333333333334,
+      "support": 258.0
+    },
+    "Humanities and the Arts": {
+      "f1-score": 0.8218527315914489,
+      "precision": 0.8084112149532711,
+      "recall": 0.8357487922705314,
+      "support": 207.0
+    },
+    "Medical and Health Sciences": {
+      "f1-score": 0.8881118881118881,
+      "precision": 0.9071428571428571,
+      "recall": 0.8698630136986302,
+      "support": 292.0
+    },
+    "Natural Sciences": {
+      "f1-score": 0.731610337972167,
+      "precision": 0.6917293233082706,
+      "recall": 0.7763713080168776,
+      "support": 237.0
+    },
+    "Social Sciences": {
+      "f1-score": 0.7953020134228188,
+      "precision": 0.7873754152823921,
+      "recall": 0.8033898305084746,
+      "support": 295.0
+    },
+    "accuracy": 0.812,
+    "macro avg": {
+      "f1-score": 0.809598814807324,
+      "precision": 0.8120659702471688,
+      "recall": 0.8088839057043252,
+      "support": 1500.0
+    },
+    "weighted avg": {
+      "f1-score": 0.8129658609678403,
+      "precision": 0.8155705822436659,
+      "recall": 0.812,
+      "support": 1500.0
+    }
+  },
+  "cohen_kappa_scored": 0.7735384799881362,
+  "confusion_matrix": {
+    "columns": [
+      "Agricultural and Veterinary Sciences",
+      "Engineering and Technology",
+      "Humanities and the Arts",
+      "Medical and Health Sciences",
+      "Natural Sciences",
+      "Social Sciences"
+    ],
+    "matrix": [
+      [
+        155,
+        4,
+        1,
+        7,
+        36,
+        8
+      ],
+      [
+        10,
+        215,
+        2,
+        2,
+        19,
+        10
+      ],
+      [
+        2,
+        1,
+        173,
+        1,
+        3,
+        27
+      ],
+      [
+        8,
+        1,
+        3,
+        254,
+        15,
+        11
+      ],
+      [
+        10,
+        21,
+        1,
+        13,
+        184,
+        8
+      ],
+      [
+        7,
+        5,
+        34,
+        3,
+        9,
+        237
+      ]
+    ],
+    "rows": [
+      "Agricultural and Veterinary Sciences",
+      "Engineering and Technology",
+      "Humanities and the Arts",
+      "Medical and Health Sciences",
+      "Natural Sciences",
+      "Social Sciences"
+    ]
+  },
+  "coverage": 1.0,
+  "generated_at": "2026-07-19T18:05:53.939674+00:00",
+  "macro_f1_all_with_abstentions": 0.809598814807324,
+  "macro_f1_scored": 0.809598814807324,
+  "mean_max_probability": 0.7910507917404175,
+  "median_max_probability": 0.8443766236305237,
+  "n_abstained": 0,
+  "n_scored": 1500,
+  "n_target": 1500,
+  "proposal_disagreement_diagnostic": {
+    "checkpoint_abstains": 0,
+    "checkpoint_matches_neither": 57,
+    "checkpoint_matches_openalex": 144,
+    "checkpoint_matches_panel": 297,
+    "n_panel_openalex_disagree": 498,
+    "reference_status": "provisional_unadjudicated_codex_panel"
+  },
+  "provenance": {
+    "checkpoint_path": "/opt/bibr-training-oecd-run/oecd_minilm_ablation_20260719/checkpoints/oecd_minilm_deepseek_teacher_20260719",
+    "encoder_name": "sentence-transformers/all-MiniLM-L6-v2",
+    "inference_config_sha256": "44c7c38bbebf60b20eaa75ce173267ade9814e6acdd2f2a0084b2609f70814ca",
+    "input_template": "title_abstract_v1",
+    "label_maps_sha256": "053d9d54ed59e5c7a8fb3ed80d4a6826180dfa5e15c4bbc190922c886b77a870",
+    "max_length": 256,
+    "model_sha256": "7a46c595cf3bb8e1eff39303786e8b7d16c95ea87f9f06b12d4a753f0be1e580",
+    "rows_path": "/opt/bibr-training-oecd-run/oecd_minilm_ablation_20260719/data/oecd_phaseA_panel_20260719.parquet",
+    "rows_sha256": "3554e33eea4680848d99268ed0745fcdbcb95f9b88cf2019f67e327adad1cd28",
+    "runtime": {
+      "cuda_device": "NVIDIA GeForce RTX 3090",
+      "device": "cuda",
+      "python": "3.12.12",
+      "sklearn": "1.8.0",
+      "torch": "2.11.0+cu130",
+      "transformers": "5.4.0"
+    },
+    "source_hashes": {
+      "scripts/diagnose_paper_classifier.py": "943f7d1e84469c0e2d1817168b607e50ba12db2d94a95774820fa0f04407e39e",
+      "src/bibr_training/models/oecd_multitask.py": "8ffeb67be8cac60b288270ef6b6a737e888b3437a399cf4e76471cdc7bd057bb",
+      "src/bibr_training/paper_classifier/transformer_dataset.py": "1214996cd87cfedbfd26e9642c53add97095ebd173a64bbbf1348577df7c204e"
+    },
+    "source_root": "/home/jakub/pco"
+  },
+  "quality_claim_allowed": false,
+  "reference_status": "provisional_unadjudicated_codex_panel"
+}

test_full_diagnostics.json

@@ -1,823 +0,0 @@
-{
-  "checkpoint": "checkpoints/paper_classifier_oecd_multitask_specter2_pco3",
-  "encoder": "allenai/specter2_base",
-  "l1": {
-    "classification_report": {
-      "Agricultural and Veterinary Sciences": {
-        "f1-score": 0.7533425541724297,
-        "precision": 0.7264374629519857,
-        "recall": 0.7823172677944462,
-        "support": 3133.0
-      },
-      "Engineering and Technology": {
-        "f1-score": 0.6882737515648908,
-        "precision": 0.7024417944349801,
-        "recall": 0.674665939460049,
-        "support": 3667.0
-      },
-      "Humanities and the Arts": {
-        "f1-score": 0.7535168195718654,
-        "precision": 0.7668845315904139,
-        "recall": 0.7406071535918245,
-        "support": 3327.0
-      },
-      "Medical and Health Sciences": {
-        "f1-score": 0.6884701978907454,
-        "precision": 0.6187433439829606,
-        "recall": 0.7759081196581197,
-        "support": 3744.0
-      },
-      "Natural Sciences": {
-        "f1-score": 0.556574047550845,
-        "precision": 0.6472351765489673,
-        "recall": 0.48819095477386937,
-        "support": 3980.0
-      },
-      "Social Sciences": {
-        "f1-score": 0.6421865715983363,
-        "precision": 0.6387706855791963,
-        "recall": 0.6456391875746714,
-        "support": 4185.0
-      },
-      "accuracy": 0.6779361045561808,
-      "macro avg": {
-        "f1-score": 0.6803939903915187,
-        "precision": 0.6834188325147507,
-        "recall": 0.6845547704754967,
-        "support": 22036.0
-      },
-      "weighted avg": {
-        "f1-score": 0.6748693256589501,
-        "precision": 0.6792990501864667,
-        "recall": 0.6779361045561808,
-        "support": 22036.0
-      }
-    },
-    "confidence_curve": [
-      {
-        "accuracy": 0.6779361045561808,
-        "coverage": 1.0,
-        "macro_f1": 0.6803939903915187,
-        "rows": 22036.0,
-        "threshold": 0.0
-      },
-      {
-        "accuracy": 0.6779361045561808,
-        "coverage": 1.0,
-        "macro_f1": 0.6803939903915187,
-        "rows": 22036.0,
-        "threshold": 0.05
-      },
-      {
-        "accuracy": 0.6779361045561808,
-        "coverage": 1.0,
-        "macro_f1": 0.6803939903915187,
-        "rows": 22036.0,
-        "threshold": 0.1
-      },
-      {
-        "accuracy": 0.6779361045561808,
-        "coverage": 1.0,
-        "macro_f1": 0.6803939903915187,
-        "rows": 22036.0,
-        "threshold": 0.15
-      },
-      {
-        "accuracy": 0.6779361045561808,
-        "coverage": 1.0,
-        "macro_f1": 0.6803939903915187,
-        "rows": 22036.0,
-        "threshold": 0.2
-      },
-      {
-        "accuracy": 0.6791010827040306,
-        "coverage": 0.9975494645126157,
-        "macro_f1": 0.6815535154674911,
-        "rows": 21982.0,
-        "threshold": 0.25
-      },
-      {
-        "accuracy": 0.681435371658733,
-        "coverage": 0.9914685060809584,
-        "macro_f1": 0.6839155666533182,
-        "rows": 21848.0,
-        "threshold": 0.3
-      },
-      {
-        "accuracy": 0.685217712859388,
-        "coverage": 0.9817571247050282,
-        "macro_f1": 0.6877510807032343,
-        "rows": 21634.0,
-        "threshold": 0.35
-      },
-      {
-        "accuracy": 0.6901283131965908,
-        "coverage": 0.9690506444000726,
-        "macro_f1": 0.6925843707449589,
-        "rows": 21354.0,
-        "threshold": 0.4
-      },
-      {
-        "accuracy": 0.6961428571428572,
-        "coverage": 0.9529860228716646,
-        "macro_f1": 0.6984668849345733,
-        "rows": 21000.0,
-        "threshold": 0.45
-      },
-      {
-        "accuracy": 0.7038267924343874,
-        "coverage": 0.9285260482846251,
-        "macro_f1": 0.7059520053072347,
-        "rows": 20461.0,
-        "threshold": 0.5
-      },
-      {
-        "accuracy": 0.717032967032967,
-        "coverage": 0.8754764930114358,
-        "macro_f1": 0.7187081937238283,
-        "rows": 19292.0,
-        "threshold": 0.55
-      },
-      {
-        "accuracy": 0.7385120350109409,
-        "coverage": 0.7880740606280632,
-        "macro_f1": 0.7391379430143319,
-        "rows": 17366.0,
-        "threshold": 0.6
-      },
-      {
-        "accuracy": 0.7534296913277805,
-        "coverage": 0.7409693229261209,
-        "macro_f1": 0.7531826505306123,
-        "rows": 16328.0,
-        "threshold": 0.65
-      },
-      {
-        "accuracy": 0.7668644395717786,
-        "coverage": 0.7036667271737157,
-        "macro_f1": 0.7657749465622468,
-        "rows": 15506.0,
-        "threshold": 0.7
-      },
-      {
-        "accuracy": 0.7807901907356948,
-        "coverage": 0.666182610274097,
-        "macro_f1": 0.7788190244613037,
-        "rows": 14680.0,
-        "threshold": 0.75
-      },
-      {
-        "accuracy": 0.7947712418300653,
-        "coverage": 0.6248865492829915,
-        "macro_f1": 0.7920474767627096,
-        "rows": 13770.0,
-        "threshold": 0.8
-      },
-      {
-        "accuracy": 0.8143037375808232,
-        "coverage": 0.5755127972408786,
-        "macro_f1": 0.8107728563245549,
-        "rows": 12682.0,
-        "threshold": 0.85
-      },
-      {
-        "accuracy": 0.8324778761061947,
-        "coverage": 0.5127972408785624,
-        "macro_f1": 0.8278814771173524,
-        "rows": 11300.0,
-        "threshold": 0.9
-      },
-      {
-        "accuracy": 0.8680169152014244,
-        "coverage": 0.4077872572154656,
-        "macro_f1": 0.86101771784818,
-        "rows": 8986.0,
-        "threshold": 0.95
-      }
-    ],
-    "confusion_matrix": {
-      "labels": [
-        "Agricultural and Veterinary Sciences",
-        "Engineering and Technology",
-        "Humanities and the Arts",
-        "Medical and Health Sciences",
-        "Natural Sciences",
-        "Social Sciences"
-      ],
-      "matrix": [
-        [
-          2451,
-          111,
-          19,
-          252,
-          146,
-          154
-        ],
-        [
-          138,
-          2474,
-          30,
-          428,
-          385,
-          212
-        ],
-        [
-          47,
-          53,
-          2464,
-          108,
-          94,
-          561
-        ],
-        [
-          190,
-          148,
-          57,
-          2905,
-          178,
-          266
-        ],
-        [
-          363,
-          531,
-          121,
-          687,
-          1943,
-          335
-        ],
-        [
-          185,
-          205,
-          522,
-          315,
-          256,
-          2702
-        ]
-      ]
-    }
-  },
-  "l2": {
-    "classification_report": {
-      "Agricultural Biotechnology": {
-        "f1-score": 0.38461538461538464,
-        "precision": 0.3125,
-        "recall": 0.5,
-        "support": 10.0
-      },
-      "Agriculture, Forestry, and Fisheries": {
-        "f1-score": 0.7068877115023073,
-        "precision": 0.639258114374034,
-        "recall": 0.790519877675841,
-        "support": 2616.0
-      },
-      "Animal and Dairy Science": {
-        "f1-score": 0.6613924050632911,
-        "precision": 0.7545126353790613,
-        "recall": 0.5887323943661972,
-        "support": 355.0
-      },
-      "Arts (arts, history of arts, performing arts, music)": {
-        "f1-score": 0.3446054750402576,
-        "precision": 0.4163424124513619,
-        "recall": 0.29395604395604397,
-        "support": 364.0
-      },
-      "Basic Medicine": {
-        "f1-score": 0.18,
-        "precision": 0.27586206896551724,
-        "recall": 0.13358070500927643,
-        "support": 539.0
-      },
-      "Biological Sciences": {
-        "f1-score": 0.3302752293577982,
-        "precision": 0.4722222222222222,
-        "recall": 0.25394190871369293,
-        "support": 1205.0
-      },
-      "Chemical Engineering": {
-        "f1-score": 0.3176470588235294,
-        "precision": 0.32238805970149254,
-        "recall": 0.3130434782608696,
-        "support": 345.0
-      },
-      "Chemical Sciences": {
-        "f1-score": 0.2659846547314578,
-        "precision": 0.3170731707317073,
-        "recall": 0.2290748898678414,
-        "support": 227.0
-      },
-      "Civil Engineering": {
-        "f1-score": 0.4732510288065844,
-        "precision": 0.4606141522029373,
-        "recall": 0.48660084626234135,
-        "support": 709.0
-      },
-      "Clinical Medicine": {
-        "f1-score": 0.5436472076631123,
-        "precision": 0.4372093023255814,
-        "recall": 0.7185857620640229,
-        "support": 2093.0
-      },
-      "Computer and Information Sciences": {
-        "f1-score": 0.5597516813243663,
-        "precision": 0.5810955961331902,
-        "recall": 0.5399201596806387,
-        "support": 1002.0
-      },
-      "Earth and Related Environmental Sciences": {
-        "f1-score": 0.4582441113490364,
-        "precision": 0.4908256880733945,
-        "recall": 0.42971887550200805,
-        "support": 747.0
-      },
-      "Economics and Business": {
-        "f1-score": 0.5002578648788035,
-        "precision": 0.5300546448087432,
-        "recall": 0.4736328125,
-        "support": 1024.0
-      },
-      "Education": {
-        "f1-score": 0.45214770158251694,
-        "precision": 0.40431266846361186,
-        "recall": 0.5128205128205128,
-        "support": 585.0
-      },
-      "Electrical Engineering, Electronic Engineering, Information Engineering": {
-        "f1-score": 0.41893491124260357,
-        "precision": 0.5071633237822349,
-        "recall": 0.35685483870967744,
-        "support": 496.0
-      },
-      "Environmental Biotechnology": {
-        "f1-score": 0.2857142857142857,
-        "precision": 0.3333333333333333,
-        "recall": 0.25,
-        "support": 12.0
-      },
-      "Environmental Engineering": {
-        "f1-score": 0.3543307086614173,
-        "precision": 0.3284671532846715,
-        "recall": 0.38461538461538464,
-        "support": 351.0
-      },
-      "Health Sciences": {
-        "f1-score": 0.35502958579881655,
-        "precision": 0.384204909284952,
-        "recall": 0.32997250229147573,
-        "support": 1091.0
-      },
-      "History and Archaeology": {
-        "f1-score": 0.6276005547850209,
-        "precision": 0.6956187548039969,
-        "recall": 0.5716993051168667,
-        "support": 1583.0
-      },
-      "Industrial Biotechnology": {
-        "f1-score": 0.3157894736842105,
-        "precision": 0.34285714285714286,
-        "recall": 0.2926829268292683,
-        "support": 41.0
-      },
-      "Languages and Literature": {
-        "f1-score": 0.5564835164835165,
-        "precision": 0.4964705882352941,
-        "recall": 0.633,
-        "support": 1000.0
-      },
-      "Law": {
-        "f1-score": 0.4176334106728538,
-        "precision": 0.3813559322033898,
-        "recall": 0.46153846153846156,
-        "support": 195.0
-      },
-      "Materials Engineering": {
-        "f1-score": 0.3352941176470588,
-        "precision": 0.4253731343283582,
-        "recall": 0.2766990291262136,
-        "support": 206.0
-      },
-      "Mathematics": {
-        "f1-score": 0.578125,
-        "precision": 0.6297872340425532,
-        "recall": 0.5342960288808665,
-        "support": 277.0
-      },
-      "Mechanical Engineering": {
-        "f1-score": 0.48933601609657945,
-        "precision": 0.5033112582781457,
-        "recall": 0.47611589663273296,
-        "support": 1277.0
-      },
-      "Media and Communications": {
-        "f1-score": 0.2660098522167488,
-        "precision": 0.20149253731343283,
-        "recall": 0.391304347826087,
-        "support": 69.0
-      },
-      "Medical Biotechnology": {
-        "f1-score": 0.07142857142857142,
-        "precision": 0.05714285714285714,
-        "recall": 0.09523809523809523,
-        "support": 21.0
-      },
-      "Medical Engineering": {
-        "f1-score": 0.18235294117647058,
-        "precision": 0.2605042016806723,
-        "recall": 0.14027149321266968,
-        "support": 221.0
-      },
-      "Nano-technology": {
-        "f1-score": 0.0,
-        "precision": 0.0,
-        "recall": 0.0,
-        "support": 9.0
-      },
-      "Philosophy, Ethics and Religion": {
-        "f1-score": 0.3373493975903614,
-        "precision": 0.34332425068119893,
-        "recall": 0.33157894736842103,
-        "support": 380.0
-      },
-      "Physical Sciences": {
-        "f1-score": 0.5873465533522191,
-        "precision": 0.5791433891992551,
-        "recall": 0.5957854406130269,
-        "support": 522.0
-      },
-      "Political Science": {
-        "f1-score": 0.36220472440944884,
-        "precision": 0.3492407809110629,
-        "recall": 0.37616822429906543,
-        "support": 428.0
-      },
-      "Psychology and Cognitive Sciences": {
-        "f1-score": 0.36259977194982895,
-        "precision": 0.4097938144329897,
-        "recall": 0.32515337423312884,
-        "support": 489.0
-      },
-      "Social and Economic Geography": {
-        "f1-score": 0.15966386554621848,
-        "precision": 0.15702479338842976,
-        "recall": 0.1623931623931624,
-        "support": 117.0
-      },
-      "Sociology": {
-        "f1-score": 0.3387308533916849,
-        "precision": 0.3843098311817279,
-        "recall": 0.3028169014084507,
-        "support": 1278.0
-      },
-      "Veterinary Science": {
-        "f1-score": 0.46387832699619774,
-        "precision": 0.5495495495495496,
-        "recall": 0.40131578947368424,
-        "support": 152.0
-      },
-      "accuracy": 0.4916954075149755,
-      "macro avg": {
-        "f1-score": 0.3901262209328489,
-        "precision": 0.4092705418263362,
-        "recall": 0.387600789346834,
-        "support": 22036.0
-      },
-      "weighted avg": {
-        "f1-score": 0.48113766501448607,
-        "precision": 0.4898729984647697,
-        "recall": 0.4916954075149755,
-        "support": 22036.0
-      }
-    },
-    "error_within_l1_share": 0.35943219355414696,
-    "l1_via_l2_rollup_accuracy": 0.6743964421855146,
-    "macro_f1": 0.3901262209328489,
-    "micro_f1": 0.4916954075149755,
-    "precision_coverage": [
-      {
-        "accuracy": 0.4916954075149755,
-        "coverage": 1.0,
-        "rows": 22036,
-        "threshold": 0.0
-      },
-      {
-        "accuracy": 0.4916954075149755,
-        "coverage": 1.0,
-        "rows": 22036,
-        "threshold": 0.05
-      },
-      {
-        "accuracy": 0.49169465371698284,
-        "coverage": 0.9999092394263932,
-        "rows": 22034,
-        "threshold": 0.1
-      },
-      {
-        "accuracy": 0.4940381013294349,
-        "coverage": 0.9933290978398983,
-        "rows": 21889,
-        "threshold": 0.15
-      },
-      {
-        "accuracy": 0.4992124525155193,
-        "coverage": 0.9795788709384643,
-        "rows": 21586,
-        "threshold": 0.2
-      },
-      {
-        "accuracy": 0.5059084893174514,
-        "coverage": 0.9600653476129969,
-        "rows": 21156,
-        "threshold": 0.25
-      },
-      {
-        "accuracy": 0.5221376417585832,
-        "coverage": 0.8763387184607007,
-        "rows": 19311,
-        "threshold": 0.3
-      },
-      {
-        "accuracy": 0.5418234930621875,
-        "coverage": 0.8078144853875476,
-        "rows": 17801,
-        "threshold": 0.35
-      },
-      {
-        "accuracy": 0.5582341329185859,
-        "coverage": 0.7586222544926484,
-        "rows": 16717,
-        "threshold": 0.4
-      },
-      {
-        "accuracy": 0.5756701295370014,
-        "coverage": 0.7076601924124161,
-        "rows": 15594,
-        "threshold": 0.45
-      },
-      {
-        "accuracy": 0.5957490999723069,
-        "coverage": 0.6554728625884916,
-        "rows": 14444,
-        "threshold": 0.5
-      },
-      {
-        "accuracy": 0.6180163922099406,
-        "coverage": 0.6035124341985841,
-        "rows": 13299,
-        "threshold": 0.55
-      },
-      {
-        "accuracy": 0.6385780118499013,
-        "coverage": 0.5514612452350699,
-        "rows": 12152,
-        "threshold": 0.6
-      },
-      {
-        "accuracy": 0.6585255948089401,
-        "coverage": 0.5035396623706662,
-        "rows": 11096,
-        "threshold": 0.65
-      },
-      {
-        "accuracy": 0.6845517103819687,
-        "coverage": 0.45502813577781814,
-        "rows": 10027,
-        "threshold": 0.7
-      },
-      {
-        "accuracy": 0.7090543708831082,
-        "coverage": 0.4064712288981666,
-        "rows": 8957,
-        "threshold": 0.75
-      },
-      {
-        "accuracy": 0.7361146335277707,
-        "coverage": 0.35786894173171174,
-        "rows": 7886,
-        "threshold": 0.8
-      },
-      {
-        "accuracy": 0.7661374091111441,
-        "coverage": 0.3058177527681975,
-        "rows": 6739,
-        "threshold": 0.85
-      },
-      {
-        "accuracy": 0.8052888969389603,
-        "coverage": 0.25054456344164094,
-        "rows": 5521,
-        "threshold": 0.9
-      },
-      {
-        "accuracy": 0.8496835443037974,
-        "coverage": 0.17208204755854056,
-        "rows": 3792,
-        "threshold": 0.95
-      }
-    ],
-    "rows": 22036,
-    "top1_accuracy": 0.4916954075149755,
-    "top2_accuracy": 0.654202214557996,
-    "top3_accuracy": 0.7418769286621891
-  },
-  "paper_type": {
-    "classification_report": {
-      "accuracy": 0.9338009644892591,
-      "case-study": {
-        "f1-score": 0.9385474860335196,
-        "precision": 0.9545454545454546,
-        "recall": 0.9230769230769231,
-        "support": 364.0
-      },
-      "commentary": {
-        "f1-score": 0.9177215189873418,
-        "precision": 0.9206349206349206,
-        "recall": 0.9148264984227129,
-        "support": 317.0
-      },
-      "corrigendum": {
-        "f1-score": 0.0,
-        "precision": 0.0,
-        "recall": 0.0,
-        "support": 0.0
-      },
-      "empirical": {
-        "f1-score": 0.9021739130434783,
-        "precision": 0.8829787234042553,
-        "recall": 0.9222222222222223,
-        "support": 270.0
-      },
-      "erratum": {
-        "f1-score": 0.9809523809523809,
-        "precision": 0.987220447284345,
-        "recall": 0.9747634069400631,
-        "support": 317.0
-      },
-      "macro avg": {
-        "f1-score": 0.8185795854622895,
-        "precision": 0.8173552652425309,
-        "recall": 0.8202954373263103,
-        "support": 2281.0
-      },
-      "meta-analysis": {
-        "f1-score": 0.928695652173913,
-        "precision": 0.898989898989899,
-        "recall": 0.960431654676259,
-        "support": 278.0
-      },
-      "retraction": {
-        "f1-score": 0.9935064935064936,
-        "precision": 0.9935064935064936,
-        "recall": 0.9935064935064936,
-        "support": 308.0
-      },
-      "review": {
-        "f1-score": 0.8870392390011891,
-        "precision": 0.9009661835748792,
-        "recall": 0.8735362997658079,
-        "support": 427.0
-      },
-      "weighted avg": {
-        "f1-score": 0.9338189452330943,
-        "precision": 0.9343624308240979,
-        "recall": 0.9338009644892591,
-        "support": 2281.0
-      }
-    },
-    "learnable_classes": [
-      "case-study",
-      "commentary",
-      "empirical",
-      "erratum",
-      "meta-analysis",
-      "retraction",
-      "review"
-    ],
-    "learnable_macro_f1": 0.9355195262426167,
-    "macro_f1": 0.9355195262426167,
-    "micro_f1": 0.9338009644892591,
-    "precision_coverage": [
-      {
-        "accuracy": 0.9338009644892591,
-        "coverage": 1.0,
-        "rows": 2281,
-        "threshold": 0.0
-      },
-      {
-        "accuracy": 0.9338009644892591,
-        "coverage": 1.0,
-        "rows": 2281,
-        "threshold": 0.05
-      },
-      {
-        "accuracy": 0.9338009644892591,
-        "coverage": 1.0,
-        "rows": 2281,
-        "threshold": 0.1
-      },
-      {
-        "accuracy": 0.9338009644892591,
-        "coverage": 1.0,
-        "rows": 2281,
-        "threshold": 0.15
-      },
-      {
-        "accuracy": 0.9338009644892591,
-        "coverage": 1.0,
-        "rows": 2281,
-        "threshold": 0.2
-      },
-      {
-        "accuracy": 0.9338009644892591,
-        "coverage": 1.0,
-        "rows": 2281,
-        "threshold": 0.25
-      },
-      {
-        "accuracy": 0.9346204475647214,
-        "coverage": 0.9991231915826392,
-        "rows": 2279,
-        "threshold": 0.3
-      },
-      {
-        "accuracy": 0.9354413702239789,
-        "coverage": 0.9982463831652784,
-        "rows": 2277,
-        "threshold": 0.35
-      },
-      {
-        "accuracy": 0.9370044052863437,
-        "coverage": 0.9951775537045155,
-        "rows": 2270,
-        "threshold": 0.4
-      },
-      {
-        "accuracy": 0.9419583517944173,
-        "coverage": 0.9894782989916703,
-        "rows": 2257,
-        "threshold": 0.45
-      },
-      {
-        "accuracy": 0.9431363838294091,
-        "coverage": 0.9868478737395879,
-        "rows": 2251,
-        "threshold": 0.5
-      },
-      {
-        "accuracy": 0.944692239072257,
-        "coverage": 0.9829022358614643,
-        "rows": 2242,
-        "threshold": 0.55
-      },
-      {
-        "accuracy": 0.9466128308658591,
-        "coverage": 0.977202981148619,
-        "rows": 2229,
-        "threshold": 0.6
-      },
-      {
-        "accuracy": 0.9495040577096483,
-        "coverage": 0.9723805348531346,
-        "rows": 2218,
-        "threshold": 0.65
-      },
-      {
-        "accuracy": 0.9519492293744334,
-        "coverage": 0.9671196843489698,
-        "rows": 2206,
-        "threshold": 0.7
-      },
-      {
-        "accuracy": 0.9539443684450525,
-        "coverage": 0.9614204296361245,
-        "rows": 2193,
-        "threshold": 0.75
-      },
-      {
-        "accuracy": 0.9567617295308187,
-        "coverage": 0.9530907496711969,
-        "rows": 2174,
-        "threshold": 0.8
-      },
-      {
-        "accuracy": 0.9618249534450651,
-        "coverage": 0.9416922402455064,
-        "rows": 2148,
-        "threshold": 0.85
-      },
-      {
-        "accuracy": 0.9635933806146572,
-        "coverage": 0.927224901359053,
-        "rows": 2115,
-        "threshold": 0.9
-      },
-      {
-        "accuracy": 0.9766666666666667,
-        "coverage": 0.789127575624726,
-        "rows": 1800,
-        "threshold": 0.95
-      }
-    ],
-    "rows": 2281
-  },
-  "test_rows": 23813
-}

test_l1_diagnostics.json

@@ -1,96 +1,96 @@
 {
   "classification_report": {
     "Agricultural and Veterinary Sciences": {
-      "f1-score": 0.7531869144524651,
-      "precision": 0.7258732978093546,
-      "recall": 0.7826364506862432,
-      "support": 3133.0
+      "f1-score": 0.7455357142857143,
+      "precision": 0.7636031092821216,
+      "recall": 0.7283035324901875,
+      "support": 2293.0
     },
     "Engineering and Technology": {
-      "f1-score": 0.688082325128633,
-      "precision": 0.7020431328036323,
-      "recall": 0.674665939460049,
-      "support": 3667.0
+      "f1-score": 0.804904214559387,
+      "precision": 0.8033037626185378,
+      "recall": 0.8065110565110565,
+      "support": 3256.0
     },
     "Humanities and the Arts": {
-      "f1-score": 0.7535921736472027,
-      "precision": 0.7667185069984448,
-      "recall": 0.740907724676886,
-      "support": 3327.0
+      "f1-score": 0.7972758229284903,
+      "precision": 0.797819173103135,
+      "recall": 0.7967332123411979,
+      "support": 2204.0
     },
     "Medical and Health Sciences": {
-      "f1-score": 0.6885440113730601,
-      "precision": 0.6186927826272088,
-      "recall": 0.7761752136752137,
-      "support": 3744.0
+      "f1-score": 0.8291206296352354,
+      "precision": 0.8154212563262876,
+      "recall": 0.8432881773399015,
+      "support": 3248.0
     },
     "Natural Sciences": {
-      "f1-score": 0.556907659269864,
-      "precision": 0.6472545757071547,
-      "recall": 0.4886934673366834,
-      "support": 3980.0
+      "f1-score": 0.6668681069647983,
+      "precision": 0.6720462850182704,
+      "recall": 0.6617691154422789,
+      "support": 3335.0
     },
     "Social Sciences": {
-      "f1-score": 0.6419900023803856,
-      "precision": 0.6395541854398862,
-      "recall": 0.6444444444444445,
-      "support": 4185.0
+      "f1-score": 0.7857319791924696,
+      "precision": 0.7822441430332923,
+      "recall": 0.7892510574769843,
+      "support": 4019.0
     },
-    "accuracy": 0.6779361045561808,
+    "accuracy": 0.7719967311359303,
     "macro avg": {
-      "f1-score": 0.6803838477086018,
-      "precision": 0.683356080230947,
-      "recall": 0.6845872067132533,
-      "support": 22036.0
+      "f1-score": 0.7715727445943492,
+      "precision": 0.7724062882302741,
+      "recall": 0.7709760252669344,
+      "support": 18355.0
     },
     "weighted avg": {
-      "f1-score": 0.6748621833993114,
-      "precision": 0.6792711444158704,
-      "recall": 0.6779361045561808,
-      "support": 22036.0
+      "f1-score": 0.7715784906213174,
+      "precision": 0.7713698855964962,
+      "recall": 0.7719967311359303,
+      "support": 18355.0
     }
   },
   "confidence_curve": [
     {
-      "accuracy": 0.6779361045561808,
+      "accuracy": 0.7719967311359303,
       "coverage": 1.0,
-      "macro_f1": 0.6803838477086018,
-      "rows": 22036.0,
+      "macro_f1": 0.7715727445943492,
+      "rows": 18355.0,
       "threshold": 0.0
     },
     {
-      "accuracy": 0.7036005667106356,
-      "coverage": 0.9288890905790524,
-      "macro_f1": 0.7057025711643554,
-      "rows": 20469.0,
+      "accuracy": 0.7976120456416892,
+      "coverage": 0.9262871152274584,
+      "macro_f1": 0.7967970019133345,
+      "rows": 17002.0,
       "threshold": 0.5
     },
     {
-      "accuracy": 0.7666365746711374,
-      "coverage": 0.7037574877473226,
-      "macro_f1": 0.7655612090310461,
-      "rows": 15508.0,
+      "accuracy": 0.8593491035106222,
+      "coverage": 0.7231816943612095,
+      "macro_f1": 0.8561099696894608,
+      "rows": 13274.0,
       "threshold": 0.7
     },
     {
-      "accuracy": 0.7944121915820029,
-      "coverage": 0.6253403521510256,
-      "macro_f1": 0.7916989698694485,
-      "rows": 13780.0,
+      "accuracy": 0.8935934177683277,
+      "coverage": 0.5893217107055299,
+      "macro_f1": 0.8880868103664564,
+      "rows": 10817.0,
       "threshold": 0.8
     },
     {
-      "accuracy": 0.8325223392019818,
-      "coverage": 0.5129333817389726,
-      "macro_f1": 0.8279376195792615,
-      "rows": 11303.0,
+      "accuracy": 0.9318342525809645,
+      "coverage": 0.38523563061836014,
+      "macro_f1": 0.9213157044536953,
+      "rows": 7071.0,
       "threshold": 0.9
     },
     {
-      "accuracy": 0.8683214603739983,
-      "coverage": 0.40769649664185875,
-      "macro_f1": 0.8613750643193868,
-      "rows": 8984.0,
+      "accuracy": 0.961308750973773,
+      "coverage": 0.2098065922092073,
+      "macro_f1": 0.9281730445872894,
+      "rows": 3851.0,
       "threshold": 0.95
     }
   ],
@@ -105,52 +105,52 @@
     ],
     "matrix": [
       [
-        2452,
-        111,
-        19,
-        252,
-        145,
-        154
+        1670,
+        53,
+        12,
+        143,
+        307,
+        108
       ],
       [
-        138,
-        2474,
-        30,
-        430,
-        385,
-        210
+        80,
+        2626,
+        12,
+        61,
+        351,
+        126
       ],
       [
-        48,
-        53,
-        2465,
-        107,
-        95,
-        559
+        19,
+        21,
+        1756,
+        46,
+        47,
+        315
       ],
       [
-        189,
-        148,
-        57,
-        2906,
-        179,
-        265
+        98,
+        54,
+        30,
+        2739,
+        201,
+        126
       ],
       [
-        364,
-        531,
-        121,
-        687,
-        1945,
-        332
+        225,
+        391,
+        55,
+        249,
+        2207,
+        208
       ],
       [
-        187,
-        207,
-        523,
-        315,
-        256,
-        2697
+        95,
+        124,
+        336,
+        121,
+        171,
+        3172
       ]
     ]
   }

test_metrics.json

@@ -1,19 +1,19 @@
 {
-  "epoch": 6.0,
-  "test_combined_macro_f1": 0.6687502230338227,
-  "test_l1_eval_rows": 22036.0,
-  "test_l1_macro_f1": 0.6803838477086018,
-  "test_l1_micro_f1": 0.6779361045561808,
-  "test_l2_constrained_macro_f1": 0.38956763499809743,
-  "test_l2_constrained_micro_f1": 0.48938101288800145,
-  "test_l2_eval_rows": 22036.0,
-  "test_l2_macro_f1": 0.39032050340786256,
-  "test_l2_micro_f1": 0.49210383009620623,
-  "test_loss": 2.819976806640625,
-  "test_paper_type_eval_rows": 2281.0,
-  "test_paper_type_macro_f1": 0.9355463179850035,
-  "test_paper_type_micro_f1": 0.9338009644892591,
-  "test_runtime": 94.0133,
-  "test_samples_per_second": 253.294,
-  "test_steps_per_second": 15.838
+  "epoch": 5.0,
+  "test_combined_macro_f1": 0.6698617929411549,
+  "test_l1_eval_rows": 18355.0,
+  "test_l1_macro_f1": 0.7715727445943492,
+  "test_l1_micro_f1": 0.7719967311359303,
+  "test_l2_constrained_macro_f1": 0.4661771349151901,
+  "test_l2_constrained_micro_f1": 0.5740845070422536,
+  "test_l2_eval_rows": 15975.0,
+  "test_l2_macro_f1": 0.4719512458194132,
+  "test_l2_micro_f1": 0.5799687010954616,
+  "test_loss": 2.2446630001068115,
+  "test_paper_type_eval_rows": 548.0,
+  "test_paper_type_macro_f1": 0.7660613884097018,
+  "test_paper_type_micro_f1": 0.8959854014598541,
+  "test_runtime": 55.1841,
+  "test_samples_per_second": 332.614,
+  "test_steps_per_second": 5.201
 }

tokenizer_config.json

@@ -5,12 +5,19 @@
   "do_lower_case": true,
   "is_local": false,
   "mask_token": "[MASK]",
-  "model_max_length": 1000000000000000019884624838656,
+  "max_length": 128,
+  "model_max_length": 512,
   "never_split": null,
+  "pad_to_multiple_of": null,
   "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
   "sep_token": "[SEP]",
+  "stride": 0,
   "strip_accents": null,
   "tokenize_chinese_chars": true,
   "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
   "unk_token": "[UNK]"
 }