lukeingawesome commited on
Commit
f96e85a
·
verified ·
1 Parent(s): 6410e2a

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +26 -0
README.md CHANGED
@@ -107,6 +107,21 @@ Each level (`leaf` = 137 labels, `upper` = 38 container groups, `anatomy` = 10 s
107
  max-over-children roll-up of the leaf predictions (`model.aggregate_hierarchy(...)`). Coarser
108
  levels are easier to match, so upper/anatomy F1 are typically higher than leaf.
109
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
  ## Inputs & conventions
111
 
112
  - Input is the **findings** text (the model was trained on CT-RATE findings + their refined
@@ -143,6 +158,17 @@ rows are the hierarchy roll-up.
143
  | upper (≥30 positives) | 19 | 0.837 | — |
144
  | anatomy | 10 | 0.869 | — |
145
 
 
 
 
 
 
 
 
 
 
 
 
146
  Leaf macro-AUC barely moves public→private (**0.989 → 0.972**), i.e. label ranking transfers to
147
  the unseen set; the F1 gap is mostly threshold / labeling-convention, not a domain failure.
148
  Separately, a radiologist spot-checked **966** reports of the public test labels (857 fully
 
107
  max-over-children roll-up of the leaf predictions (`model.aggregate_hierarchy(...)`). Coarser
108
  levels are easier to match, so upper/anatomy F1 are typically higher than leaf.
109
 
110
+ ### Per-label best F1 (threshold tuning)
111
+
112
+ The default decision threshold is a single global value, but the F1-optimal threshold differs per
113
+ label. To get the **best achievable F1 per label** (and the threshold that achieves it) against a
114
+ ground-truth label set:
115
+
116
+ ```python
117
+ # gt: a DataFrame with the 137 label columns (ternary; positive == 1), or a binary array
118
+ res = model.per_label_best_f1(reports, gt, tokenizer=tok, level="leaf", min_pos=30)
119
+ res["macro_best_f1_min_pos"] # macro best-F1 over labels with >= min_pos positives
120
+ res["per_label"]["Pleural effusion"] # {'best_f1':.., 'best_threshold':.., 'n_pos':..}
121
+ ```
122
+
123
+ Per-label threshold tuning lifts macro-F1 by ~4–6 points over the fixed-0.5 threshold (see below).
124
+
125
  ## Inputs & conventions
126
 
127
  - Input is the **findings** text (the model was trained on CT-RATE findings + their refined
 
158
  | upper (≥30 positives) | 19 | 0.837 | — |
159
  | anatomy | 10 | 0.869 | — |
160
 
161
+ **Per-label best F1** (threshold swept per label to maximize F1; macro over leaf labels with ≥30
162
+ positives, via `model.per_label_best_f1`):
163
+
164
+ | Eval set | macro best-F1 (≥30) | macro-F1 @0.5 (≥30) | macro best-F1 (all evaluated) |
165
+ |---|--:|--:|--:|
166
+ | CT-RATE public | **0.907** | 0.866 | 0.844 |
167
+ | Private | **0.820** | 0.761 | 0.795 |
168
+
169
+ F1-optimal thresholds vary widely by label (~0.04–0.75), so per-label tuning recovers ~4–6 macro-F1
170
+ points over a single global threshold.
171
+
172
  Leaf macro-AUC barely moves public→private (**0.989 → 0.972**), i.e. label ranking transfers to
173
  the unseen set; the F1 gap is mostly threshold / labeling-convention, not a domain failure.
174
  Separately, a radiologist spot-checked **966** reports of the public test labels (857 fully