AliceYin commited on
Commit
3074f27
·
verified ·
1 Parent(s): a679dc5

Improve public model card for launch

Browse files
Files changed (1) hide show
  1. README.md +80 -50
README.md CHANGED
@@ -1,8 +1,12 @@
1
  ---
2
  license: apache-2.0
 
 
3
  base_model: FacebookAI/roberta-large
4
  datasets:
5
  - google-research-datasets/go_emotions
 
 
6
  language:
7
  - en
8
  tags:
@@ -12,87 +16,113 @@ tags:
12
  - roberta
13
  - focal-loss
14
  - threshold-optimization
15
- metrics:
16
- - f1
17
- pipeline_tag: text-classification
18
- library_name: transformers
19
  ---
20
 
21
- # GoEmotions RoBERTa Large Focal SOTA
22
 
23
- RoBERTa-large multi-label classifier fine-tuned on the public GoEmotions
24
- `simplified` split. The model predicts 28 labels: 27 fine-grained emotions plus
25
- `neutral`.
26
 
27
- ## Results
 
 
 
 
 
28
 
29
- The selected policy is coordinate threshold search optimized on validation
30
- macro-F1.
 
 
31
 
32
- | Split | Macro F1 | Micro F1 | Samples F1 | Hamming Loss |
 
 
33
  | --- | ---: | ---: | ---: | ---: |
34
- | Validation | 0.565864 | 0.596637 | 0.605125 | 0.034424 |
35
- | Test | 0.533020 | 0.576651 | 0.585942 | 0.035820 |
36
 
37
- The per-label threshold candidate reached test macro-F1 `0.534994`, but the
38
- validation-selected policy is the coordinate threshold policy above.
39
 
40
- ## Training
 
 
 
 
 
41
 
42
- - Base model: `FacebookAI/roberta-large`
43
- - Dataset: `google-research-datasets/go_emotions`, `simplified`
44
- - Loss: focal loss, alpha `0.38`, gamma `2.8`
45
- - Epochs: `4`
46
- - Learning rate: `1e-5`
47
- - Batch: `2`, gradient accumulation `16`
48
- - Mixed precision: disabled for stability on Kaggle T4
49
- - Seed: `42`
50
 
51
- Training provenance: Kaggle kernel
52
- `kevin250304/goemotions-roberta-large-focal-sweep`.
 
53
 
54
- ## Usage
55
 
56
  ```python
57
  import json
58
- import numpy as np
59
  import torch
 
60
  from transformers import AutoModelForSequenceClassification, AutoTokenizer
61
 
62
  repo_id = "AliceYin/goemotions-roberta-large-focal-sota"
 
63
  tokenizer = AutoTokenizer.from_pretrained(repo_id)
64
  model = AutoModelForSequenceClassification.from_pretrained(repo_id)
 
 
 
 
65
 
66
- labels = json.load(open("labels.json"))["label_names"]
67
- thresholds = json.load(open("thresholds.json"))["coordinate"]
68
- threshold_array = np.array([thresholds[label] for label in labels])
69
 
70
- text = "I can't believe this worked so well."
71
- inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
72
  with torch.no_grad():
73
- probs = torch.sigmoid(model(**inputs).logits).cpu().numpy()[0]
74
 
75
  predicted = [
76
- label for label, prob, threshold in zip(labels, probs, threshold_array)
 
77
  if prob >= threshold
78
  ]
79
  print(predicted)
80
  ```
81
 
82
- For remote loading of thresholds and labels, download `labels.json` and
83
- `thresholds.json` from this repository alongside the model files.
84
-
85
- ## Files
86
 
87
- - `model.safetensors`: fine-tuned RoBERTa-large weights
88
- - `config.json`, `tokenizer.json`, `tokenizer_config.json`: Transformers files
89
- - `thresholds.json`: fixed, global, per-label, and coordinate thresholds
90
- - `labels.json`: label names and neutral index
91
- - `metrics.json`: full validation/test metrics
92
- - `kaggle-run.log`: training log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93
 
94
- ## Limitations
95
 
96
- This model is trained on Reddit comments and inherits the dataset's domain,
97
- annotation, and class imbalance limitations. Validate before using in
98
- high-stakes settings.
 
1
  ---
2
  license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: text-classification
5
  base_model: FacebookAI/roberta-large
6
  datasets:
7
  - google-research-datasets/go_emotions
8
+ metrics:
9
+ - f1
10
  language:
11
  - en
12
  tags:
 
16
  - roberta
17
  - focal-loss
18
  - threshold-optimization
19
+ - sota-level
 
 
 
20
  ---
21
 
22
+ # GoEmotions RoBERTa-large Focal SOTA-Level Classifier
23
 
24
+ This model is a RoBERTa-large multi-label emotion classifier trained on the
25
+ public GoEmotions simplified split. It predicts 27 fine-grained emotions plus
26
+ `neutral` from English Reddit-style text.
27
 
28
+ The run uses focal loss for label imbalance and validation-tuned coordinate
29
+ thresholds for multi-label decisions. It is a strong public-reference result:
30
+ the validation-selected policy reached test macro-F1 0.5330, while the strongest
31
+ public model card found during this iteration reported test macro-F1 0.519.
32
+
33
+ ## Links
34
 
35
+ - Kaggle model artifact: https://www.kaggle.com/models/kevin250304/goemotions-roberta-large-focal-sota/Transformers/roberta-large-focal-seed42
36
+ - Training kernel: https://www.kaggle.com/code/kevin250304/goemotions-roberta-large-focal-sweep
37
+ - Dataset: https://huggingface.co/datasets/google-research-datasets/go_emotions
38
+ - GoEmotions paper: https://aclanthology.org/2020.acl-main.372/
39
 
40
+ ## Results
41
+
42
+ | Split | Macro-F1 | Micro-F1 | Samples-F1 | Subset accuracy |
43
  | --- | ---: | ---: | ---: | ---: |
44
+ | Validation | 0.5659 | 0.5966 | 0.6051 | 0.4784 |
45
+ | Test | 0.5330 | 0.5767 | 0.5859 | 0.4695 |
46
 
47
+ Additional threshold candidates on test:
 
48
 
49
+ | Threshold policy | Test macro-F1 |
50
+ | --- | ---: |
51
+ | Fixed 0.5 | 0.5184 |
52
+ | Global threshold | 0.5320 |
53
+ | Validation coordinate search | 0.5330 |
54
+ | Per-label thresholds | 0.5350 |
55
 
56
+ The exported `thresholds.json` stores the validation-selected coordinate
57
+ thresholds used for the headline test metrics.
58
+
59
+ ## Intended Use
60
+
61
+ Use this model for research, benchmarking, exploratory emotion analysis, and
62
+ building GoEmotions-compatible classifiers. It is best suited to English
63
+ short-form text that resembles the public GoEmotions data distribution.
64
 
65
+ This model should not be used as the sole basis for decisions that affect
66
+ people in high-stakes settings. Emotion labels are subjective, culturally
67
+ dependent, and sensitive to context that may not be present in a single comment.
68
 
69
+ ## Quick Start
70
 
71
  ```python
72
  import json
 
73
  import torch
74
+ from huggingface_hub import hf_hub_download
75
  from transformers import AutoModelForSequenceClassification, AutoTokenizer
76
 
77
  repo_id = "AliceYin/goemotions-roberta-large-focal-sota"
78
+
79
  tokenizer = AutoTokenizer.from_pretrained(repo_id)
80
  model = AutoModelForSequenceClassification.from_pretrained(repo_id)
81
+ threshold_data = json.load(open(hf_hub_download(repo_id, "thresholds.json")))
82
+ labels = json.load(open(hf_hub_download(repo_id, "labels.json")))["label_names"]
83
+ threshold_map = threshold_data[threshold_data["selected"]]
84
+ thresholds = [threshold_map[label] for label in labels]
85
 
86
+ text = "I finally got this working and I am so relieved."
87
+ inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=160)
 
88
 
 
 
89
  with torch.no_grad():
90
+ probs = torch.sigmoid(model(**inputs).logits)[0]
91
 
92
  predicted = [
93
+ {"label": label, "score": float(prob)}
94
+ for label, prob, threshold in zip(labels, probs, thresholds)
95
  if prob >= threshold
96
  ]
97
  print(predicted)
98
  ```
99
 
100
+ ## Training Details
 
 
 
101
 
102
+ - Base model: `FacebookAI/roberta-large`
103
+ - Dataset: `google-research-datasets/go_emotions`, simplified configuration
104
+ - Loss: focal loss, alpha 0.38, gamma 2.8
105
+ - Epochs: 4
106
+ - Learning rate: 1e-5
107
+ - Batch size: 2 with gradient accumulation 16
108
+ - Mixed precision: disabled for stability
109
+ - Threshold selection: validation macro-F1 coordinate search
110
+ - Seed: 42
111
+
112
+ ## Citation
113
+
114
+ ```bibtex
115
+ @inproceedings{demszky-etal-2020-goemotions,
116
+ title = "{G}o{E}motions: A Dataset of Fine-Grained Emotions",
117
+ author = "Demszky, Dorottya and Movshovitz-Attias, Dana and Ko, Jeongwoo and Cowen, Alan and Nemade, Gaurav and Ravi, Sujith",
118
+ booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
119
+ year = "2020",
120
+ pages = "4040--4054"
121
+ }
122
+ ```
123
 
124
+ ## Reproducibility
125
 
126
+ The Kaggle artifact includes `metrics.json`, `thresholds.json`, `labels.json`,
127
+ the tokenizer, the model weights, and the Kaggle run log. The training kernel
128
+ and experiment notes record the exact settings used for the reported metrics.