Point Kaggle notebook link to kevin250304

a9eccc1 verified about 1 month ago

6.25 kB

license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
base_model: FacebookAI/roberta-large
datasets:
  - google-research-datasets/go_emotions
metrics:
  - f1
language:
  - en
tags:
  - goemotions
  - emotion-classification
  - multi-label-classification
  - roberta
  - roberta-large
  - focal-loss
  - threshold-optimization
  - nlp
model-index:
  - name: GoEmotions RoBERTa-large Focal Loss Classifier
    results:
      - task:
          type: text-classification
          name: Multi-label emotion classification
        dataset:
          name: GoEmotions simplified
          type: google-research-datasets/go_emotions
          config: simplified
          split: test
        metrics:
          - type: f1
            value: 0.5330202487288448
            name: Macro-F1
          - type: f1
            value: 0.5766508516761297
            name: Micro-F1
          - type: f1
            value: 0.5859415444821746
            name: Samples-F1

GoEmotions RoBERTa-large Focal Loss Classifier

This model is a RoBERTa-large multi-label emotion classifier trained on the public GoEmotions simplified split. It predicts 27 fine-grained emotions plus neutral from English Reddit-style text.

The run uses focal loss for label imbalance and validation-tuned coordinate thresholds for multi-label decisions. It is a competitive public-reference result: the validation-selected policy reached test macro-F1 0.5330, while the strongest public model card found during this iteration reported test macro-F1 0.519. This is not presented as formal SOTA because there is no official GoEmotions leaderboard comparison here.

Maintainer

GitHub: Kevin-Li-2025
Kaggle: kevin250304
Hugging Face: AliceYin

Results

Split	Macro-F1	Micro-F1	Samples-F1	Subset accuracy
Validation	0.5659	0.5966	0.6051	0.4784
Test	0.5330	0.5767	0.5859	0.4695

Threshold selection on validation:

Threshold policy	Validation macro-F1	Validation micro-F1	Validation samples-F1
Fixed 0.5	0.5147	0.6021	0.6086
Global validation-tuned threshold	0.5383	0.5676	0.5783
Per-label thresholds	0.5634	0.5925	0.6007
Coordinate thresholds	0.5659	0.5966	0.6051

Additional threshold candidates on test:

Threshold policy	Test macro-F1
Fixed 0.5	0.5184
Global threshold	0.5320
Validation coordinate search	0.5330
Per-label thresholds	0.5350

The headline result uses the validation-selected coordinate threshold policy to avoid test-set overfitting. The per-label threshold candidate reached the highest test macro-F1, but it was not selected by validation macro-F1 and is therefore not the headline policy. The exported thresholds.json stores all threshold policies plus selected: "coordinate".

Intended Use

Use this model for research, benchmarking, exploratory emotion analysis, and building GoEmotions-compatible classifiers. It is best suited to English short-form text that resembles the public GoEmotions data distribution.

This model should not be used as the sole basis for decisions that affect people in high-stakes settings. Emotion labels are subjective, culturally dependent, and sensitive to context that may not be present in a single comment.

Quick Start

import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

HF_MODEL_ID = "AliceYin/goemotions-roberta-large-focal-sota"
KAGGLE_MODEL_URL = (
    "https://www.kaggle.com/models/kevin250304/"
    "goemotions-roberta-large-focal-sota/Transformers/roberta-large-focal-seed42"
)

tokenizer = AutoTokenizer.from_pretrained(HF_MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(HF_MODEL_ID)

with open(hf_hub_download(HF_MODEL_ID, "thresholds.json"), encoding="utf-8") as f:
    threshold_data = json.load(f)
with open(hf_hub_download(HF_MODEL_ID, "labels.json"), encoding="utf-8") as f:
    labels = json.load(f)["label_names"]

selected_policy = threshold_data["selected"]
selected_thresholds = threshold_data[selected_policy]
threshold_map = (
    selected_thresholds["per_label"]
    if selected_policy == "global"
    else selected_thresholds
)
thresholds = [threshold_map[label] for label in labels]

text = "I finally got this working and I am so relieved."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=160)

with torch.no_grad():
    probs = torch.sigmoid(model(**inputs).logits)[0]

predicted = [
    {"label": label, "score": float(prob)}
    for label, prob, threshold in zip(labels, probs, thresholds)
    if prob >= threshold
]
print(predicted)

Training Details

Base model: FacebookAI/roberta-large
Dataset: google-research-datasets/go_emotions, simplified configuration
Loss: focal loss, alpha 0.38, gamma 2.8
Epochs: 4
Learning rate: 1e-5
Batch size: 2 with gradient accumulation 16
Mixed precision: disabled for stability
Threshold selection: validation macro-F1 coordinate search
Seed: 42

Citation

@inproceedings{demszky-etal-2020-goemotions,
  title = "{G}o{E}motions: A Dataset of Fine-Grained Emotions",
  author = "Demszky, Dorottya and Movshovitz-Attias, Dana and Ko, Jeongwoo and Cowen, Alan and Nemade, Gaurav and Ravi, Sujith",
  booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
  year = "2020",
  doi = "10.18653/v1/2020.acl-main.372",
  pages = "4040--4054"
}

Reproducibility

The Kaggle artifact includes metrics.json, thresholds.json, labels.json, the tokenizer, the model weights, and the Kaggle run log. The training script and experiment notes record the exact settings used for the reported metrics.

AliceYin
/

goemotions-roberta-large-focal-sota