Add SilverSet fine-tuned model for trade-news event deduplication

6784955 verified 2 months ago

4.22 kB

license: mit
base_model: ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli
datasets:
  - lyutovad/TradeNewsEventDedup
language:
  - en
library_name: transformers
pipeline_tag: text-classification
tags:
  - text-classification
  - duplicate-detection
  - trade-news
  - information-retrieval
  - event-deduplication
  - nli
  - roberta
  - entailment

trade-news-dedup-roberta-large-nli

Event-level duplicate detection in trade news — NLI-based classifier (RoBERTa-large), fine-tuned on the weakly-supervised SilverSet of the TradeNewsEventDedup project.

Обнаружение дубликатов событий во внешнеторговых новостях: бинарная классификация пар новостных саммари (дубликат / не-дубликат) на уровне идентичности торгового события.

⭐ Highest PR-AUC and Recall on the GoldSet; largest relative gain from fine-tuning.

Task

Given two trade-news summaries, predict whether they describe the same real-world trade event (same country, commodity, trade action, numerical values and time) — not mere text similarity. The model was fine-tuned with structured hard negatives (semantically close but materially different cases: updates U and related events R).

Base model: ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli
Training data: lyutovad/TradeNewsEventDedup — SilverSet (16,097 LLM-labeled pairs, fine-tuning only)
Evaluation: GoldSet (1,469 manually validated pairs) — see results below
Output head: Three-class NLI head. The duplicate signal is taken from the entailment logit (index 0); use a symmetric average over both input orderings. See finetune_meta.json for head_info.

Results (GoldSet, fine-tuned)

Metric	Value
PR-AUC	0.9396
F1	0.8968
Accuracy	0.9095
Recall	0.9444

Thresholds are tuned on the GoldSet (optimistic estimate). Baselines (lexical/embedding similarity) reach high recall but low precision; fine-tuning with hard negatives improves precision and reduces false positives. Full protocol and ablations are in the paper.

How to use

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "afafos/trade-news-dedup-roberta-large-nli"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

a = "Iraq and Lebanon signed an agreement and seven MoUs on trade and investment."
b = "Iraq and Lebanon announced a new partnership framework, including seven MoUs."

def dup_score(x, y):
    with torch.no_grad():
        logits = model(**tok(x, y, return_tensors="pt", truncation=True)).logits
    return torch.softmax(logits, dim=-1)[0, 0].item()   # index 0 == entailment ~ duplicate

# symmetric signal over both orderings (see finetune_meta.json -> head_info)
p_duplicate = 0.5 * (dup_score(a, b) + dup_score(b, a))
print(p_duplicate)

Limitations

Summaries are machine-translated to English; quality depends on the preprocessing pipeline.
Training labels (SilverSet) are LLM-generated (weak supervision) — possible label bias.
Decision threshold tuned on the evaluation set; validation by a single annotator.
Evaluated on a single domain (trade / foreign-economic news).

Links & citation

📦 Dataset: https://huggingface.co/datasets/lyutovad/TradeNewsEventDedup
💻 Code: https://github.com/SaidKamalov/trade-news-duplicates
Paper: Event-Level Duplicate Detection in Trade News under Hard-Negative Supervision — D. Liutova, S. Kamalov, A. Afanasev, T. Mukhtarov.

@misc{tradenews_event_dedup,
  title  = {Event-Level Duplicate Detection in Trade News under Hard-Negative Supervision},
  author = {Liutova, Daria and Kamalov, Said and Afanasev, Andrew and Mukhtarov, Timerlan},
  year   = {2026},
  note   = {Dataset: lyutovad/TradeNewsEventDedup; Code: https://github.com/SaidKamalov/trade-news-duplicates}
}