Hugging Face's logo

siddhm11
/
researchit-reranker-phase6

ml-intern
Model card Files
xet
 Community
researchit-reranker-phase6 / CHANGELOG.md
siddhm11's picture
siddhm11
Update CHANGELOG: add V4 and V5 results. V5 achieves Hard Neg AUC=0.837
9856611 verified
preview code
|
Raw
History Blame
3.71 kB

Changelog

v5.0.0 — Graph + Metadata Model (2026-05-11)

V5: Combined Graph Topology + Turso Paper Metadata

Added real paper metadata features from Turso DB (1.6M papers):

  • Citation counts, influential citations, publication dates
  • Primary category matching (same field = relevance signal)
  • Recency scores, citation-per-year velocity
  • Citation count ratios (candidate vs survey)

This broke through the graph-only ceiling. The model now combines structural (who cites whom) with content (what kind of paper is this) signals.

Metric CitationCount V4 (graph) V5 (graph+meta) Oracle
nDCG@10 0.837 0.946 1.000 1.000
Hard Neg AUC 0.628 0.778 0.837 1.000
MRR 0.876 0.902 0.938 0.938

Hard Neg AUC: 0.837 — model can distinguish expert-included from expert-excluded papers 83.7% of the time.

Top Features (V5)

  1. n_surveys_citing (33,610) — cross-survey consensus
  2. cand_cited_by (5,473) — total incoming citations from graph
  3. hub_score (1,091) — cited by well-connected papers
  4. survey_refs (1,036) — complexity of the survey paper
  5. cand_recency (971) — newer papers get slight boost
  6. cand_influential_cit (716) — Semantic Scholar influential citations
  7. same_category (607) — category match matters!
  8. cited_by_survey_refs (584) — 2-hop citation paths
  9. authority (579) — PageRank-like authority
  10. cit_ratio (572) — relative citation strength

Model Details

  • 25 features (12 graph + 5 advanced graph + 8 metadata)
  • 200 trees, 127 leaves, lr=0.05
  • 2.7 MB model file
  • Trained on 58,749 rows, 486 queries

v4.0.0 — Advanced Graph Features (2026-05-11)

V4: Extended Graph Topology

Added 12 new features beyond V3:

  • Authority score (PageRank-like)
  • Year difference and recency
  • Clustering coefficient
  • Preferential attachment
  • Indirect citation paths
Metric V3 V4
Hard Neg AUC 0.738 0.778

v3.0.0 — Cross-Survey Authority Model + Eval V2 (2026-05-10)

New Evaluation Framework (Eval V2)

The old eval tested "can you find a cited paper?" which inflated metrics. The new eval tests "can you rank papers an expert considers essential above ones they deliberately excluded?"

  • Label scheme: 5 tiers (essential=4, relevant=3, useful=2, niche=1, hard_negative=0)
  • Hard negatives: Papers cited by OTHER surveys in same field but NOT this one (expert exclusions)
  • Key metric: Hard Negative AUC (can model tell cited from expert-excluded?)
  • Data: 500 comprehensive papers (≥30 references each), 60,574 rows
  • Design doc: EVAL_V2_DESIGN.md

V3 Model: Cross-Survey Authority

Trained on cross-survey consensus labels with 13 graph-based features.

Metric Random CitationCount V3 LightGBM Oracle
Hard Neg AUC 0.504 0.628 0.738 1.000

Key Insight

V1/V2 models are essentially popularity rankers. V3+ moves beyond this by learning cross-survey authority.

v2.0.0 — V2 LightGBM (2025-04-27)

  • nDCG@10: 0.9186, MRR: 1.0000
  • 26,494 training rows, 532 queries
  • Model size: 353 KB

v1.0.0 — Production Model (2025-04-27)

  • nDCG@10: 0.8791 (vs heuristic 0.2641, +232.8%)
  • 90,993 training rows, 1,857 queries
  • Model size: 948 KB

v0.1.0 — Synthetic Proof of Concept (2025-04-27)

  • Full pipeline tested on synthetic data
  • nDCG@10: 0.9985

v0.0.1 — Pipeline Design (2025-04-27)

  • 3-script pipeline created
  • 37-feature schema designed