| # Changelog |
|
|
| ## v5.0.0 — Graph + Metadata Model (2026-05-11) |
|
|
| ### V5: Combined Graph Topology + Turso Paper Metadata |
| Added real paper metadata features from Turso DB (1.6M papers): |
| - Citation counts, influential citations, publication dates |
| - Primary category matching (same field = relevance signal) |
| - Recency scores, citation-per-year velocity |
| - Citation count ratios (candidate vs survey) |
|
|
| **This broke through the graph-only ceiling.** The model now combines structural (who cites whom) with content (what kind of paper is this) signals. |
|
|
| | Metric | CitationCount | V4 (graph) | **V5 (graph+meta)** | Oracle | |
| |--------|---------------|------------|---------------------|--------| |
| | nDCG@10 | 0.837 | 0.946 | **1.000** | 1.000 | |
| | Hard Neg AUC | 0.628 | 0.778 | **0.837** | 1.000 | |
| | MRR | 0.876 | 0.902 | **0.938** | 0.938 | |
|
|
| **Hard Neg AUC: 0.837** — model can distinguish expert-included from expert-excluded papers 83.7% of the time. |
|
|
| ### Top Features (V5) |
| 1. `n_surveys_citing` (33,610) — cross-survey consensus |
| 2. `cand_cited_by` (5,473) — total incoming citations from graph |
| 3. `hub_score` (1,091) — cited by well-connected papers |
| 4. `survey_refs` (1,036) — complexity of the survey paper |
| 5. `cand_recency` (971) — newer papers get slight boost |
| 6. `cand_influential_cit` (716) — Semantic Scholar influential citations |
| 7. `same_category` (607) — category match matters! |
| 8. `cited_by_survey_refs` (584) — 2-hop citation paths |
| 9. `authority` (579) — PageRank-like authority |
| 10. `cit_ratio` (572) — relative citation strength |
|
|
| ### Model Details |
| - **25 features** (12 graph + 5 advanced graph + 8 metadata) |
| - **200 trees**, 127 leaves, lr=0.05 |
| - **2.7 MB** model file |
| - Trained on 58,749 rows, 486 queries |
|
|
| --- |
|
|
| ## v4.0.0 — Advanced Graph Features (2026-05-11) |
|
|
| ### V4: Extended Graph Topology |
| Added 12 new features beyond V3: |
| - Authority score (PageRank-like) |
| - Year difference and recency |
| - Clustering coefficient |
| - Preferential attachment |
| - Indirect citation paths |
|
|
| | Metric | V3 | **V4** | |
| |--------|-----|--------| |
| | Hard Neg AUC | 0.738 | **0.778** | |
|
|
| --- |
|
|
| ## v3.0.0 — Cross-Survey Authority Model + Eval V2 (2026-05-10) |
|
|
| ### New Evaluation Framework (Eval V2) |
| The old eval tested "can you find a cited paper?" which inflated metrics. The new eval tests "can you rank papers an expert considers essential above ones they deliberately excluded?" |
|
|
| - **Label scheme**: 5 tiers (essential=4, relevant=3, useful=2, niche=1, hard_negative=0) |
| - **Hard negatives**: Papers cited by OTHER surveys in same field but NOT this one (expert exclusions) |
| - **Key metric**: Hard Negative AUC (can model tell cited from expert-excluded?) |
| - **Data**: 500 comprehensive papers (≥30 references each), 60,574 rows |
| - **Design doc**: `EVAL_V2_DESIGN.md` |
| |
| ### V3 Model: Cross-Survey Authority |
| Trained on **cross-survey consensus labels** with 13 graph-based features. |
| |
| | Metric | Random | CitationCount | V3 LightGBM | Oracle | |
| |--------|--------|---------------|-------------|--------| |
| | Hard Neg AUC | 0.504 | 0.628 | **0.738** | 1.000 | |
| |
| ### Key Insight |
| V1/V2 models are essentially popularity rankers. V3+ moves beyond this by learning cross-survey authority. |
| |
| --- |
| |
| ## v2.0.0 — V2 LightGBM (2025-04-27) |
| |
| - nDCG@10: **0.9186**, MRR: **1.0000** |
| - 26,494 training rows, 532 queries |
| - Model size: 353 KB |
| |
| --- |
| |
| ## v1.0.0 — Production Model (2025-04-27) |
| |
| - nDCG@10: **0.8791** (vs heuristic 0.2641, +232.8%) |
| - 90,993 training rows, 1,857 queries |
| - Model size: 948 KB |
| |
| --- |
| |
| ## v0.1.0 — Synthetic Proof of Concept (2025-04-27) |
| |
| - Full pipeline tested on synthetic data |
| - nDCG@10: 0.9985 |
| |
| --- |
| |
| ## v0.0.1 — Pipeline Design (2025-04-27) |
| |
| - 3-script pipeline created |
| - 37-feature schema designed |
| |
