File size: 2,038 Bytes
bfb1eb8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | # Changelog
## v1.0.0 — Production Model (2025-04-27)
### Trained on Real Data
- **242,179 citation edges** from Semantic Scholar API (50K sampled papers from 1.6M corpus)
- **90,993 training rows** (1,857 queries, pre-2023 papers)
- **7,007 eval rows** (143 queries, 2023+ papers)
- Strict time-split with verified no temporal leakage
### Results
- nDCG@10: **0.8791** (vs heuristic 0.2641 → +232.8%)
- nDCG@5: **0.8250** (vs heuristic 0.1819 → +353.6%)
- MRR: **0.8795** (vs heuristic 0.2906 → +202.7%)
- HR@10: **1.0000** (vs heuristic 0.6638 → +50.6%)
- Latency: **0.371ms** per 100 candidates (2.7× under 1ms budget)
- Model size: **948 KB**
### Key Features (by importance)
1. `candidate_num_cited_by` (75,203) — dominant signal
2. `age_ratio` (7,597)
3. `candidate_position` (6,765)
4. `cosine_x_citations` (2,383)
5. `qdrant_cosine_score` (2,353)
### Model Details
- LightGBM lambdarank, 500 rounds with early stopping at iteration 141
- 37-feature schema (26 active + 11 zero-filled user features)
- Heuristic baseline: qdrant_cosine_score as proxy for ewma_longterm_similarity
---
## v0.1.0 — Synthetic Proof of Concept (2025-04-27)
### Pipeline Validation
- Full 3-script pipeline tested end-to-end on synthetic data
- 6-category test suite: data quality, model learning, fair comparison, prod readiness, feature analysis, honest verdict
- All tests passed
### Synthetic Results
- nDCG@10: 0.9985 (vs heuristic 0.9111 → +9.6%)
- 91.4% per-query win rate vs heuristic
- 0.088ms latency, 286 KB model size
- Train-eval gap: 0.0008 (no overfitting)
---
## Pipeline Design (2025-04-27)
### Created
- `01_fetch_citation_edges.py` — S2 API batch + bulk download + checkpoint/resume
- `02_generate_training_triples.py` — Qdrant ANN + Turso metadata + time-split
- `03_train_lightgbm.py` — lambdarank training + heuristic comparison + latency benchmark
- `test_full_pipeline.py` — comprehensive 6-category test suite
- 37-feature schema designed spanning content, user behavior, and cross features
|