Phase 0: branch setup for v2-edge-reliability

CHANGELOG.md

@@ -0,0 +1,9 @@
+# CHANGELOG — RECON v2 (Edge Reliability)
+
+## [Unreleased]
+
+### Phase 0 — Branch Setup
+- Created v2-edge-reliability branch from main
+- Added CHANGELOG.md
+- v1 remains on main branch, deployed to HF Spaces
+- All v2 development happens on this branch

RECON_CONTEXT.md

@@ -0,0 +1,438 @@
+# RECON — Full Project Context & Research Direction
+**Last updated:** April 2026  
+**Purpose:** Complete handoff document for Claude Code sessions. Contains everything discussed across the research planning conversation — architecture, eval results, collaboration context, professor feedback, and the revised technical direction.
+
+---
+
+## 1. What RECON Is
+
+RECON is a four-agent LangGraph state machine for temporally-aware scientific literature retrieval. Its defining contribution is treating **temporal supersession** as a first-class retrieval failure mode — something no existing RAG evaluation framework (RAGAS, ARES, TREC RAG) currently measures.
+
+**The core argument:** A 2019 paper with 800 citations scores high on cosine similarity and high on authority. If a 2023 paper explicitly refutes its claims, retrieving the 2019 paper as evidence produces a confident but stale answer. Standard RAG has no mechanism to detect this. RECON's critic does.
+
+**Repos:**
+- GitHub: https://github.com/MukulRay1603/project-recon
+- HF Space (live, Gradio): https://huggingface.co/spaces/MukulRay/recon
+
+---
+
+## 2. Architecture (Source-Verified)
+
+```
+session_loader → planner → retriever → critic
+                                           │
+                        ┌──────────────────┴──────────────────┐
+                        │ PASS / FORCED_PASS                   │ STALE / CONTRADICTED / INSUFFICIENT
+                        ▼                                      ▼
+                   synthesizer → END              retry_retriever → critic (max 2 retries)
+```
+
+### Agent Responsibilities
+
+**session_loader** (`graph.py:51`)
+- Loads prior session context from SQLite before planner runs
+- Fails silently — pipeline continues if session load fails
+
+**planner** (`planner.py`)
+- LLM: Groq LLaMA 3.3-70B, temperature 0.2
+- Decomposes query into 2–3 temporally-typed sub-questions: foundational / recent / contested
+- Session-aware: injects last 3 prior queries to avoid repetition
+- Fallback: uses raw query if LLM output unparseable
+
+**retriever** (`retriever.py` + `retriever_utils.py`)
+- Semantic Scholar REST API via direct `requests.get()` to `graph/v1/paper/search`
+- `sleep(3)` rate limit guard per S2 call
+- Each paper scored: `hybrid_score = semantic_sim × 0.5 + recency × 0.3 + authority × 0.2`
+- Three recency decay configs: `none` / `linear` / `log` (parameterized via `decay_config` state field)
+- Linear decay: `max(0, 1 − age/20)` where age = current_year − paper_year
+- Log decay: `max(0, 1 − log(1+age)/log(21))`
+- Authority: `min(1.0, log(1+citations)/log(10001))` (log-normalized)
+- DuckDuckGo web search in parallel (`ddgs`, `region="wt-wt"`)
+- Tavily fallback if DDG fails
+- Results cached to `data/cache/{md5_hash}.json`
+- On HF Spaces: cache dir is `/tmp/recon_cache` via `RECON_CACHE_DIR` env var
+
+**critic** (`critic.py`)
+- LLM: Groq LLaMA 3.3-70B, temperature 0.1
+- Verdict order (sequential, first match wins):
+  1. `FORCED_PASS` — retry_count ≥ 2 (hard ceiling)
+  2. `INSUFFICIENT` — fewer than 3 papers retrieved
+  3. `INSUFFICIENT` — fewer than 3 papers with hybrid_score ≥ 0.40
+  4. `STALE` — mean paper age > 24 months
+  5. `CONTRADICTED` — LLM pairwise check on top-4 papers, only pairs with ≥ 2yr gap
+  6. `PASS` — all checks clear
+- On non-PASS: LLM rewrites sub-questions with strategy: `broaden` / `recent` / `probe_contradiction`
+- Rewritten questions stored in state as `rewritten_questions`
+- `calibration_bin` field set to verdict for eval aggregation
+
+**retry_retriever** (`graph.py:74`)
+- Uses `rewritten_questions` from critic instead of `sub_questions` from planner
+- Merges new papers with existing set — deduplication by `paper_id`
+- Re-sorts merged set by `hybrid_score` descending
+- Merges web results by URL deduplication
+- Increments `retry_count`
+
+**synthesizer** (`synthesizer.py`)
+- LLM: Groq LLaMA 3.3-70B, temperature 0.3
+- Produces four-section brief: Overview / Key Findings / Active Debates / Outlook
+- Per-claim confidence scoring: HIGH / MEDIUM / LOW based on hybrid_score + year
+- Inline citations formatted as `[Author et al., Year]`
+- Calls `log_verdict()` on every completed run — populates `verdict_log` table
+- Calls `save_turn()` — persists query + position + claims to `session_turns` table
+- Generates `export_md` field — full session as downloadable markdown
+
+### State Schema (`state.py`)
+```python
+class ResearchState(TypedDict):
+    original_query: str
+    session_id: str
+    session_context: Optional[SessionContext]
+    sub_questions: list[str]
+    retrieved_papers: list[Paper]
+    citation_graph: dict                    # {paper_id: [cited_ids]}
+    web_results: list[WebResult]
+    critic_verdict: str                     # PASS/STALE/CONTRADICTED/INSUFFICIENT/FORCED_PASS
+    critic_notes: str
+    rewritten_questions: list[str]
+    retry_count: int
+    synthesized_position: str
+    claim_confidences: list[Claim]
+    session_update: Optional[SessionUpdate]
+    export_md: str
+    decay_config: str                       # "none" | "linear" | "log"
+    calibration_bin: str
+    latency_ms: float
+```
+
+### Verdict Constants
+```
+PASS | STALE | CONTRADICTED | INSUFFICIENT | FORCED_PASS
+```
+
+---
+
+## 3. Database Schema
+
+SQLite at `data/sessions.db` locally, `/tmp/recon_sessions.db` on HF Spaces.
+
+**sessions:** `session_id TEXT PK, created_at TEXT, updated_at TEXT`  
+**session_turns:** `id INTEGER PK, session_id FK, query TEXT, position TEXT, claim_json TEXT`  
+**verdict_log:** `id INTEGER PK, session_id FK, query TEXT, verdict TEXT, retry_count INTEGER, decay_config TEXT, latency_ms REAL, timestamp TEXT`
+
+---
+
+## 4. Evaluation Results (Real, Source-Verified)
+
+| Architecture | Staleness Catch Rate | Position Accuracy |
+|---|---|---|
+| Single-pass RAG (baseline) | 0% | 32.3% |
+| RECON no decay | 42% | 38.1% |
+| RECON log decay | 38% | 36.7% |
+| **RECON linear decay** | **52%** | **43.9%** |
+
+- Benchmark: 130 questions sourced from real ML survey paper supersession chains
+- Evaluation method: LLM-as-judge via Groq on 5 architecture variants with backoff
+- Contradiction catch rate: 0% — known limitation, acknowledged honestly in results
+- Contradiction detection inflated by post-hoc heuristics was discovered and dropped (E8 in blueprint)
+
+---
+
+## 5. Tech Stack Rules (Never Break These)
+
+| Rule | Detail |
+|---|---|
+| S2 API | Direct `requests.get()` to `graph/v1/paper/search` ONLY. Never `semanticscholar` library |
+| DDG package | `ddgs` (not `duckduckgo-search`). Always `region="wt-wt"` |
+| Gradio | NOT in `requirements.txt`. Only in `sdk_version: 6.10.0` in README YAML |
+| SQLite path | `data/sessions.db` locally, `/tmp/recon_sessions.db` on HF via `SESSION_DB_PATH` env var |
+| Cache dir | `data/cache/` locally, `/tmp/recon_cache` on HF via `RECON_CACHE_DIR` env var |
+| Python | 3.12 locally (3.13 on HF) |
+| Verdict strings | `PASS / STALE / CONTRADICTED / INSUFFICIENT / FORCED_PASS` — exact case everywhere |
+| Eval numbers | Real only. 52% staleness catch rate, 43.9% position accuracy |
+
+---
+
+## 6. Novelty Assessment
+
+### What is genuinely novel
+- **Staleness catch rate** as a formal RAG evaluation metric. No existing framework (RAGAS, ARES, TREC RAG) measures temporal supersession as a failure mode.
+- **Four-verdict critic with failure-mode-specific query rewriting.** CRAG (2024) has three verdicts based on relevance (Correct/Incorrect/Ambiguous) — it does not detect temporal supersession. RECON's STALE verdict is a distinct concept from relevance failure.
+- **130-question superseded-claims benchmark.** No public benchmark targets ML literature staleness detection.
+
+### What is not novel
+- Multi-agent LangGraph pipelines — common pattern
+- Hybrid retrieval scoring (semantic + recency + authority) — exists in literature
+- LLM-as-judge for RAG evaluation — standard practice
+
+### Closest related work
+- **CRAG (Jan 2024)** — corrective RAG with relevance evaluator. No temporal supersession concept.
+- **TG-RAG (Oct 2025)** — temporal GraphRAG. Requires pre-existing timestamps on graph edges. NASA KG has none.
+- **T-GRAG (Aug 2025)** — dynamic GraphRAG for temporal conflicts. Same limitation as TG-RAG.
+- **DEAN (Feb 2024)** — outdated fact detection in KGs using structural contrastive learning. No LLM critic, no RAG connection, no scientific domain application.
+- **FOS Benchmark (Nov 2025)** — temporal scientific graph benchmark for interdisciplinary link prediction. Different task.
+
+---
+
+## 7. The NASA EO Knowledge Graph — Professor's Dataset
+
+**HF Dataset:** https://huggingface.co/datasets/nasa-gesdisc/nasa-eo-knowledge-graph  
+**HF Model (GNN):** https://huggingface.co/nasa-gesdisc/edgraph-gnn-graphsage  
+**HF Publications Dataset:** https://huggingface.co/datasets/nasa-gesdisc/es-publications-researchareas  
+**DOI:** 10.57967/hf/3463  
+**Version:** v1.2.0, October 2025  
+**Total nodes:** 150,351  
+
+### The Seven Node Types
+| Node Type | What It Represents |
+|---|---|
+| Dataset | Satellite/EO datasets from NASA DAACs + 184 providers |
+| Publication | Scientific papers citing those datasets |
+| ScienceKeyword | GCMD-controlled vocabulary tags (Ozone, Precipitation, etc.) |
+| Instrument | Sensors used to collect data (AIRS, MODIS, etc.) |
+| Platform | Satellites carrying instruments (Aqua, Terra, etc.) |
+| Project | Scientific missions (MERRA-2, etc.) |
+| DataCenter | NASA DAACs and affiliated institutions |
+
+### Known Schema Facts (from dataset card)
+- Every node has properties: `globalId`, `doi`, `pagerank_global`, and node-type-specific fields
+- Publication nodes have: `title`, `authors`, `year`, `doi`, `abstract`, `url`
+- **Relationship properties are null across all types** — edges carry no weight, no timestamp, no metadata
+- Available formats: JSON (JSONL), GraphML, Cypher (Neo4j)
+
+### The GNN Model
+- Architecture: Heterogeneous GraphSAGE (PyTorch Geometric)
+- Base embeddings: `nasa-impact/nasa-smd-ibm-st-v2` (fine-tuned)
+- Task: Link prediction for missing Dataset → ScienceKeyword edges
+- Purpose: Find datasets in the archive that are missing keyword tags they should have
+
+### The Companion Publications Dataset
+- ~12K GES-DISC citing publications, each classified into 20 applied research areas
+- Built by fine-tuning NASA's own LLM on labeled abstracts
+- 87% classification accuracy into research areas
+- This is the supervision signal for the GNN
+
+### The Core Gap (Our Identified Problem)
+The graph knows **what** is connected. It does not know **whether those connections are still trustworthy today.**
+
+- All Dataset → Publication edges are treated identically regardless of publication year
+- A 2008 paper and a 2024 paper carry the same weight during GNN message passing
+- The GNN's keyword predictions are being shaped by stale signal
+- The graph has no mechanism to distinguish a foundational citation from an outdated one
+
+The professor's own published work (NTRS 2024) explicitly identifies dataset version supersession as an open problem: *"Datasets undergo a life cycle where older versions are replaced by newer versions... It is challenging when publications citing a dataset need to be traced over the entire lifecycle."* — This is the gap we are filling.
+
+---
+
+## 8. The Collaboration — Prof Armin Mehrabian
+
+**Who:** NLP expert, NASA GES-DISC researcher. Main contributor on `nasa-eo-knowledge-graph` (commit author: `arminmehrabian`). The professor who shared the dataset link with Mukul in class.
+
+**Status:** Active, interested. Said "let us work on it and see if we can improve and publish it" after being shown RECON.
+
+**His feedback on RECON (received April 2026):**
+After trying the HF Space, he sent five specific points:
+
+1. *"I had no idea how to assess the validity of the results"* — output lacks explainability about WHY something was flagged. The verdict alone isn't enough for a domain expert to trust it.
+2. *"Are you actually analyzing content for contradiction, or mostly relying on metadata?"* — Honest methodological question. Currently STALE is mostly metadata-driven (age). CONTRADICTED uses LLM pairwise check on abstracts.
+3. *"I would be careful not to treat recency as validity. Older papers can be foundational."* — **The sharpest critique.** Pure age-based decay is too blunt. A 2003 paper with 10,000 citations is not stale.
+4. *"Think in terms of edge reliability rather than just staleness."* — He reframed the concept. Not "is this paper old?" but "how much should we trust this edge right now?" This is a richer and more defensible framing.
+5. *"Have you considered combining your signal with something like PageRank or citation centrality to preserve important older work?"* — He is pointing at `pagerank_global`, which already exists on every node in his graph. This is him telling us the answer.
+
+---
+
+## 9. The Revised Technical Direction — Edge Reliability Scoring
+
+### The Pivot
+**Old framing:** Staleness detection — flag papers that are old  
+**New framing:** Edge Reliability Scoring — score how much each Dataset→Publication edge should be trusted right now
+
+This survives the foundational paper objection. A 1998 paper with high PageRank and thousands of citations is reliable. A 2022 paper with 3 citations that contradicts it may be less reliable (or an emerging challenger — needs content signal to distinguish).
+
+### The New Formula
+
+```
+edge_reliability(dataset_id, publication_id) =
+    (citation_centrality × w1)
+  + (recency_signal     × w2)
+  + (content_coherence  × w3)
+```
+
+**citation_centrality** — `pagerank_global` from the professor's graph nodes, normalized to [0,1]. High PageRank = foundational = high reliability contribution regardless of age.
+
+**recency_signal** — RECON's existing linear decay formula: `max(0, 1 − age/20)`. Now just one component, not the whole score. Tunable weight `w2`.
+
+**content_coherence** — Does this paper's abstract still align with current scientific consensus on this topic? This is the LLM component. Query Semantic Scholar + OpenAlex for papers on the same topic published in the last 3 years, run a lightweight LLM check: "Does [older paper] make claims that are contradicted or superseded by [newer paper]?" Binary or scored output. This is the content analysis Armin asked about.
+
+**Suggested starting weights:** w1=0.4, w2=0.3, w3=0.3 — to be ablated.
+
+### What Changes in RECON
+
+| Current RECON | Revised RECON (v2) |
+|---|---|
+| STALE = mean paper age > 24 months | STALE = low edge_reliability across all three signals |
+| Recency is the primary signal | Recency is one of three weighted inputs |
+| No use of citation network position | `pagerank_global` from graph feeds directly in |
+| Contradiction is binary LLM check | Contradiction weighted by both papers' centrality |
+| Output shows verdict only | Output shows which signal drove the verdict + network trust score |
+| Metric: staleness catch rate | Metric: edge reliability precision + staleness catch rate (both) |
+
+### The Explainability Fix
+The prof said he couldn't assess the output. The fix: synthesizer adds a trust summary per paper:
+
+```
+[Smith et al., 2019]  
+Network reliability: LOW  
+Reason: low PageRank (0.12), newer work by Chen et al. 2023 addresses same claim with higher centrality
+```
+
+This gives a domain expert exactly what they need to sanity-check the verdict.
+
+---
+
+## 10. OpenAlex — The New Data Source
+
+The professor mentioned OpenAlex during the in-person conversation as a better source than Semantic Scholar alone for Earth science papers.
+
+**What it is:** Fully open scholarly database, 271M+ works, free REST API, CC0 license. Replacement for the discontinued Microsoft Academic Graph. Better coverage of Earth science journals and institutional repositories than Semantic Scholar.
+
+**API:** `api.openalex.org/works?filter=doi:YOUR_DOI`  
+Returns: title, year, cited_by_count, abstract, open_access status, referenced_works, citing_works
+
+**Plan:** Run Semantic Scholar + OpenAlex in parallel, deduplicate by DOI, merge into unified paper pool before the hybrid scorer runs. This directly addresses the paywall concern raised in conversation.
+
+### Retriever Stack (v2)
+```
+Query
+  ↓
+Semantic Scholar API  (~220M papers, strong for ML/CS)
+  +
+OpenAlex API          (~271M papers, strong for Earth science, fully open)
+  +
+CrossRef API          (DOI resolution, metadata fill-in for gaps)
+  ↓
+Merged, deduplicated by DOI
+  ↓
+RECON hybrid scorer   (semantic × 0.5 + recency × 0.3 + authority × 0.2)
+  ↓
+RECON critic v2       (PASS / STALE / CONTRADICTED / INSUFFICIENT)
+  ↓
+Synthesizer with trust summary per claim
+```
+
+---
+
+## 11. Paper Strategy
+
+### Paper 1 — RECON Standalone (arXiv, cs.IR)
+**Owner:** Mukul (solo)  
+**Status:** Ready to write. System is live. Eval is done.  
+**Target:** arXiv cs.IR preprint — timestamps the contribution  
+**Key ask from prof:** arXiv endorsement (first-time submitter needs it — he can click once)  
+**Sections:**
+1. Introduction — temporal supersession as an unaddressed RAG failure mode
+2. Related Work — CRAG, Self-RAG, TG-RAG, DEAN — positioned clearly against each
+3. System Design — four-agent architecture, four-verdict critic, decay ablation
+4. Evaluation — 52% vs 0% result, 130Q benchmark, 5-architecture comparison
+5. Limitations — 0% contradiction catch rate (honest), single domain, single LLM
+6. Future Work — edge reliability extension, NASA KG application (seeds Paper 2)
+
+**Important:** The revised "edge reliability" concept introduced from prof's feedback DOES NOT dilute Paper 1. Paper 1 is about the staleness catch rate metric and benchmark. Paper 2 is the graph extension.
+
+### Paper 2 — Joint with Prof Armin
+**Owner:** Prof Armin leads, Mukul co-authors  
+**Target:** ECIR, SIGIR, or AI for Earth Science workshop  
+**Contribution:** Edge reliability scoring for heterogeneous scientific knowledge graphs — no timestamps needed, scores derived from PageRank + recency + content coherence  
+**Mukul brings:** The edge reliability mechanism, staleness scorer code, evaluation harness, OpenAlex integration  
+**Prof brings:** Dataset access + credentials, GNN training infrastructure, domain expertise, institutional affiliation, venue selection  
+**Authorship:** Prof first, Mukul second — this is his dataset, his domain. Second author on a NASA GES-DISC paper is a strong outcome at this stage.
+
+---
+
+## 12. Immediate Next Steps (Ordered)
+
+### This week
+- [ ] Integrate OpenAlex API as second retriever source in `retriever.py`
+  - Endpoint: `api.openalex.org/works?filter=doi:X`
+  - Merge with S2 results, deduplicate by DOI
+  - Test on 20 questions from the existing benchmark
+
+- [ ] Add `network_reliability` field to the synthesizer output
+  - Show which signal (age / centrality / content) drove each verdict
+  - This directly addresses prof's "I couldn't assess the output" feedback
+
+- [ ] Upload 130Q benchmark as a public HF dataset
+  - Dataset card: what it contains, how it was built, what it measures
+  - Required before arXiv paper — paper cites it
+
+### Next 2 weeks
+- [ ] Prototype edge_reliability scorer
+  - Input: `(publication_id, pagerank_score, year, topic_query)`
+  - Output: reliability score [0,1] + which signal dominated
+  - Does NOT require KG credentials yet — can test on publication nodes from the public JSON
+
+- [ ] Ask prof for arXiv endorsement
+  - Natural timing: after showing him the OpenAlex integration + trust summary in the UI
+  - One sentence ask: "I'm planning to write this up for arXiv cs.IR — would you be okay endorsing my submission as a first-time submitter?"
+
+- [ ] Write the RECON arXiv paper
+  - 6–8 pages
+  - Do NOT include Paper 2 material — keep it clean and focused on the standalone contribution
+  - Mention edge reliability as future work
+
+### Before Paper 2
+- [ ] Get KG credentials from prof (needed to run GNN training with weighted edges)
+- [ ] Ask technical questions about publication nodes:
+  - Is the `year` field populated for all publication nodes?
+  - Are DOIs resolvable to Semantic Scholar/OpenAlex for all publications?
+  - What does `pagerank_global` represent exactly — is it global graph PageRank or something else?
+- [ ] Build Earth science staleness benchmark from `es-publications-researchareas`
+  - Find papers citing old dataset versions vs papers citing newer versions with improvement language
+  - Target: ~50–100 verified supersession chains
+
+---
+
+## 13. Open Questions (Unresolved, Need Discussion)
+
+**Technical:**
+- What weight split between w1 (centrality), w2 (recency), w3 (content) performs best? Needs ablation.
+- How to handle papers with no `pagerank_global` (new nodes, disconnected nodes)?
+- Should content_coherence use the paper's abstract only, or full text where available (OpenAlex has some open access full text)?
+- For a paper that is old BUT has high and *growing* citation trajectory — should that override age? Dynamic signal vs static snapshot?
+
+**Research design:**
+- Is the right evaluation metric "edge reliability precision" — i.e., do the edges we score LOW actually correspond to superseded science? Need ground truth for that.
+- Can the 130Q ML benchmark be partially adapted to test the revised formula, or does it need a new benchmark?
+- The contradiction catch rate is 0%. Armin's edge reliability framing might actually unlock this — a contradiction is just two edges pointing in opposite directions with conflicting content. Worth exploring.
+
+**Collaboration:**
+- What level of involvement does prof want in the RECON paper (Paper 1)? Acknowledgement only, or something more? Don't assume — ask when the time is right.
+- What venues does he prefer for Paper 2? His prior work appeared at NTRS, AGU, ESIP — might prefer an Earth science venue over a pure IR venue.
+
+---
+
+## 14. Key People & Links Reference
+
+| Item | Detail |
+|---|---|
+| Mukul's GitHub | https://github.com/MukulRay1603/project-recon |
+| RECON HF Space | https://huggingface.co/spaces/MukulRay/recon |
+| NASA KG Dataset | https://huggingface.co/datasets/nasa-gesdisc/nasa-eo-knowledge-graph |
+| NASA GNN Model | https://huggingface.co/nasa-gesdisc/edgraph-gnn-graphsage |
+| NASA Publications Dataset | https://huggingface.co/datasets/nasa-gesdisc/es-publications-researchareas |
+| OpenAlex API docs | https://docs.openalex.org |
+| Semantic Scholar API | https://api.semanticscholar.org/graph/v1/paper/search |
+| CrossRef API | https://api.crossref.org/works/{doi} |
+| CRAG paper (related work) | https://arxiv.org/abs/2401.15884 |
+| TG-RAG paper (related work) | https://arxiv.org/abs/2510.13590 |
+| DEAN paper (related work) | https://arxiv.org/abs/2402.03732 |
+| Prof's NTRS paper on dataset lifecycle | https://ntrs.nasa.gov/citations/20240010838 |
+
+---
+
+## 15. How To Use This Document in Claude Code
+
+When starting a new session in the RECON repo with Claude Code, paste this at the start:
+
+> "Read RECON_CONTEXT.md in the project root for full context on what RECON is, where we are in the research, and what needs to be built next. Then do a code audit — read the key source files (graph.py, critic.py, retriever.py, retriever_utils.py, synthesizer.py, state.py) and tell me: (1) the current state of the codebase vs what's described in the context doc, (2) what's already implemented vs what's planned, (3) any discrepancies or things that look wrong."
+
+That gives Claude Code everything it needs to do a proper audit and continue from exactly here.