Hugging Face's logo

Spaces:
MukulRay
/
recon
Sleeping

App Files Community
recon / RECON_CONTEXT.md
MukulRay's picture
MukulRay
Phase 0: branch setup for v2-edge-reliability
77669a9
preview code
|
Raw
History Blame
23.6 kB

RECON — Full Project Context & Research Direction

Last updated: April 2026
Purpose: Complete handoff document for Claude Code sessions. Contains everything discussed across the research planning conversation — architecture, eval results, collaboration context, professor feedback, and the revised technical direction.

1. What RECON Is

RECON is a four-agent LangGraph state machine for temporally-aware scientific literature retrieval. Its defining contribution is treating temporal supersession as a first-class retrieval failure mode — something no existing RAG evaluation framework (RAGAS, ARES, TREC RAG) currently measures.

The core argument: A 2019 paper with 800 citations scores high on cosine similarity and high on authority. If a 2023 paper explicitly refutes its claims, retrieving the 2019 paper as evidence produces a confident but stale answer. Standard RAG has no mechanism to detect this. RECON's critic does.

Repos:

2. Architecture (Source-Verified) 

session_loader → planner → retriever → critic
                                           │
                        ┌──────────────────┴──────────────────┐
                        │ PASS / FORCED_PASS                   │ STALE / CONTRADICTED / INSUFFICIENT
                        ▼                                      ▼
                   synthesizer → END              retry_retriever → critic (max 2 retries)

Agent Responsibilities

session_loader (graph.py:51)

  • Loads prior session context from SQLite before planner runs
  • Fails silently — pipeline continues if session load fails

planner (planner.py)

  • LLM: Groq LLaMA 3.3-70B, temperature 0.2
  • Decomposes query into 2–3 temporally-typed sub-questions: foundational / recent / contested
  • Session-aware: injects last 3 prior queries to avoid repetition
  • Fallback: uses raw query if LLM output unparseable

retriever (retriever.py + retriever_utils.py)

  • Semantic Scholar REST API via direct requests.get() to graph/v1/paper/search
  • sleep(3) rate limit guard per S2 call
  • Each paper scored: hybrid_score = semantic_sim × 0.5 + recency × 0.3 + authority × 0.2
  • Three recency decay configs: none / linear / log (parameterized via decay_config state field)
  • Linear decay: max(0, 1 − age/20) where age = current_year − paper_year
  • Log decay: max(0, 1 − log(1+age)/log(21))
  • Authority: min(1.0, log(1+citations)/log(10001)) (log-normalized)
  • DuckDuckGo web search in parallel (ddgs, region="wt-wt")
  • Tavily fallback if DDG fails
  • Results cached to data/cache/{md5_hash}.json
  • On HF Spaces: cache dir is /tmp/recon_cache via RECON_CACHE_DIR env var

critic (critic.py)

  • LLM: Groq LLaMA 3.3-70B, temperature 0.1
  • Verdict order (sequential, first match wins):
    1. FORCED_PASS — retry_count ≥ 2 (hard ceiling)
    2. INSUFFICIENT — fewer than 3 papers retrieved
    3. INSUFFICIENT — fewer than 3 papers with hybrid_score ≥ 0.40
    4. STALE — mean paper age > 24 months
    5. CONTRADICTED — LLM pairwise check on top-4 papers, only pairs with ≥ 2yr gap
    6. PASS — all checks clear
  • On non-PASS: LLM rewrites sub-questions with strategy: broaden / recent / probe_contradiction
  • Rewritten questions stored in state as rewritten_questions
  • calibration_bin field set to verdict for eval aggregation

retry_retriever (graph.py:74)

  • Uses rewritten_questions from critic instead of sub_questions from planner
  • Merges new papers with existing set — deduplication by paper_id
  • Re-sorts merged set by hybrid_score descending
  • Merges web results by URL deduplication
  • Increments retry_count

synthesizer (synthesizer.py)

  • LLM: Groq LLaMA 3.3-70B, temperature 0.3
  • Produces four-section brief: Overview / Key Findings / Active Debates / Outlook
  • Per-claim confidence scoring: HIGH / MEDIUM / LOW based on hybrid_score + year
  • Inline citations formatted as [Author et al., Year]
  • Calls log_verdict() on every completed run — populates verdict_log table
  • Calls save_turn() — persists query + position + claims to session_turns table
  • Generates export_md field — full session as downloadable markdown

State Schema (state.py) 

class ResearchState(TypedDict):
    original_query: str
    session_id: str
    session_context: Optional[SessionContext]
    sub_questions: list[str]
    retrieved_papers: list[Paper]
    citation_graph: dict                    # {paper_id: [cited_ids]}
    web_results: list[WebResult]
    critic_verdict: str                     # PASS/STALE/CONTRADICTED/INSUFFICIENT/FORCED_PASS
    critic_notes: str
    rewritten_questions: list[str]
    retry_count: int
    synthesized_position: str
    claim_confidences: list[Claim]
    session_update: Optional[SessionUpdate]
    export_md: str
    decay_config: str                       # "none" | "linear" | "log"
    calibration_bin: str
    latency_ms: float

Verdict Constants 

PASS | STALE | CONTRADICTED | INSUFFICIENT | FORCED_PASS

3. Database Schema

SQLite at data/sessions.db locally, /tmp/recon_sessions.db on HF Spaces.

sessions: session_id TEXT PK, created_at TEXT, updated_at TEXT
session_turns: id INTEGER PK, session_id FK, query TEXT, position TEXT, claim_json TEXT
verdict_log: id INTEGER PK, session_id FK, query TEXT, verdict TEXT, retry_count INTEGER, decay_config TEXT, latency_ms REAL, timestamp TEXT

4. Evaluation Results (Real, Source-Verified)

Architecture Staleness Catch Rate Position Accuracy
Single-pass RAG (baseline) 0% 32.3%
RECON no decay 42% 38.1%
RECON log decay 38% 36.7%
RECON linear decay 52% 43.9%
  • Benchmark: 130 questions sourced from real ML survey paper supersession chains
  • Evaluation method: LLM-as-judge via Groq on 5 architecture variants with backoff
  • Contradiction catch rate: 0% — known limitation, acknowledged honestly in results
  • Contradiction detection inflated by post-hoc heuristics was discovered and dropped (E8 in blueprint)

5. Tech Stack Rules (Never Break These)

Rule Detail
S2 API Direct requests.get() to graph/v1/paper/search ONLY. Never semanticscholar library
DDG package ddgs (not duckduckgo-search). Always region="wt-wt"
Gradio NOT in requirements.txt. Only in sdk_version: 6.10.0 in README YAML
SQLite path data/sessions.db locally, /tmp/recon_sessions.db on HF via SESSION_DB_PATH env var
Cache dir data/cache/ locally, /tmp/recon_cache on HF via RECON_CACHE_DIR env var
Python 3.12 locally (3.13 on HF)
Verdict strings PASS / STALE / CONTRADICTED / INSUFFICIENT / FORCED_PASS — exact case everywhere
Eval numbers Real only. 52% staleness catch rate, 43.9% position accuracy

6. Novelty Assessment

What is genuinely novel

  • Staleness catch rate as a formal RAG evaluation metric. No existing framework (RAGAS, ARES, TREC RAG) measures temporal supersession as a failure mode.
  • Four-verdict critic with failure-mode-specific query rewriting. CRAG (2024) has three verdicts based on relevance (Correct/Incorrect/Ambiguous) — it does not detect temporal supersession. RECON's STALE verdict is a distinct concept from relevance failure.
  • 130-question superseded-claims benchmark. No public benchmark targets ML literature staleness detection.

What is not novel

  • Multi-agent LangGraph pipelines — common pattern
  • Hybrid retrieval scoring (semantic + recency + authority) — exists in literature
  • LLM-as-judge for RAG evaluation — standard practice

Closest related work

  • CRAG (Jan 2024) — corrective RAG with relevance evaluator. No temporal supersession concept.
  • TG-RAG (Oct 2025) — temporal GraphRAG. Requires pre-existing timestamps on graph edges. NASA KG has none.
  • T-GRAG (Aug 2025) — dynamic GraphRAG for temporal conflicts. Same limitation as TG-RAG.
  • DEAN (Feb 2024) — outdated fact detection in KGs using structural contrastive learning. No LLM critic, no RAG connection, no scientific domain application.
  • FOS Benchmark (Nov 2025) — temporal scientific graph benchmark for interdisciplinary link prediction. Different task.

7. The NASA EO Knowledge Graph — Professor's Dataset

HF Dataset: https://huggingface.co/datasets/nasa-gesdisc/nasa-eo-knowledge-graph
HF Model (GNN): https://huggingface.co/nasa-gesdisc/edgraph-gnn-graphsage
HF Publications Dataset: https://huggingface.co/datasets/nasa-gesdisc/es-publications-researchareas
DOI: 10.57967/hf/3463
Version: v1.2.0, October 2025
Total nodes: 150,351

The Seven Node Types

Node Type What It Represents
Dataset Satellite/EO datasets from NASA DAACs + 184 providers
Publication Scientific papers citing those datasets
ScienceKeyword GCMD-controlled vocabulary tags (Ozone, Precipitation, etc.)
Instrument Sensors used to collect data (AIRS, MODIS, etc.)
Platform Satellites carrying instruments (Aqua, Terra, etc.)
Project Scientific missions (MERRA-2, etc.)
DataCenter NASA DAACs and affiliated institutions

Known Schema Facts (from dataset card)

  • Every node has properties: globalId, doi, pagerank_global, and node-type-specific fields
  • Publication nodes have: title, authors, year, doi, abstract, url
  • Relationship properties are null across all types — edges carry no weight, no timestamp, no metadata
  • Available formats: JSON (JSONL), GraphML, Cypher (Neo4j)

The GNN Model

  • Architecture: Heterogeneous GraphSAGE (PyTorch Geometric)
  • Base embeddings: nasa-impact/nasa-smd-ibm-st-v2 (fine-tuned)
  • Task: Link prediction for missing Dataset → ScienceKeyword edges
  • Purpose: Find datasets in the archive that are missing keyword tags they should have

The Companion Publications Dataset

  • ~12K GES-DISC citing publications, each classified into 20 applied research areas
  • Built by fine-tuning NASA's own LLM on labeled abstracts
  • 87% classification accuracy into research areas
  • This is the supervision signal for the GNN

The Core Gap (Our Identified Problem)

The graph knows what is connected. It does not know whether those connections are still trustworthy today.

  • All Dataset → Publication edges are treated identically regardless of publication year
  • A 2008 paper and a 2024 paper carry the same weight during GNN message passing
  • The GNN's keyword predictions are being shaped by stale signal
  • The graph has no mechanism to distinguish a foundational citation from an outdated one

The professor's own published work (NTRS 2024) explicitly identifies dataset version supersession as an open problem: "Datasets undergo a life cycle where older versions are replaced by newer versions... It is challenging when publications citing a dataset need to be traced over the entire lifecycle." — This is the gap we are filling.

8. The Collaboration — Prof Armin Mehrabian

Who: NLP expert, NASA GES-DISC researcher. Main contributor on nasa-eo-knowledge-graph (commit author: arminmehrabian). The professor who shared the dataset link with Mukul in class.

Status: Active, interested. Said "let us work on it and see if we can improve and publish it" after being shown RECON.

His feedback on RECON (received April 2026): After trying the HF Space, he sent five specific points:

  1. "I had no idea how to assess the validity of the results" — output lacks explainability about WHY something was flagged. The verdict alone isn't enough for a domain expert to trust it.
  2. "Are you actually analyzing content for contradiction, or mostly relying on metadata?" — Honest methodological question. Currently STALE is mostly metadata-driven (age). CONTRADICTED uses LLM pairwise check on abstracts.
  3. "I would be careful not to treat recency as validity. Older papers can be foundational." — The sharpest critique. Pure age-based decay is too blunt. A 2003 paper with 10,000 citations is not stale.
  4. "Think in terms of edge reliability rather than just staleness." — He reframed the concept. Not "is this paper old?" but "how much should we trust this edge right now?" This is a richer and more defensible framing.
  5. "Have you considered combining your signal with something like PageRank or citation centrality to preserve important older work?" — He is pointing at pagerank_global, which already exists on every node in his graph. This is him telling us the answer.

9. The Revised Technical Direction — Edge Reliability Scoring

The Pivot

Old framing: Staleness detection — flag papers that are old
New framing: Edge Reliability Scoring — score how much each Dataset→Publication edge should be trusted right now

This survives the foundational paper objection. A 1998 paper with high PageRank and thousands of citations is reliable. A 2022 paper with 3 citations that contradicts it may be less reliable (or an emerging challenger — needs content signal to distinguish).

The New Formula 

edge_reliability(dataset_id, publication_id) =
    (citation_centrality × w1)
  + (recency_signal     × w2)
  + (content_coherence  × w3)

citation_centrality — pagerank_global from the professor's graph nodes, normalized to [0,1]. High PageRank = foundational = high reliability contribution regardless of age.

recency_signal — RECON's existing linear decay formula: max(0, 1 − age/20). Now just one component, not the whole score. Tunable weight w2.

content_coherence — Does this paper's abstract still align with current scientific consensus on this topic? This is the LLM component. Query Semantic Scholar + OpenAlex for papers on the same topic published in the last 3 years, run a lightweight LLM check: "Does [older paper] make claims that are contradicted or superseded by [newer paper]?" Binary or scored output. This is the content analysis Armin asked about.

Suggested starting weights: w1=0.4, w2=0.3, w3=0.3 — to be ablated.

What Changes in RECON

Current RECON Revised RECON (v2)
STALE = mean paper age > 24 months STALE = low edge_reliability across all three signals
Recency is the primary signal Recency is one of three weighted inputs
No use of citation network position pagerank_global from graph feeds directly in
Contradiction is binary LLM check Contradiction weighted by both papers' centrality
Output shows verdict only Output shows which signal drove the verdict + network trust score
Metric: staleness catch rate Metric: edge reliability precision + staleness catch rate (both)

The Explainability Fix

The prof said he couldn't assess the output. The fix: synthesizer adds a trust summary per paper:

[Smith et al., 2019]  
Network reliability: LOW  
Reason: low PageRank (0.12), newer work by Chen et al. 2023 addresses same claim with higher centrality

This gives a domain expert exactly what they need to sanity-check the verdict.

10. OpenAlex — The New Data Source

The professor mentioned OpenAlex during the in-person conversation as a better source than Semantic Scholar alone for Earth science papers.

What it is: Fully open scholarly database, 271M+ works, free REST API, CC0 license. Replacement for the discontinued Microsoft Academic Graph. Better coverage of Earth science journals and institutional repositories than Semantic Scholar.

API: api.openalex.org/works?filter=doi:YOUR_DOI
Returns: title, year, cited_by_count, abstract, open_access status, referenced_works, citing_works

Plan: Run Semantic Scholar + OpenAlex in parallel, deduplicate by DOI, merge into unified paper pool before the hybrid scorer runs. This directly addresses the paywall concern raised in conversation.

Retriever Stack (v2) 

Query
  ↓
Semantic Scholar API  (~220M papers, strong for ML/CS)
  +
OpenAlex API          (~271M papers, strong for Earth science, fully open)
  +
CrossRef API          (DOI resolution, metadata fill-in for gaps)
  ↓
Merged, deduplicated by DOI
  ↓
RECON hybrid scorer   (semantic × 0.5 + recency × 0.3 + authority × 0.2)
  ↓
RECON critic v2       (PASS / STALE / CONTRADICTED / INSUFFICIENT)
  ↓
Synthesizer with trust summary per claim

11. Paper Strategy

Paper 1 — RECON Standalone (arXiv, cs.IR)

Owner: Mukul (solo)
Status: Ready to write. System is live. Eval is done.
Target: arXiv cs.IR preprint — timestamps the contribution
Key ask from prof: arXiv endorsement (first-time submitter needs it — he can click once)
Sections:

  1. Introduction — temporal supersession as an unaddressed RAG failure mode
  2. Related Work — CRAG, Self-RAG, TG-RAG, DEAN — positioned clearly against each
  3. System Design — four-agent architecture, four-verdict critic, decay ablation
  4. Evaluation — 52% vs 0% result, 130Q benchmark, 5-architecture comparison
  5. Limitations — 0% contradiction catch rate (honest), single domain, single LLM
  6. Future Work — edge reliability extension, NASA KG application (seeds Paper 2)

Important: The revised "edge reliability" concept introduced from prof's feedback DOES NOT dilute Paper 1. Paper 1 is about the staleness catch rate metric and benchmark. Paper 2 is the graph extension.

Paper 2 — Joint with Prof Armin

Owner: Prof Armin leads, Mukul co-authors
Target: ECIR, SIGIR, or AI for Earth Science workshop
Contribution: Edge reliability scoring for heterogeneous scientific knowledge graphs — no timestamps needed, scores derived from PageRank + recency + content coherence
Mukul brings: The edge reliability mechanism, staleness scorer code, evaluation harness, OpenAlex integration
Prof brings: Dataset access + credentials, GNN training infrastructure, domain expertise, institutional affiliation, venue selection
Authorship: Prof first, Mukul second — this is his dataset, his domain. Second author on a NASA GES-DISC paper is a strong outcome at this stage.

12. Immediate Next Steps (Ordered)

This week

  • Integrate OpenAlex API as second retriever source in retriever.py

    • Endpoint: api.openalex.org/works?filter=doi:X
    • Merge with S2 results, deduplicate by DOI
    • Test on 20 questions from the existing benchmark

  • Add network_reliability field to the synthesizer output

    • Show which signal (age / centrality / content) drove each verdict
    • This directly addresses prof's "I couldn't assess the output" feedback

  • Upload 130Q benchmark as a public HF dataset

    • Dataset card: what it contains, how it was built, what it measures
    • Required before arXiv paper — paper cites it

Next 2 weeks

  • Prototype edge_reliability scorer

    • Input: (publication_id, pagerank_score, year, topic_query)
    • Output: reliability score [0,1] + which signal dominated
    • Does NOT require KG credentials yet — can test on publication nodes from the public JSON

  • Ask prof for arXiv endorsement

    • Natural timing: after showing him the OpenAlex integration + trust summary in the UI
    • One sentence ask: "I'm planning to write this up for arXiv cs.IR — would you be okay endorsing my submission as a first-time submitter?"

  • Write the RECON arXiv paper

    • 6–8 pages
    • Do NOT include Paper 2 material — keep it clean and focused on the standalone contribution
    • Mention edge reliability as future work

Before Paper 2

  • Get KG credentials from prof (needed to run GNN training with weighted edges)
  • Ask technical questions about publication nodes:
    • Is the year field populated for all publication nodes?
    • Are DOIs resolvable to Semantic Scholar/OpenAlex for all publications?
    • What does pagerank_global represent exactly — is it global graph PageRank or something else?
  • Build Earth science staleness benchmark from es-publications-researchareas
    • Find papers citing old dataset versions vs papers citing newer versions with improvement language
    • Target: ~50–100 verified supersession chains

13. Open Questions (Unresolved, Need Discussion)

Technical:

  • What weight split between w1 (centrality), w2 (recency), w3 (content) performs best? Needs ablation.
  • How to handle papers with no pagerank_global (new nodes, disconnected nodes)?
  • Should content_coherence use the paper's abstract only, or full text where available (OpenAlex has some open access full text)?
  • For a paper that is old BUT has high and growing citation trajectory — should that override age? Dynamic signal vs static snapshot?

Research design:

  • Is the right evaluation metric "edge reliability precision" — i.e., do the edges we score LOW actually correspond to superseded science? Need ground truth for that.
  • Can the 130Q ML benchmark be partially adapted to test the revised formula, or does it need a new benchmark?
  • The contradiction catch rate is 0%. Armin's edge reliability framing might actually unlock this — a contradiction is just two edges pointing in opposite directions with conflicting content. Worth exploring.

Collaboration:

  • What level of involvement does prof want in the RECON paper (Paper 1)? Acknowledgement only, or something more? Don't assume — ask when the time is right.
  • What venues does he prefer for Paper 2? His prior work appeared at NTRS, AGU, ESIP — might prefer an Earth science venue over a pure IR venue.

14. Key People & Links Reference

Item Detail
Mukul's GitHub https://github.com/MukulRay1603/project-recon
RECON HF Space https://huggingface.co/spaces/MukulRay/recon
NASA KG Dataset https://huggingface.co/datasets/nasa-gesdisc/nasa-eo-knowledge-graph
NASA GNN Model https://huggingface.co/nasa-gesdisc/edgraph-gnn-graphsage
NASA Publications Dataset https://huggingface.co/datasets/nasa-gesdisc/es-publications-researchareas
OpenAlex API docs https://docs.openalex.org
Semantic Scholar API https://api.semanticscholar.org/graph/v1/paper/search
CrossRef API https://api.crossref.org/works/{doi}
CRAG paper (related work) https://arxiv.org/abs/2401.15884
TG-RAG paper (related work) https://arxiv.org/abs/2510.13590
DEAN paper (related work) https://arxiv.org/abs/2402.03732
Prof's NTRS paper on dataset lifecycle https://ntrs.nasa.gov/citations/20240010838

15. How To Use This Document in Claude Code

When starting a new session in the RECON repo with Claude Code, paste this at the start:

"Read RECON_CONTEXT.md in the project root for full context on what RECON is, where we are in the research, and what needs to be built next. Then do a code audit — read the key source files (graph.py, critic.py, retriever.py, retriever_utils.py, synthesizer.py, state.py) and tell me: (1) the current state of the codebase vs what's described in the context doc, (2) what's already implemented vs what's planned, (3) any discrepancies or things that look wrong."

That gives Claude Code everything it needs to do a proper audit and continue from exactly here.