MukulRay commited on
Commit
7624a2f
Β·
1 Parent(s): 0bf564b

docs: commit eval summary; clarify critic as LLM-assisted-judge; fix test imports

Browse files

- eval/results/summary.csv: un-gitignored and committed. All numeric
claims in the README eval table (52% staleness catch, 43.9% position
accuracy, 17.1s avg latency) are now reproducible on clone.
- README: clarified that the critic uses deterministic threshold
routing for STALE/INSUFFICIENT/PASS verdicts and an LLM-as-a-Judge
prompt only for pairwise CONTRADICTED detection.
- Test_Files/test_setup.py: removed β€” dev-time environment check script
(Phase 1 setup) with imports no longer in requirements.txt.

Files changed (4) hide show
  1. .gitignore +2 -1
  2. README.md +4 -2
  3. Test_Files/test_setup.py +0 -28
  4. eval/results/summary.csv +6 -0
.gitignore CHANGED
@@ -25,7 +25,8 @@ data/cache/
25
  *.db
26
 
27
  # RECON β€” eval outputs (large, reproducible)
28
- eval/results/
 
29
  eval/survey_sources.json
30
  eval/survey_sources_summary.txt
31
 
 
25
  *.db
26
 
27
  # RECON β€” eval outputs (large, reproducible)
28
+ eval/results/*
29
+ !eval/results/summary.csv
30
  eval/survey_sources.json
31
  eval/survey_sources_summary.txt
32
 
README.md CHANGED
@@ -43,7 +43,9 @@ session_loader β†’ planner β†’ retriever β†’ critic β†’ synthesizer
43
  - `CONTRADICTED` β€” claims conflict across retrieved sources
44
  - `INSUFFICIENT` β€” not enough high-quality evidence to synthesize
45
 
46
- If the critic flags anything other than PASS, the retriever tries again with a refined query (max 2 retries). This retry loop is what drives the staleness catch rate improvement.
 
 
47
 
48
  **Synthesizer** β€” produces a structured research position: overview, key findings, active debates, and a per-claim confidence table with source attribution.
49
 
@@ -102,7 +104,7 @@ This is a **reference dataset used in evaluation** β€” not auto-generated from l
102
  | Fallback retrieval | DuckDuckGo (`ddgs`) + Tavily |
103
  | Embeddings | `all-MiniLM-L6-v2` |
104
  | Session memory | SQLite |
105
- | Eval | LLM-as-judge + custom staleness catch rate metric |
106
  | UI | Gradio 6.10 |
107
 
108
  One deliberate choice worth noting: the `semanticscholar` PyPI library was explicitly avoided due to a pagination hang bug on large result sets. All S2 calls go through direct `requests.get()` to `graph/v1/paper/search`.
 
43
  - `CONTRADICTED` β€” claims conflict across retrieved sources
44
  - `INSUFFICIENT` β€” not enough high-quality evidence to synthesize
45
 
46
+ The critic combines deterministic threshold routing with an LLM-assisted contradiction check. STALE, INSUFFICIENT, and PASS verdicts are assigned based on hardcoded thresholds (mean paper age, minimum result count, score cutoffs). CONTRADICTED is determined by calling Groq with a structured pairwise prompt that returns a `{"contradicts": bool, "reason": "..."}` JSON verdict β€” a canonical LLM-as-a-Judge pattern applied to contradiction detection specifically.
47
+
48
+ If the critic issues anything other than PASS, the retriever tries again with a refined query (max 2 retries). This retry loop is what drives the staleness catch rate improvement.
49
 
50
  **Synthesizer** β€” produces a structured research position: overview, key findings, active debates, and a per-claim confidence table with source attribution.
51
 
 
104
  | Fallback retrieval | DuckDuckGo (`ddgs`) + Tavily |
105
  | Embeddings | `all-MiniLM-L6-v2` |
106
  | Session memory | SQLite |
107
+ | Eval | LLM-assisted contradiction detection (Groq structured prompt) + custom staleness catch rate metric |
108
  | UI | Gradio 6.10 |
109
 
110
  One deliberate choice worth noting: the `semanticscholar` PyPI library was explicitly avoided due to a pagination hang bug on large result sets. All S2 calls go through direct `requests.get()` to `graph/v1/paper/search`.
Test_Files/test_setup.py DELETED
@@ -1,28 +0,0 @@
1
- from dotenv import load_dotenv
2
- import os
3
-
4
- load_dotenv()
5
-
6
- print("=== Testing imports ===")
7
- import langgraph; print("βœ“ langgraph")
8
- import langchain; print(f"βœ“ langchain {langchain.__version__}")
9
- import gradio; print(f"βœ“ gradio {gradio.__version__}")
10
- import semanticscholar; print("βœ“ semanticscholar")
11
- from sentence_transformers import SentenceTransformer; print("βœ“ sentence-transformers")
12
- import networkx; print(f"βœ“ networkx {networkx.__version__}")
13
-
14
- print("\n=== Testing API keys ===")
15
- groq_key = os.getenv("GROQ_API_KEY")
16
- s2_key = os.getenv("S2_API_KEY")
17
- tavily_key = os.getenv("TAVILY_API_KEY")
18
- print(f"βœ“ GROQ_API_KEY: {'set' if groq_key else 'MISSING'}")
19
- print(f"βœ“ S2_API_KEY: {'set β€” will activate in 1-3 days' if s2_key else 'not set yet (pending)' }")
20
- print(f"βœ“ TAVILY_API_KEY: {'set' if tavily_key else 'MISSING'}")
21
-
22
- print("\n=== Testing Groq connection ===")
23
- from langchain_groq import ChatGroq
24
- llm = ChatGroq(model="llama-3.3-70b-versatile", api_key=groq_key)
25
- response = llm.invoke("Say exactly: setup confirmed")
26
- print(f"βœ“ Groq response: {response.content}")
27
-
28
- print("\nβœ… Phase 1 complete β€” all systems go")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
eval/results/summary.csv ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ architecture,total_questions,position_match_rate,staleness_catch_rate,contradiction_catch_rate,avg_latency_ms,retry_rate,error_rate
2
+ single_rag,130,0.3231,0.0,0.3,4786.3,0.0,0.0
3
+ naive_multi,130,0.4462,0.0,1.0,23866.0,0.0,0.0
4
+ recon_none,130,0.4769,0.42,1.0,21817.6,0.3923,0.0
5
+ recon_linear,130,0.4385,0.52,1.0,17094.4,0.3385,0.0
6
+ recon_log,130,0.4308,0.38,1.0,15943.1,0.3615,0.0