docs: commit eval summary; clarify critic as LLM-assisted-judge; fix test imports
Browse files- eval/results/summary.csv: un-gitignored and committed. All numeric
claims in the README eval table (52% staleness catch, 43.9% position
accuracy, 17.1s avg latency) are now reproducible on clone.
- README: clarified that the critic uses deterministic threshold
routing for STALE/INSUFFICIENT/PASS verdicts and an LLM-as-a-Judge
prompt only for pairwise CONTRADICTED detection.
- Test_Files/test_setup.py: removed β dev-time environment check script
(Phase 1 setup) with imports no longer in requirements.txt.
- .gitignore +2 -1
- README.md +4 -2
- Test_Files/test_setup.py +0 -28
- eval/results/summary.csv +6 -0
.gitignore
CHANGED
|
@@ -25,7 +25,8 @@ data/cache/
|
|
| 25 |
*.db
|
| 26 |
|
| 27 |
# RECON β eval outputs (large, reproducible)
|
| 28 |
-
eval/results/
|
|
|
|
| 29 |
eval/survey_sources.json
|
| 30 |
eval/survey_sources_summary.txt
|
| 31 |
|
|
|
|
| 25 |
*.db
|
| 26 |
|
| 27 |
# RECON β eval outputs (large, reproducible)
|
| 28 |
+
eval/results/*
|
| 29 |
+
!eval/results/summary.csv
|
| 30 |
eval/survey_sources.json
|
| 31 |
eval/survey_sources_summary.txt
|
| 32 |
|
README.md
CHANGED
|
@@ -43,7 +43,9 @@ session_loader β planner β retriever β critic β synthesizer
|
|
| 43 |
- `CONTRADICTED` β claims conflict across retrieved sources
|
| 44 |
- `INSUFFICIENT` β not enough high-quality evidence to synthesize
|
| 45 |
|
| 46 |
-
|
|
|
|
|
|
|
| 47 |
|
| 48 |
**Synthesizer** β produces a structured research position: overview, key findings, active debates, and a per-claim confidence table with source attribution.
|
| 49 |
|
|
@@ -102,7 +104,7 @@ This is a **reference dataset used in evaluation** β not auto-generated from l
|
|
| 102 |
| Fallback retrieval | DuckDuckGo (`ddgs`) + Tavily |
|
| 103 |
| Embeddings | `all-MiniLM-L6-v2` |
|
| 104 |
| Session memory | SQLite |
|
| 105 |
-
| Eval | LLM-
|
| 106 |
| UI | Gradio 6.10 |
|
| 107 |
|
| 108 |
One deliberate choice worth noting: the `semanticscholar` PyPI library was explicitly avoided due to a pagination hang bug on large result sets. All S2 calls go through direct `requests.get()` to `graph/v1/paper/search`.
|
|
|
|
| 43 |
- `CONTRADICTED` β claims conflict across retrieved sources
|
| 44 |
- `INSUFFICIENT` β not enough high-quality evidence to synthesize
|
| 45 |
|
| 46 |
+
The critic combines deterministic threshold routing with an LLM-assisted contradiction check. STALE, INSUFFICIENT, and PASS verdicts are assigned based on hardcoded thresholds (mean paper age, minimum result count, score cutoffs). CONTRADICTED is determined by calling Groq with a structured pairwise prompt that returns a `{"contradicts": bool, "reason": "..."}` JSON verdict β a canonical LLM-as-a-Judge pattern applied to contradiction detection specifically.
|
| 47 |
+
|
| 48 |
+
If the critic issues anything other than PASS, the retriever tries again with a refined query (max 2 retries). This retry loop is what drives the staleness catch rate improvement.
|
| 49 |
|
| 50 |
**Synthesizer** β produces a structured research position: overview, key findings, active debates, and a per-claim confidence table with source attribution.
|
| 51 |
|
|
|
|
| 104 |
| Fallback retrieval | DuckDuckGo (`ddgs`) + Tavily |
|
| 105 |
| Embeddings | `all-MiniLM-L6-v2` |
|
| 106 |
| Session memory | SQLite |
|
| 107 |
+
| Eval | LLM-assisted contradiction detection (Groq structured prompt) + custom staleness catch rate metric |
|
| 108 |
| UI | Gradio 6.10 |
|
| 109 |
|
| 110 |
One deliberate choice worth noting: the `semanticscholar` PyPI library was explicitly avoided due to a pagination hang bug on large result sets. All S2 calls go through direct `requests.get()` to `graph/v1/paper/search`.
|
Test_Files/test_setup.py
DELETED
|
@@ -1,28 +0,0 @@
|
|
| 1 |
-
from dotenv import load_dotenv
|
| 2 |
-
import os
|
| 3 |
-
|
| 4 |
-
load_dotenv()
|
| 5 |
-
|
| 6 |
-
print("=== Testing imports ===")
|
| 7 |
-
import langgraph; print("β langgraph")
|
| 8 |
-
import langchain; print(f"β langchain {langchain.__version__}")
|
| 9 |
-
import gradio; print(f"β gradio {gradio.__version__}")
|
| 10 |
-
import semanticscholar; print("β semanticscholar")
|
| 11 |
-
from sentence_transformers import SentenceTransformer; print("β sentence-transformers")
|
| 12 |
-
import networkx; print(f"β networkx {networkx.__version__}")
|
| 13 |
-
|
| 14 |
-
print("\n=== Testing API keys ===")
|
| 15 |
-
groq_key = os.getenv("GROQ_API_KEY")
|
| 16 |
-
s2_key = os.getenv("S2_API_KEY")
|
| 17 |
-
tavily_key = os.getenv("TAVILY_API_KEY")
|
| 18 |
-
print(f"β GROQ_API_KEY: {'set' if groq_key else 'MISSING'}")
|
| 19 |
-
print(f"β S2_API_KEY: {'set β will activate in 1-3 days' if s2_key else 'not set yet (pending)' }")
|
| 20 |
-
print(f"β TAVILY_API_KEY: {'set' if tavily_key else 'MISSING'}")
|
| 21 |
-
|
| 22 |
-
print("\n=== Testing Groq connection ===")
|
| 23 |
-
from langchain_groq import ChatGroq
|
| 24 |
-
llm = ChatGroq(model="llama-3.3-70b-versatile", api_key=groq_key)
|
| 25 |
-
response = llm.invoke("Say exactly: setup confirmed")
|
| 26 |
-
print(f"β Groq response: {response.content}")
|
| 27 |
-
|
| 28 |
-
print("\nβ
Phase 1 complete β all systems go")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
eval/results/summary.csv
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
architecture,total_questions,position_match_rate,staleness_catch_rate,contradiction_catch_rate,avg_latency_ms,retry_rate,error_rate
|
| 2 |
+
single_rag,130,0.3231,0.0,0.3,4786.3,0.0,0.0
|
| 3 |
+
naive_multi,130,0.4462,0.0,1.0,23866.0,0.0,0.0
|
| 4 |
+
recon_none,130,0.4769,0.42,1.0,21817.6,0.3923,0.0
|
| 5 |
+
recon_linear,130,0.4385,0.52,1.0,17094.4,0.3385,0.0
|
| 6 |
+
recon_log,130,0.4308,0.38,1.0,15943.1,0.3615,0.0
|