Document EmpathRAG Core merged plan

docs/CURRENT_STATUS_AUDIT_FOR_RESEARCH_MODEL.md

@@ -1,19 +1,21 @@
 # EmpathRAG Current Status Audit
 
-Date: 2026-04-30  
-Branch: `codex-v2-safety-hardening`  
+Date: 2026-05-01  
+Branch: `codex/v2.5-support-navigator`  
 Audience: project team, research planning, MSML demo planning  
-Status: active V2 prototype work in progress
+Status: EmpathRAG Core consolidation in progress
 
 ## 1. One-Line Summary
 
-EmpathRAG is a safety-aware student-support retrieval prototype for UMD-style mental health and campus support help seeking. The current V2 direction is not to act as a therapist, diagnostician, or emergency service; it should route student concerns to appropriate support paths, show grounded sources, and fail closed when safety risk appears.
+EmpathRAG Core is a guarded conversational RAG system for UMD-style support navigation. It is not a therapist, diagnostician, counselor, crisis-prevention system, or emergency service; it should route student concerns to appropriate support paths, show grounded sources, and fail closed when safety risk appears.
+
+The current research hook is not "we made a chatbot." The hook is: V1 open empathetic RAG is evaluated as a baseline, its multi-turn safety failure modes motivate the redesign, and Core is evaluated as a guarded alternative.
 
 ## 2. Overall Idea
 
 The original idea was a mental-health-adjacent RAG system that can respond empathetically to student distress while grounding its answers in relevant resources.
 
-The refined V2 idea is more precise:
+The refined Core idea is more precise:
 
 - Detect the type of student-support need.
 - Detect whether the message is ordinary support, wellbeing/grounding, crisis, or emergency.
@@ -22,7 +24,22 @@ The refined V2 idea is more precise:
 - Show the user why the system retrieved each source.
 - Avoid pretending to provide therapy, diagnosis, crisis counseling, or clinical authority.
 
-This framing is much stronger for class presentation and research because it turns the project from a generic chatbot into a safety-aware support router.
+This framing is much stronger for class presentation and research because it turns the project from a generic chatbot into a safety-aware, source-grounded support navigator.
+
+## 2A. Merged Reviewer Consensus
+
+Keep these decisions locked:
+
+- One active product direction: EmpathRAG Core.
+- V1/V2/V2.5 remain checkpoints and baselines, not separate products.
+- UMD specificity remains the main demo moat.
+- Use synthetic labeled data only for route/safety training and evaluation.
+- Do not scrape Reddit, TikTok, Discord, private chats, or real student stories for the new dataset.
+- Keep V1 BERTScore/Wilcoxon/adversarial work as baseline evaluation.
+- Make multi-turn V1 vs Core safety evaluation the headline result.
+- Keep Integrated Gradients explainability as a named contribution.
+- Use "resource registry" or "service objects" in papers/presentations, not "service graph" unless graph edges/traversal are added.
+- Keep the TF-IDF/logistic router as a fast scaffold and baseline; train a RoBERTa route classifier only after Karthik delivers labeled route data.
 
 ## 3. What We Have Right Now
 

docs/EMPATHRAG_CORE_MASTER_CHECKLIST.md

@@ -0,0 +1,214 @@
+# EmpathRAG Core Master Checklist
+
+Date: 2026-05-01  
+Active branch: `codex/v2.5-support-navigator`  
+Status: one consolidated product direction; V1/V2/V2.5 are baselines/checkpoints only.
+
+## Core Thesis
+
+EmpathRAG started as emotion-aware RAG for student mental-health-adjacent support. Evaluation exposed structural risks in open empathetic generation: missed escalation, sycophantic validation, dependency reinforcement, and ungrounded advice.
+
+EmpathRAG Core is the guarded redesign: a UMD-specific conversational RAG system for support navigation, not therapy, diagnosis, counseling, crisis prevention, or clinical care.
+
+## Named Contributions
+
+1. Hybrid lexical + ML safety architecture with four-mode escalation.
+2. Multi-turn safety evaluation framework comparing V1 open RAG against Core guarded RAG.
+3. Integrated Gradients explainability for safety decisions from the DeBERTa guardrail.
+
+Supporting engineering contributions:
+
+- UMD-specific resource registry with service objects and verified provenance.
+- Output-side groundedness and anti-sycophancy guard.
+- Peer-helper mode for "I am worried about someone else" scenarios.
+
+## Phase 1: Resource Registry
+
+- [x] Preserve curated UMD corpus and source metadata.
+- [x] Add profile path scaffold under `data/profiles/umd/`.
+- [ ] Rename paper/docs wording from "service graph" to "resource registry", "service objects", or "service-tagged resource schema".
+- [ ] Expand UMD service objects toward roughly 80 verified entries if time allows.
+- [ ] Add/verify UMD-specific high-value resources:
+  - [ ] Counseling Center
+  - [ ] After-hours crisis support
+  - [ ] ADS
+  - [ ] Graduate Ombuds
+  - [ ] Graduate School support
+  - [ ] Campus Pantry / basic-needs support if verified
+  - [ ] CARE to Stop Violence if verified
+  - [ ] MHEART if verified
+- [ ] Use `unknown` for missing hours/location/contact; never invent details.
+
+## Phase 2: Core Pipeline
+
+- [x] Add unified `EmpathRAGCore.run_turn(...)` runtime interface.
+- [x] Keep fast demo startup reliable.
+- [x] Add TF-IDF/logistic route and tier classifier scaffold.
+- [x] Keep deterministic fallback when model artifacts are missing.
+- [x] Add output metadata for route, tier, confidence, retrieval mode, output guard, and latency.
+- [ ] Make hard lexical safety precheck an explicit Stage 1 before any ML classifier.
+- [ ] Optionally call the real DeBERTa safety guardrail for deeper eval and explainability.
+- [ ] Keep Integrated Gradients available for safety decisions without blocking the live demo.
+- [ ] Add RoBERTa route classifier only after labeled route dataset exists.
+
+Router ablation plan:
+
+- [x] Rule router baseline.
+- [x] TF-IDF/logistic router baseline.
+- [ ] RoBERTa route classifier trained on Karthik's labeled synthetic data.
+- [ ] Full hybrid system: lexical safety + ML routing + resource registry retrieval + output guard.
+
+## Phase 3: Safety And Output Gates
+
+- [x] Four safety tiers:
+  - `imminent_safety`
+  - `high_distress`
+  - `support_navigation`
+  - `wellbeing`
+- [x] Deterministic crisis/imminent response path.
+- [x] Rule-based output guard catches pure validation, harmful agreement, dependency language, and unsupported resource claims.
+- [ ] Add explicit `out_of_scope` handling in response planner and UI.
+- [ ] Add NLI-style groundedness check for response claims against top retrieved service objects when feasible.
+- [ ] Ensure crisis/imminent prompts never enter normal generation.
+
+## Phase 4: Response Layer And Demo
+
+- [x] Constrained response planner for non-crisis responses.
+- [x] Recommended next action included in Core result metadata.
+- [x] Gradio demo shows route, tier, confidence, retrieval mode, source cards, and output guard.
+- [ ] Make peer-helper mode a first-class visible feature, not only a route label.
+- [ ] Add peer-helper-specific wording:
+  - what to say
+  - what not to say
+  - when to escalate
+  - do not handle immediate safety alone
+- [ ] Polish 5-6 scripted demo prompts.
+- [ ] Keep demo framed as support navigation, not clinical help.
+
+## Phase 5A: Single-Turn Ablation
+
+Goal: broad comparison across router/system variants.
+
+Variants:
+
+- [ ] Rule router
+- [ ] TF-IDF/logistic router
+- [ ] RoBERTa route classifier
+- [ ] Full hybrid Core system
+
+Metrics:
+
+- route accuracy
+- safety tier accuracy
+- intercept accuracy
+- source appropriateness / source hit rate
+- avoid-list violation rate
+- unsafe generation count
+- pure validation / no-action count
+- ungrounded action count
+- latency
+
+Target dataset:
+
+- 300-500 synthetic single-turn prompts.
+- 14 route labels.
+- 4 safety tiers.
+- no real/private student data.
+- no Reddit/TikTok/Discord scraping.
+- no method details.
+
+## Phase 5B: Multi-Turn Headline Evaluation
+
+This is the paper hook.
+
+Goal: compare V1 open/emotion-aware RAG against EmpathRAG Core on vulnerability-amplifying conversations.
+
+Scenario categories:
+
+- slow escalation
+- dependency formation
+- help rejection
+- secrecy requests
+- peer-helper/friend risk
+- ambiguous academic idioms
+- sycophancy traps
+- method-seeking pressure without method details
+- ordinary academic stress
+
+Metrics:
+
+- missed escalation count
+- turn-by-turn route correctness
+- turn-by-turn tier correctness
+- intercept by required turn
+- dependency reinforcement
+- pure validation without redirect
+- method leakage
+- unsafe source use
+- latency
+
+Minimum target:
+
+- 30 multi-turn scenarios for class demo/paper draft.
+- 50 multi-turn scenarios for stronger research version.
+
+## Phase 6: V1 Results Preservation
+
+Do not orphan the original V1 work. It becomes baseline rigor.
+
+- [ ] Keep BERTScore and Wilcoxon results in the paper as V1 baseline evaluation.
+- [ ] Preserve adversarial safety evaluation from V1.
+- [ ] Use V1 results to motivate the Core redesign.
+- [ ] Present V1 as useful but structurally risky, not as "bad work."
+
+Paper structure:
+
+- Section 4: V1 baseline evaluation.
+- Section 5: what V1 missed.
+- Section 6: EmpathRAG Core architecture.
+- Section 7: Eval A single-turn ablation and Eval B multi-turn V1 vs Core benchmark.
+
+## Phase 7: Karthik Dataset Deadline
+
+Deadline: end of Day 5 from the current sprint.
+
+Blocking deliverables:
+
+- [ ] `single_turn_labeled.csv`: 300-500 synthetic prompts.
+- [ ] `multi_turn_scenarios.jsonl`: at least 30 scenarios, target 50.
+- [ ] `source_target_map.csv`.
+- [ ] `risky_ambiguous_cases.csv`.
+- [ ] `resource_profile_additions.csv`.
+- [ ] `README_dataset_notes.md`.
+
+Setup/verification tasks for Karthik if he runs the repo locally:
+
+- [ ] Pull latest `codex/v2.5-support-navigator`.
+- [ ] Verify tokenizer/model dependencies.
+- [ ] Install missing protobuf dependency if tokenizer loading requires it.
+- [ ] Verify `src/models/guardrail_ig.py` runs locally.
+
+Fallback if dataset slips:
+
+- [ ] Build a smaller hand-labeled set of about 150 prompts with Karthik.
+- [ ] Train/evaluate TF-IDF and, if feasible, a small RoBERTa route classifier.
+- [ ] Report small-N limitation honestly.
+- [ ] Do not block the demo on RoBERTa route training.
+
+## Claims To Avoid
+
+- Clinically validated.
+- Treats depression or anxiety.
+- Provides therapy, counseling, diagnosis, or intervention.
+- Prevents suicide or reduces fatal incidents.
+- Replaces professional or campus support.
+- Safe for real student deployment.
+
+## Immediate Next Build Tasks
+
+- [ ] Update docs to consistently use EmpathRAG Core and resource registry wording.
+- [ ] Add explicit Stage-1 lexical safety precheck metadata to Core output.
+- [ ] Add optional DeBERTa + IG hook to Core, disabled or skipped for fast demo when needed.
+- [ ] Improve peer-helper UI and tests.
+- [ ] Split eval reports into Eval A and Eval B summaries.
+- [ ] Keep demo polished and deterministic for the class presentation.

docs/KARTHIK_EMPATHRAG_CORE_DATASET_V2_REQUEST.md

@@ -8,6 +8,15 @@ Please create a new folder:
 empathrag_core_dataset_v2/
 ```
 
+Target deadline: **end of Day 5 of the current sprint**. This dataset is now on the critical path for training and evaluating the stronger route classifier. If the full set is not ready by then, please send a smaller but clean partial delivery rather than waiting.
+
+Before dataset work, if you are running the repo locally:
+
+- Pull the latest `codex/v2.5-support-navigator` branch.
+- Verify tokenizer/model dependencies.
+- Install `protobuf` if tokenizer loading complains about it.
+- Verify `src/models/guardrail_ig.py` can run locally, or note the exact error in `README_dataset_notes.md`.
+
 ## Required Files
 
 ```text
@@ -105,7 +114,7 @@ Rules:
 
 ## 2. `multi_turn_scenarios.jsonl`
 
-Target size: **50 scenarios**, each 3-6 turns.
+Target size: **30 scenarios minimum**, target **50 scenarios**, each 3-6 turns.
 
 Schema:
 
@@ -139,6 +148,9 @@ Include:
 - advisor conflict
 - low mood without immediate risk
 - explicit imminent safety
+- peer-helper/friend safety escalation
+- sycophancy traps
+- help-rejection and secrecy requests
 
 ## 3. `source_target_map.csv`
 
@@ -242,3 +254,14 @@ Before sending back:
 - Crisis examples are safe but still labelable.
 - `split` is populated for every single-turn row.
 - Sources are official or clearly marked as unknown/needs review.
+
+## Fallback Delivery
+
+If the full 300-500 row dataset cannot be finished by the deadline, send:
+
+- at least 150 well-labeled single-turn prompts
+- at least 15 multi-turn scenarios
+- complete route/tier/intercept/source labels
+- a note explaining what remains incomplete
+
+This fallback is still useful for a small-N route classifier and a class presentation ablation.

docs/PAPER_FRAMING.md

@@ -2,11 +2,11 @@
 
 Working title:
 
-EmpathRAG Core: Guarded Conversational Retrieval for Emotional Support Navigation
+EmpathRAG Core: Multi-Turn Safety Evaluation and Guarded Conversational Retrieval for UMD Support Navigation
 
 ## Core Story
 
-EmpathRAG V1 was an emotion-aware RAG system. It is useful as a baseline, but broad empathetic generation creates structural risks for mental-health-adjacent student support:
+EmpathRAG V1 was an emotion-aware RAG system. It is useful as a baseline, and its original evaluation work should stay in the paper. But broad empathetic generation creates structural risks for mental-health-adjacent student support:
 
 - generic validation
 - ungrounded advice
@@ -14,20 +14,27 @@ EmpathRAG V1 was an emotion-aware RAG system. It is useful as a baseline, but br
 - poor distinction between ordinary stress and safety risk
 - insufficient source/resource transparency
 
-EmpathRAG Core pivots to a safer architecture:
+EmpathRAG Core is the guarded redesign:
 
 - campus-specific support navigation
 - four-mode safety ladder
 - hybrid ML + rule route classification
-- service graph filtering
+- resource registry / service-object filtering
 - usage-mode gated retrieval
 - trajectory escalation
 - output-side guardrail
+- Integrated Gradients explanation for safety decisions
 - transparent source cards
 
+## Contributions
+
+1. Hybrid lexical + ML safety architecture with explicit mode-tiered escalation.
+2. Multi-turn safety evaluation framework comparing open V1 behavior against guarded Core behavior.
+3. Integrated Gradients explainability for safety decisions from the DeBERTa guardrail.
+
 ## Research Question
 
-Can a hybrid ML/rule router with graph-grounded retrieval and hard safety gates reduce inappropriate validation, ungrounded actions, and missed escalation while improving route accuracy and actionability compared with ungated RAG or generic LLM responses?
+Can a guarded conversational RAG architecture reduce missed escalation, inappropriate validation, ungrounded support actions, and unsafe source use compared with open emotion-aware RAG, especially in multi-turn student-support scenarios?
 
 ## Baselines
 
@@ -38,9 +45,44 @@ Optional ablations:
 
 - Core without output guard
 - Core without trajectory escalation
-- Core without service graph filtering
+- Core without resource registry filtering
 - Core rule-only router vs Core hybrid ML router
 
+## Evaluation Design
+
+### Section 4: V1 Baseline Evaluation
+
+Keep the original V1 results as baseline rigor:
+
+- BERTScore / reference-response evaluation.
+- Wilcoxon analysis for emotion-conditioned retrieval.
+- Single-turn adversarial safety comparison.
+
+These results establish that V1 was measured seriously before the redesign.
+
+### Eval A: Single-Turn Ablation
+
+Compare:
+
+- rule router
+- TF-IDF/logistic router
+- RoBERTa route classifier, once Karthik's route-labeled dataset exists
+- full hybrid Core system
+
+### Eval B: Multi-Turn Headline Benchmark
+
+Compare V1 open RAG against Core guarded RAG on multi-turn cases:
+
+- slow escalation
+- dependency formation
+- help rejection
+- peer-helper/friend risk
+- ambiguous academic idioms
+- sycophancy traps
+- method-seeking pressure without method details
+
+This is the main paper hook.
+
 ## Metrics
 
 - route accuracy