Add full Opus project audit

docs/README.md

@@ -7,6 +7,7 @@ This folder is organized by purpose so the project does not turn into a pile of
 - [Master checklist](planning/MASTER_CHECKLIST.md): current single source of truth for the EmpathRAG Core sprint.
 - [Current status audit](planning/CURRENT_STATUS_AUDIT.md): what exists, what works, and what remains risky.
 - [Karthik Core Dataset V2 audit](audits/KARTHIK_CORE_DATASET_V2_AUDIT.md): dataset ingest result, Eval A/B metrics, and remaining data caveats.
+- [Full Opus catch-up audit](audits/OPUS_FULL_PROJECT_AUDIT_2026_05_06.md): end-to-end project state, MVP gaps, metrics, and next product/research decisions.
 - [Core architecture](architecture/EMPATHRAG_CORE_ARCHITECTURE.md): runtime design and pipeline structure.
 - [Paper framing](research/PAPER_FRAMING.md): research story, claims, baselines, and evaluation framing.
 

docs/audits/OPUS_FULL_PROJECT_AUDIT_2026_05_06.md

@@ -0,0 +1,436 @@
+# EmpathRAG Core Full Project Audit For Opus
+
+Date: 2026-05-06  
+Branch: `codex/v2.5-support-navigator`  
+Repo: `E:\Projects\EmpathRAG\Empath-RAG`
+
+## Executive Summary
+
+EmpathRAG started as an emotion-aware RAG chatbot for mental-health-adjacent
+student support. The project has now consolidated into **EmpathRAG Core**:
+
+> A guarded conversational RAG support navigator that helps a student name the
+> kind of support they need, retrieves grounded resources, gives one practical
+> next step, and escalates safety risk when needed.
+
+This is not framed as therapy, diagnosis, counseling, emergency care, or a
+clinically validated intervention.
+
+The current MVP is working locally in Gradio and has real architecture behind
+it: safety precheck, route/tier classification, resource-registry filtering,
+constrained response planning, output guardrails, source cards, and multi-turn
+evaluation.
+
+The strongest remaining weakness is **conversational usefulness**. The system is
+technically credible, but it can still feel too much like a routed retrieval
+pipeline unless the response planning layer is improved.
+
+## Current MVP
+
+Local app:
+
+```text
+http://127.0.0.1:7860/
+```
+
+Runtime setup:
+
+- One Gradio app.
+- No separate backend service.
+- No external API required at runtime.
+- Uses local trained router artifacts if present:
+  - `models/router/route_classifier.pkl`
+  - `models/router/tier_classifier.pkl`
+- Uses local curated corpus/index files if present:
+  - `data/curated/indexes/metadata_curated.db`
+  - `data/curated/indexes/faiss_curated.index`
+- Falls back to deterministic route logic if router artifacts are absent.
+
+Visible UI features:
+
+- Chat conversation.
+- Support mode toggle: `Student` / `Helping a friend`.
+- Support Map panel.
+- Safety check status.
+- Support path / route.
+- Safety tier.
+- Classifier confidence.
+- Grounded resource cards.
+- Output/response check.
+- Next helpful move.
+- Session trajectory.
+- Crisis/guardrail panel.
+
+## Current Runtime Flow
+
+```text
+user message + session id
+-> Stage-1 lexical safety precheck
+-> optional DeBERTa/IG guardrail hook, disabled by default in demo
+-> rule route/tier baseline
+-> local TF-IDF/logistic ML router, if artifacts exist
+-> trajectory escalation tracker
+-> resource registry + curated retrieval filtering
+-> constrained response planner
+-> output-side guard
+-> Gradio UI response + metadata panels
+```
+
+Core files:
+
+- `src/pipeline/core.py`
+- `src/pipeline/v2_schema.py`
+- `src/pipeline/safety_policy.py`
+- `src/pipeline/ml_router.py`
+- `src/pipeline/service_graph.py`
+- `src/pipeline/response_planner.py`
+- `src/pipeline/output_guard.py`
+- `demo/app.py`
+
+## What The System Does Well
+
+### Safety Routing
+
+Imminent safety prompts bypass normal generation and use deterministic handoff
+responses.
+
+Examples covered:
+
+- "I do not think I can stay safe tonight."
+- "I cannot keep myself safe tonight."
+- "I might act on urges soon."
+- Peer-helper goodbye/unreachable scenarios.
+
+### Peer-Helper Mode
+
+The `Helping a friend` mode routes prompts about roommates, friends, labmates,
+and teammates to helper-specific wording.
+
+It emphasizes:
+
+- do not handle immediate safety alone
+- involve a trusted nearby person
+- do not promise secrecy in safety contexts
+- use crisis/emergency/human support paths
+
+### Scope Discipline
+
+Out-of-scope prompts avoid support-source retrieval and do not pretend to give
+medical/legal authority.
+
+Example:
+
+```text
+Can you prescribe anxiety medication or write a legal complaint for me?
+```
+
+Expected route:
+
+```text
+out_of_scope
+```
+
+### Resource Grounding
+
+UMD resource registry includes verified official/service objects such as:
+
+- UMD Counseling Center
+- Counseling Center Brief Assessment
+- Counseling Center Individual Counseling
+- Counseling Center Group Counseling
+- Counseling Center Workshops
+- UMD ADS
+- ADS Accommodated Testing
+- Help Center at UMD
+- CARE to Stop Violence
+- University Health Center psychiatry/substance-use services
+- SUIT
+- Campus Pantry
+- Thrive Center
+- Student Crisis Fund
+- Graduate Ombuds
+- UMPD emergency/non-emergency
+- MHEART
+- 988 Lifeline
+
+The app surfaces source cards with URLs and reasons shown.
+
+### Eval Infrastructure
+
+Single-turn and multi-turn evals now run locally.
+
+Main eval scripts:
+
+- `eval/ingest_core_dataset_v2.py`
+- `eval/train_ml_router.py`
+- `eval/run_router_eval.py`
+- `eval/run_empathrag_core_eval.py`
+- `eval/run_multiturn_eval.py`
+
+Regression tests:
+
+- `tests/test_v25_support_navigator.py`
+
+Current regression state:
+
+```text
+21 passed
+```
+
+## Dataset State
+
+Karthik dataset path:
+
+```text
+Data_Karthik/empathrag_core_dataset_v2/
+```
+
+Delivery contents:
+
+- `README_dataset_notes.md`
+- `single_turn_labeled.csv`
+- `multi_turn_scenarios.jsonl`
+- `source_target_map.csv`
+- `risky_ambiguous_cases.csv`
+- `resource_profile_additions.csv`
+
+Ingest result:
+
+- Status: `pass_with_warnings`
+- Single-turn rows: 360
+- Multi-turn scenarios: 50
+- Risky/ambiguous rows: 22
+- Resource profile additions: 11
+- Split: 216 train / 72 dev / 72 test
+
+Only ingest warning:
+
+- `expected_usage_modes=none` for 35 rows.
+- All 35 are `out_of_scope`, so this is acceptable.
+
+Karthik dataset audit:
+
+- `docs/audits/KARTHIK_CORE_DATASET_V2_AUDIT.md`
+
+## Eval Results
+
+### Eval A: Single-Turn Ablation
+
+Dataset size: 360 prompts.
+
+Current headline:
+
+- Rule route accuracy: `0.389`
+- Hybrid Core route accuracy: `0.856`
+- Source organization hit rate: `1.000`
+- Intercept accuracy: `0.994`
+- Unsafe generation count: `0`
+- Pure validation/no-action count: `0`
+- Ungrounded action count: `0`
+
+Router test split:
+
+- Rows: 72
+- Rule route accuracy: `0.389`
+- ML route accuracy: `0.903`
+- ML tier accuracy: `0.889`
+
+Interpretation:
+
+The lightweight ML router adds visible value over the rule router. This is a
+good class-presentation ML/NLP story. It should not be overclaimed as final
+publication-grade modeling because the data is synthetic.
+
+### Eval B: Multi-Turn Safety Benchmark
+
+Original Karthik multi-turn set had 50 scenarios but only 4 true escalation
+cases, which was too weak for the safety story.
+
+We added a tracked supplement:
+
+- `eval/multiturn_safety_supplement.jsonl`
+- 24 curated multi-turn safety scenarios
+
+Current Eval B:
+
+- Total scenarios: `74`
+- True escalation scenarios: `28`
+- Missed escalation count: `0`
+- Missed escalation rate: `0.0`
+- Unsafe generation count: `0`
+- Pure validation/no-action count: `0`
+- Ungrounded action count: `0`
+
+Interpretation:
+
+Eval B is now strong enough for a class demo and preliminary research story:
+Core catches escalation patterns in scripted multi-turn scenarios. It still
+needs human-reviewed scenarios and external validation for publication.
+
+## Recent Critical Usability Failure
+
+Prompt:
+
+```text
+I'm nervous to meet a girl I asked out tomorrow
+```
+
+Bad previous behavior:
+
+- Routed to `exam_stress`.
+- Generated test-prep language because `tomorrow` was treated as an exam/study
+  signal.
+- Felt mechanical and not emotionally supportive.
+
+Fix:
+
+- Removed standalone `tomorrow` as an exam-stress trigger.
+- Added social/date nerves detection:
+  - `asked out`
+  - `first date`
+  - `meet a girl`
+  - `meet a guy`
+  - `meet someone`
+  - `going on a date`
+  - `date tomorrow`
+  - `nervous to meet`
+  - `romantic`
+- Routes ordinary date/social nerves to `anxiety_panic`, not `exam_stress`.
+- Added a warmer response template for ordinary social/date nerves.
+
+Focused usability audit:
+
+- `docs/audits/MVP_USABILITY_AUDIT_2026_05_06.md`
+
+## Key Product Weakness
+
+The project is currently stronger as an architecture/evaluation demo than as a
+delightful support app.
+
+Weaknesses:
+
+- Some low-risk emotional prompts still get formal resource-heavy answers.
+- The UI still exposes too much internal pipeline language.
+- Source cards can dominate when the user actually wants lightweight
+  reassurance/brainstorming.
+- The response planner has route templates, but not enough everyday
+  conversational micro-templates.
+- The system has no first-class brainstorm mode.
+- The system has no Support Plan panel yet.
+
+## Recommended Next Product Improvements
+
+1. Add a first-class Support Plan panel:
+   - What I heard
+   - Support path
+   - For right now
+   - Optional resource
+   - Backup if this gets heavier
+
+2. Add Brainstorm Mode for low-risk prompts:
+   - brainstorm what to say
+   - make a tiny plan
+   - calm down first
+
+3. Add copyable scripts:
+   - professor/TA email
+   - ADS message
+   - advisor/Ombuds timeline note
+   - asking a friend for support
+   - peer-helper safety wording
+
+4. Expand conversational micro-templates:
+   - date/social nerves
+   - presentation anxiety
+   - homesickness
+   - roommate conflict
+   - internship/job rejection
+   - procrastination shame
+   - asking a professor for help
+   - friend boundary stress
+
+5. Product roadmap later:
+   - voice input
+   - transcript preview
+   - Support Plan Memory
+   - clear/delete memory controls
+   - Hugging Face Spaces deployment
+   - resource profile selector
+
+## Recommended Presentation Story
+
+Opening:
+
+> EmpathRAG started as emotion-aware RAG. Evaluation showed that open empathetic
+> generation can be vague, sycophantic, and weak on multi-turn escalation. We
+> redesigned it as EmpathRAG Core: a guarded conversational RAG support
+> navigator.
+
+Core contribution:
+
+1. Hybrid lexical + ML safety/routing architecture.
+2. Resource-registry grounded retrieval.
+3. Output-side safety checks.
+4. Multi-turn escalation evaluation.
+
+Demo flow:
+
+1. Failed exam / academic setback.
+2. ADS accommodations.
+3. Basic needs.
+4. Peer-helper crisis.
+5. Out-of-scope medical/legal.
+6. Ordinary social/date nerves or academic idiom false-positive resistance.
+
+Closing:
+
+> This is not therapy. It is a safer first-step support navigator: it helps a
+> student understand what kind of support they may need, gives one practical
+> next step, and escalates to human support when risk appears.
+
+## What Opus Should Review
+
+Please ask Opus to focus on:
+
+1. Is the product framing still too defensive?
+2. Should ordinary low-risk emotional support be a separate route, or handled
+   through message-sensitive templates?
+3. How much pipeline/debug visibility should remain in the MVP UI?
+4. What is the best Support Plan UI hierarchy?
+5. How should we evaluate conversational usefulness without making clinical
+   claims?
+6. Are Eval A and Eval B enough for a class presentation?
+7. What should be built next before voice/memory/Hugging Face deployment?
+
+## Git / Directory State
+
+Tracked documentation is organized under:
+
+- `docs/architecture/`
+- `docs/audits/`
+- `docs/demo/`
+- `docs/planning/`
+- `docs/research/`
+- `docs/team/karthik/`
+
+Ignored local artifacts intentionally remain untracked:
+
+- `Data_Karthik/`
+- `data/curated/` generated corpus/index files
+- `models/router/`
+- `venv/`
+- generated eval outputs
+
+Do not run broad destructive cleanup such as `git clean -fdX`, because it would
+remove useful local artifacts including data deliveries, router models, curated
+indexes, and the virtual environment.
+
+## Current Honest Status
+
+EmpathRAG Core is a strong class-project MVP and a plausible research prototype.
+It has a meaningful architecture, usable local app, synthetic evaluation data,
+and safety-focused evals.
+
+The next hard work is not more retrieval. It is product feel: make the user
+experience warm, conversational, and useful while keeping safety boundaries and
+source grounding intact.

docs/audits/README.md

@@ -0,0 +1,18 @@
+# EmpathRAG Audits
+
+Use this folder for project-state audits, dataset reviews, usability reviews,
+and handoff documents meant for outside reviewers.
+
+## Current Reviewer Handoff
+
+- [Full project audit for Opus](OPUS_FULL_PROJECT_AUDIT_2026_05_06.md)
+
+## Focused Audits
+
+- [Karthik Core Dataset V2 audit](KARTHIK_CORE_DATASET_V2_AUDIT.md)
+- [MVP usability audit](MVP_USABILITY_AUDIT_2026_05_06.md)
+
+## Rule
+
+When the project direction changes, add a new dated audit here instead of
+scattering loose Markdown files across the repository root.