eval/README_CORE_DATASET_V2_PIPELINE.md

EmpathRAG Core Dataset V2 Pipeline

Use this when Karthik sends empathrag_core_dataset_v2/.

Expected Delivery Folder

Place the folder at:

Data_Karthik\empathrag_core_dataset_v2

Required files:

• README_dataset_notes.md
• single_turn_labeled.csv
• multi_turn_scenarios.jsonl
• source_target_map.csv
• risky_ambiguous_cases.csv
• resource_profile_additions.csv

Ingest And Validate

.\venv\Scripts\python.exe -B eval\ingest_core_dataset_v2.py --delivery-dir Data_Karthik\empathrag_core_dataset_v2

Outputs:

• eval\empathrag_core_supervised.csv
• eval\multiturn_scenarios.jsonl
• eval\core_dataset_v2_ingest_report.json
• eval\core_dataset_v2_ingest_report.md

The script validates labels, required columns, duplicate IDs, multi-turn scenario shape, and resource-profile additions. It does not automatically merge resource additions into the runtime registry; those must be manually reviewed first.

Train Router

.\venv\Scripts\python.exe -B eval\train_ml_router.py

This writes local model artifacts under models\router\, which is ignored by git. If models are missing, the demo still falls back to deterministic routing.

Eval A: Single-Turn Router Ablation

.\venv\Scripts\python.exe -B eval\run_router_eval.py
.\venv\Scripts\python.exe -B eval\run_empathrag_core_eval.py

Primary metric: route accuracy.

Secondary metrics: safety tier accuracy, intercept accuracy, source hit rate, source avoid-list violations, unsafe generation, no-action responses, ungrounded action, and latency.

Eval B: Multi-Turn Safety Benchmark

.\venv\Scripts\python.exe -B eval\run_multiturn_eval.py

Primary metric: missed escalation rate.

Secondary metrics: dependency reinforcement, pure validation/no-action, unsafe generation, method leakage, and final safety tier correctness.

Smoke Test With Fixture

.\venv\Scripts\python.exe -B eval\ingest_core_dataset_v2.py `
  --delivery-dir eval\fixtures\core_dataset_v2_sample `
  --output eval\empathrag_core_supervised.sample.csv `
  --multiturn-output eval\multiturn_scenarios.sample.jsonl `
  --report-json eval\core_dataset_v2_ingest_report.sample.json `
  --report-md eval\core_dataset_v2_ingest_report.sample.md

This fixture is only for testing the ingest gate; it is not research data.