File size: 2,383 Bytes
e143b4a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | # EmpathRAG Core Dataset V2 Pipeline
Use this when Karthik sends `empathrag_core_dataset_v2/`.
## Expected Delivery Folder
Place the folder at:
```powershell
Data_Karthik\empathrag_core_dataset_v2
```
Required files:
- `README_dataset_notes.md`
- `single_turn_labeled.csv`
- `multi_turn_scenarios.jsonl`
- `source_target_map.csv`
- `risky_ambiguous_cases.csv`
- `resource_profile_additions.csv`
## Ingest And Validate
```powershell
.\venv\Scripts\python.exe -B eval\ingest_core_dataset_v2.py --delivery-dir Data_Karthik\empathrag_core_dataset_v2
```
Outputs:
- `eval\empathrag_core_supervised.csv`
- `eval\multiturn_scenarios.jsonl`
- `eval\core_dataset_v2_ingest_report.json`
- `eval\core_dataset_v2_ingest_report.md`
The script validates labels, required columns, duplicate IDs, multi-turn scenario
shape, and resource-profile additions. It does not automatically merge resource
additions into the runtime registry; those must be manually reviewed first.
## Train Router
```powershell
.\venv\Scripts\python.exe -B eval\train_ml_router.py
```
This writes local model artifacts under `models\router\`, which is ignored by
git. If models are missing, the demo still falls back to deterministic routing.
## Eval A: Single-Turn Router Ablation
```powershell
.\venv\Scripts\python.exe -B eval\run_router_eval.py
.\venv\Scripts\python.exe -B eval\run_empathrag_core_eval.py
```
Primary metric: route accuracy.
Secondary metrics: safety tier accuracy, intercept accuracy, source hit rate,
source avoid-list violations, unsafe generation, no-action responses,
ungrounded action, and latency.
## Eval B: Multi-Turn Safety Benchmark
```powershell
.\venv\Scripts\python.exe -B eval\run_multiturn_eval.py
```
Primary metric: missed escalation rate.
Secondary metrics: dependency reinforcement, pure validation/no-action,
unsafe generation, method leakage, and final safety tier correctness.
## Smoke Test With Fixture
```powershell
.\venv\Scripts\python.exe -B eval\ingest_core_dataset_v2.py `
--delivery-dir eval\fixtures\core_dataset_v2_sample `
--output eval\empathrag_core_supervised.sample.csv `
--multiturn-output eval\multiturn_scenarios.sample.jsonl `
--report-json eval\core_dataset_v2_ingest_report.sample.json `
--report-md eval\core_dataset_v2_ingest_report.sample.md
```
This fixture is only for testing the ingest gate; it is not research data.
|