docs/research/REPRODUCIBILITY.md

Reproducibility

What a third-party reviewer or fork can and cannot reproduce from this repo, and the exact commands to do so.

TL;DR

Evaluation	Inputs in repo?	Reproducible?
Eval B (74 multi-turn scenarios)	✓ eval/multiturn_scenarios.jsonl + eval/multiturn_safety_supplement.jsonl	✓ Full reproduction
Per-layer ablation (5 variants × Eval B)	✓ Same inputs	✓ Full reproduction
Unguarded Llama baseline (Eval B)	✓ Same inputs	✓ Full reproduction (requires Groq API key)
Drift sweep (29 cells × 14 routes × 3 stages)	✓ Probes hard-coded in script	✓ Full reproduction (requires Groq)
F-1 stage × ISSS contract (12 cells)	✓ Probes hard-coded	✓ Full reproduction (requires Groq)
Sycophancy probes (25 cells)	✓ Probes hard-coded	✓ Full reproduction (requires Groq)
Prompt-injection probes (16 cells)	✓ Probes hard-coded	✓ Full reproduction (requires Groq)
Fairness spot-check (18 paired prompts)	✓ Probes hard-coded	✓ Full reproduction (requires Groq)
Eval A (360 single-turn)	⚠ Requires Karthik's V2 dataset under Data_Karthik/ (untracked by design — teammate's delivery)	⚠ Inputs needed; obtain from co-author
ML router training	✗ Trained artifacts under models/router/ are intentionally untracked	⚠ Re-train from Karthik's labeled CSV via eval/train_ml_router.py
Curated retrieval index	✗ FAISS index under data/curated/indexes/ is intentionally untracked	⚠ Re-build via src/data/build_curated_index.py from data/curated/resources_seed.jsonl

Commands

After cloning + venv setup + .env with GROQ_API_KEY (and optionally ANTHROPIC_API_KEY):

# Headline safety eval: 0 / 28 missed escalations (rephraser ON)
EMPATHRAG_REPHRASER_ENABLED=1 ./venv/Scripts/python.exe eval/run_multiturn_eval.py

# Same-model unguarded baseline: 9 / 28 missed escalations
./venv/Scripts/python.exe eval/run_unguarded_baseline.py

# Per-layer ablation: ~20 min, ~$1 in Groq calls
EMPATHRAG_REPHRASER_ENABLED=1 ./venv/Scripts/python.exe eval/run_ablation_eval.py

# Targeted failure-mode sweeps (each ~3-5 min, ~$0.05)
./venv/Scripts/python.exe eval/sweep_rephraser_drift.py
./venv/Scripts/python.exe eval/sweep_f1_stage_isss.py
./venv/Scripts/python.exe eval/sweep_sycophancy_probes.py
./venv/Scripts/python.exe eval/sweep_prompt_injection.py
./venv/Scripts/python.exe eval/sweep_fairness_spot_check.py

# Resource URL health
./venv/Scripts/python.exe eval/audit_resource_urls.py

# Regression suite
./venv/Scripts/python.exe -m pytest tests/test_v25_support_navigator.py -v

Expected results at commit 847587d (or newer)

Eval	Headline
Eval B (rephraser ON)	0 / 28 missed escalation, 0 unsafe, 0 ungrounded, ~570 ms avg latency
Ablation (rephraser ON)	baseline 0 / 28; no_stage1_precheck 22 / 28; others 0 / 28
Unguarded Llama 3.3 70B	9 / 28 missed escalation (CI95 [0.148, 0.494]), 2 harm-endorsement turns
Drift sweep	27-29 / 29 clean (1-2 stochastic LLM-level flags within tolerance)
F-1 stage × ISSS contract	12 / 12 pass
Sycophancy probes	25 / 25 clean
Prompt-injection probes	16 / 16 clean
Fairness spot-check	18 / 18 no divergence
Resource URL audit	60 / 63 live (3 SAMHSA TLS quirks are not real outages)
Regression tests	21 / 21 pass

Numbers from stochastic-LLM evaluations will vary turn-to-turn within the tolerances above. Deterministic-template evaluations are stable across runs.

What's intentionally untracked

• Data_Karthik/ — teammate dataset deliveries. Not the repo's to redistribute.
• data/curated/indexes/ — FAISS index + SQLite metadata, regenerable from resources_seed.jsonl.
• models/router/ — ML router artifacts (TF-IDF vectorizer + logistic regression), regenerable via eval/train_ml_router.py once the labeled dataset is present.
• .env — API keys.
• Generated eval reports under eval/sweep_*.md, eval/ablation_*.md, eval/unguarded_*.md, eval/audit_*.md — regenerable from the scripts.

See .gitignore for the exhaustive list and the rationale per pattern.

Versioning and dataset provenance

This evaluation set is based on Karthik's V2 delivery (Data_Karthik/empathrag_core_dataset_v2/, ingested 2026-04-30, 216/72/72 train/dev/test split for Eval A; 50 + 24 multi-turn = 74 scenarios for Eval B). The next data pull (docs/team/karthik/KARTHIK_DATA_REQUEST_V4.md) extends with authority-misconduct scenarios, sycophancy probes, topic-shift scenarios, incomplete-message scenarios, and real anonymized student turns.

Honest caveats

• n = 28 escalation scenarios is small enough that absolute claims like "0% missed escalation in deployment" are not warranted by this evaluation alone. The meaningful claim is the comparison against the unguarded baseline (0/28 vs 9/28, non-overlapping CIs).
• Synthetic data structurally differs from real student phrasing. Numbers here are prototype evidence, not deployment claims.
• Stochastic LLM components mean drift / sycophancy / injection sweeps will vary by 1-2 cells run-to-run. The patterns are stable; individual cells are not.
• Provider availability affects all rephraser-ON results. Groq Llama 3.3 70B is the primary path; Anthropic Haiku 4.5 is the fallback. With both unavailable, the system falls through to deterministic templates and the rephraser metrics become identical to the deterministic baseline.