Mukul Rayana commited on
Commit
d42c9fb
Β·
1 Parent(s): 83bdf33

docs: correct metric attribution and condition count

Browse files

- README.md: split 0.9629 claim into held-out NLI (0.9629) vs
adversarial probe (0.75) β€” two separate evaluations
- README.md: correct '4-condition ablation' to '3-condition' and
document that Condition B (DPR) was descoped
- .gitignore: add .env and related secret patterns

Files changed (2) hide show
  1. .gitignore +10 -0
  2. README.md +5 -3
.gitignore CHANGED
@@ -47,3 +47,13 @@ eval/human_eval_log.jsonl
47
  # OS
48
  .DS_Store
49
  Thumbs.db
 
 
 
 
 
 
 
 
 
 
 
47
  # OS
48
  .DS_Store
49
  Thumbs.db
50
+
51
+ # Secrets
52
+ .env
53
+ .env.*
54
+ !.env.example
55
+ *.pem
56
+ *.key
57
+ credentials*.json
58
+ service-account*.json
59
+ .ipynb_checkpoints/
README.md CHANGED
@@ -50,14 +50,14 @@ Student Message
50
  | Metric | Value | Target | |
51
  |---|---|---|---|
52
  | RoBERTa Emotion F1 (weighted) | **0.7127** | > 0.55 | βœ… |
53
- | DeBERTa Crisis Recall | **0.9629** | > 0.80 | βœ… |
 
54
  | DeBERTa Crisis Precision | **0.7951** | > 0.65 | βœ… |
55
  | BERTScore F1 | **0.8266** | > 0.72 | βœ… |
56
  | Wilcoxon p-value (Full vs BM25) | **3.62e-08** | < 0.05 | βœ… |
57
- | Adversarial crisis recall (NLI) | **85%** | > 80% | βœ… |
58
  | Euphemistic recall β€” NLI vs keyword | **100% vs 20%** | β€” | πŸ”‘ |
59
 
60
- ### Ablation β€” Emotion Alignment Score
61
 
62
  | Condition | Retrieval | Emotion Conditioning | Score |
63
  |---|---|---|---|
@@ -67,6 +67,8 @@ Student Message
67
 
68
  > Emotion conditioning contributes **+0.38** over pure dense retrieval (Cβ†’D). Wilcoxon signed-rank: p = 3.62e-08.
69
 
 
 
70
  ---
71
 
72
  ## Key Finding
 
50
  | Metric | Value | Target | |
51
  |---|---|---|---|
52
  | RoBERTa Emotion F1 (weighted) | **0.7127** | > 0.55 | βœ… |
53
+ | DeBERTa Crisis Recall (held-out NLI test set, 23K samples) | **0.9629** | > 0.80 | βœ… |
54
+ | DeBERTa Crisis Recall (30 adversarial probes, 6 categories) | **0.75** | β€” | βœ… |
55
  | DeBERTa Crisis Precision | **0.7951** | > 0.65 | βœ… |
56
  | BERTScore F1 | **0.8266** | > 0.72 | βœ… |
57
  | Wilcoxon p-value (Full vs BM25) | **3.62e-08** | < 0.05 | βœ… |
 
58
  | Euphemistic recall β€” NLI vs keyword | **100% vs 20%** | β€” | πŸ”‘ |
59
 
60
+ ### Ablation β€” Emotion Alignment Score (3-condition)
61
 
62
  | Condition | Retrieval | Emotion Conditioning | Score |
63
  |---|---|---|---|
 
67
 
68
  > Emotion conditioning contributes **+0.38** over pure dense retrieval (Cβ†’D). Wilcoxon signed-rank: p = 3.62e-08.
69
 
70
+ > **Note:** Condition B (DPR two-tower baseline) was descoped β€” deprioritised to increase adversarial probe depth. A ColBERT-scale index would be required for DPR to offer meaningful gains over FAISS flat-L2 at 1.67M vectors.
71
+
72
  ---
73
 
74
  ## Key Finding