Mukul Rayana commited on
Commit Β·
d42c9fb
1
Parent(s): 83bdf33
docs: correct metric attribution and condition count
Browse files- README.md: split 0.9629 claim into held-out NLI (0.9629) vs
adversarial probe (0.75) β two separate evaluations
- README.md: correct '4-condition ablation' to '3-condition' and
document that Condition B (DPR) was descoped
- .gitignore: add .env and related secret patterns
- .gitignore +10 -0
- README.md +5 -3
.gitignore
CHANGED
|
@@ -47,3 +47,13 @@ eval/human_eval_log.jsonl
|
|
| 47 |
# OS
|
| 48 |
.DS_Store
|
| 49 |
Thumbs.db
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
# OS
|
| 48 |
.DS_Store
|
| 49 |
Thumbs.db
|
| 50 |
+
|
| 51 |
+
# Secrets
|
| 52 |
+
.env
|
| 53 |
+
.env.*
|
| 54 |
+
!.env.example
|
| 55 |
+
*.pem
|
| 56 |
+
*.key
|
| 57 |
+
credentials*.json
|
| 58 |
+
service-account*.json
|
| 59 |
+
.ipynb_checkpoints/
|
README.md
CHANGED
|
@@ -50,14 +50,14 @@ Student Message
|
|
| 50 |
| Metric | Value | Target | |
|
| 51 |
|---|---|---|---|
|
| 52 |
| RoBERTa Emotion F1 (weighted) | **0.7127** | > 0.55 | β
|
|
| 53 |
-
| DeBERTa Crisis Recall | **0.9629** | > 0.80 | β
|
|
|
|
|
| 54 |
| DeBERTa Crisis Precision | **0.7951** | > 0.65 | β
|
|
| 55 |
| BERTScore F1 | **0.8266** | > 0.72 | β
|
|
| 56 |
| Wilcoxon p-value (Full vs BM25) | **3.62e-08** | < 0.05 | β
|
|
| 57 |
-
| Adversarial crisis recall (NLI) | **85%** | > 80% | β
|
|
| 58 |
| Euphemistic recall β NLI vs keyword | **100% vs 20%** | β | π |
|
| 59 |
|
| 60 |
-
### Ablation β Emotion Alignment Score
|
| 61 |
|
| 62 |
| Condition | Retrieval | Emotion Conditioning | Score |
|
| 63 |
|---|---|---|---|
|
|
@@ -67,6 +67,8 @@ Student Message
|
|
| 67 |
|
| 68 |
> Emotion conditioning contributes **+0.38** over pure dense retrieval (CβD). Wilcoxon signed-rank: p = 3.62e-08.
|
| 69 |
|
|
|
|
|
|
|
| 70 |
---
|
| 71 |
|
| 72 |
## Key Finding
|
|
|
|
| 50 |
| Metric | Value | Target | |
|
| 51 |
|---|---|---|---|
|
| 52 |
| RoBERTa Emotion F1 (weighted) | **0.7127** | > 0.55 | β
|
|
| 53 |
+
| DeBERTa Crisis Recall (held-out NLI test set, 23K samples) | **0.9629** | > 0.80 | β
|
|
| 54 |
+
| DeBERTa Crisis Recall (30 adversarial probes, 6 categories) | **0.75** | β | β
|
|
| 55 |
| DeBERTa Crisis Precision | **0.7951** | > 0.65 | β
|
|
| 56 |
| BERTScore F1 | **0.8266** | > 0.72 | β
|
|
| 57 |
| Wilcoxon p-value (Full vs BM25) | **3.62e-08** | < 0.05 | β
|
|
|
|
|
| 58 |
| Euphemistic recall β NLI vs keyword | **100% vs 20%** | β | π |
|
| 59 |
|
| 60 |
+
### Ablation β Emotion Alignment Score (3-condition)
|
| 61 |
|
| 62 |
| Condition | Retrieval | Emotion Conditioning | Score |
|
| 63 |
|---|---|---|---|
|
|
|
|
| 67 |
|
| 68 |
> Emotion conditioning contributes **+0.38** over pure dense retrieval (CβD). Wilcoxon signed-rank: p = 3.62e-08.
|
| 69 |
|
| 70 |
+
> **Note:** Condition B (DPR two-tower baseline) was descoped β deprioritised to increase adversarial probe depth. A ColBERT-scale index would be required for DPR to offer meaningful gains over FAISS flat-L2 at 1.67M vectors.
|
| 71 |
+
|
| 72 |
---
|
| 73 |
|
| 74 |
## Key Finding
|