Dinegonos commited on
Commit
a6ba841
Β·
verified Β·
1 Parent(s): b38770a

Add comprehensive README with experiment design and literature review

Browse files
Files changed (1) hide show
  1. README.md +133 -0
README.md ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # SLM+RAG Anonymization for TRAM Threat Reports
2
+
3
+ ## Hypothesis
4
+ **Hβ‚€**: Anonymization of CTI threat reports via SLM (Small Language Model) + RAG does NOT significantly decrease the downstream ATT&CK technique classification accuracy.
5
+
6
+ **H₁**: Anonymization via SLM+RAG causes a statistically significant drop (>2% F1) in ATT&CK classification performance.
7
+
8
+ ## Experiment Architecture
9
+
10
+ ```
11
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
12
+ β”‚ EXPERIMENT PIPELINE β”‚
13
+ β”‚ β”‚
14
+ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
15
+ β”‚ β”‚ Raw CTI │───▢│ ATT&CK Classifier │──▢ F1_original β”‚
16
+ β”‚ β”‚ Report β”‚ β”‚ (SecureBERT) β”‚ β”‚
17
+ β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
18
+ β”‚ β”‚ β”‚
19
+ β”‚ β–Ό β”‚
20
+ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
21
+ β”‚ β”‚ SLM Anonymizer + RAG β”‚ β”‚
22
+ β”‚ β”‚ β”‚ β”‚
23
+ β”‚ β”‚ Step 1: NER Detection β”‚ β”‚
24
+ β”‚ β”‚ - GLiNER / SecBERT NER β”‚ β”‚
25
+ β”‚ β”‚ - Entity types: β”‚ β”‚
26
+ β”‚ β”‚ ORG, THREAT_ACTOR, β”‚ β”‚
27
+ β”‚ β”‚ MALWARE, TOOL, IP, β”‚ β”‚
28
+ β”‚ β”‚ LOC, CVE β”‚ β”‚
29
+ β”‚ β”‚ β”‚ β”‚
30
+ β”‚ β”‚ Step 2: RAG Context β”‚ β”‚
31
+ β”‚ β”‚ - ATT&CK KB embeddings β”‚ β”‚
32
+ β”‚ β”‚ - Guides what to β”‚ β”‚
33
+ β”‚ β”‚ preserve vs. mask β”‚ β”‚
34
+ β”‚ β”‚ β”‚ β”‚
35
+ β”‚ β”‚ Step 3: SLM Replacement β”‚ β”‚
36
+ β”‚ β”‚ - Typed placeholders β”‚ β”‚
37
+ β”‚ β”‚ - [MALWARE_1], etc. β”‚ β”‚
38
+ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
39
+ β”‚ β–Ό β”‚
40
+ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
41
+ β”‚ β”‚ Anonymized │───▢│ ATT&CK Classifier │──▢ F1_anon β”‚
42
+ β”‚ β”‚ CTI Report β”‚ β”‚ (same SecureBERT) β”‚ β”‚
43
+ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
44
+ β”‚ β”‚
45
+ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
46
+ β”‚ β”‚ EVALUATION β”‚ β”‚
47
+ β”‚ β”‚ - Ξ”F1 = F1_original - F1_anon β”‚ β”‚
48
+ β”‚ β”‚ - McNemar's test for statistical significance β”‚ β”‚
49
+ β”‚ β”‚ - Per-technique F1 comparison β”‚ β”‚
50
+ β”‚ β”‚ - Entity leakage rate β”‚ β”‚
51
+ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
52
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
53
+ ```
54
+
55
+ ## Anonymization Strategies (Ablation)
56
+
57
+ | Strategy ID | Method | Description |
58
+ |-------------|--------|-------------|
59
+ | `baseline` | None | No anonymization (control) |
60
+ | `placeholder` | NER β†’ Typed Placeholder | `APT29` β†’ `[THREAT_ACTOR_1]` |
61
+ | `slm_replace` | SLM generates synthetic replacements | `APT29` β†’ `ThreatGroup-Alpha` |
62
+ | `slm_rag` | SLM + RAG-guided anonymization | RAG retrieves ATT&CK context, SLM preserves behavioral terms |
63
+ | `full_redact` | Full entity redaction | `APT29` β†’ `[REDACTED]` |
64
+
65
+ ## Datasets
66
+
67
+ | Dataset | HF ID | Usage |
68
+ |---------|-------|-------|
69
+ | Security-TTP-Mapping | [`tumeteor/Security-TTP-Mapping`](https://huggingface.co/datasets/tumeteor/Security-TTP-Mapping) | Train/eval ATT&CK classifier |
70
+ | CTI-Bench (ATE) | [`AI4Sec/cti-bench`](https://huggingface.co/datasets/AI4Sec/cti-bench) config `cti-ate` | Eval benchmark |
71
+ | CTI-Bench (TAA) | [`AI4Sec/cti-bench`](https://huggingface.co/datasets/AI4Sec/cti-bench) config `cti-taa` | Natural anonymization baseline |
72
+ | AnnoCTR | [`priamai/AnnoCTR`](https://huggingface.co/datasets/priamai/AnnoCTR) | NER training data |
73
+
74
+ ## Models
75
+
76
+ | Component | Model | HF ID | Size |
77
+ |-----------|-------|-------|------|
78
+ | ATT&CK Classifier | SecureBERT | [`ehsanaghaei/SecureBERT`](https://huggingface.co/ehsanaghaei/SecureBERT) | 125M |
79
+ | ATT&CK Classifier v2 | SecureBERT 2.0 | [`cisco-ai/SecureBERT2.0-base`](https://huggingface.co/cisco-ai/SecureBERT2.0-base) | 149M |
80
+ | Semantic Ranker | SentSecBert | [`QCRI/SentSecBert_10k`](https://huggingface.co/QCRI/SentSecBert_10k) | ~110M |
81
+ | SLM Anonymizer | Foundation-Sec-8B | [`fdtn-ai/Foundation-Sec-8B-Instruct`](https://huggingface.co/fdtn-ai/Foundation-Sec-8B-Instruct) | 8B |
82
+ | NER Model | GLiNER | [`urchade/gliner_mediumv2.1`](https://huggingface.co/urchade/gliner_mediumv2.1) | 90M |
83
+
84
+ ## Quick Start
85
+
86
+ ### Phase 1: Regex-only anonymization (no GPU needed)
87
+ ```bash
88
+ python experiments/run_experiment.py \
89
+ --classifier-model ehsanaghaei/SecureBERT \
90
+ --epochs 5 \
91
+ --batch-size 16 \
92
+ --hub-model-id Dinegonos/securbert-ttp-classifier
93
+ ```
94
+
95
+ ### Phase 2: GLiNER NER + anonymization (GPU needed)
96
+ ```bash
97
+ python experiments/run_experiment.py \
98
+ --classifier-model ehsanaghaei/SecureBERT \
99
+ --use-gliner \
100
+ --hub-model-id Dinegonos/securbert-ttp-classifier
101
+ ```
102
+
103
+ ### Phase 3: Full SLM+RAG pipeline (A10G/A100 needed)
104
+ ```bash
105
+ python experiments/run_experiment.py \
106
+ --classifier-model ehsanaghaei/SecureBERT \
107
+ --use-gliner \
108
+ --use-slm-rag \
109
+ --slm-model fdtn-ai/Foundation-Sec-8B-Instruct \
110
+ --hub-model-id Dinegonos/securbert-ttp-classifier
111
+ ```
112
+
113
+ ## Key References
114
+
115
+ 1. **TRAM**: [github.com/center-for-threat-informed-defense/tram](https://github.com/center-for-threat-informed-defense/tram)
116
+ 2. **NCE Matching for TTP**: [arXiv:2401.10337](https://arxiv.org/abs/2401.10337) β€” F1@3=0.555 on TRAM
117
+ 3. **Privacy-Preserving NLP**: [arXiv:2306.05561](https://arxiv.org/abs/2306.05561) β€” NER-PS drops <0.4% F1
118
+ 4. **CTIBench**: [arXiv:2406.07599](https://arxiv.org/abs/2406.07599) β€” GPT-4 F1=0.639 on CTI-ATE
119
+ 5. **SecureBERT**: [arXiv:2204.02685](https://arxiv.org/abs/2204.02685)
120
+ 6. **SecureBERT 2.0**: [arXiv:2510.00240](https://arxiv.org/abs/2510.00240) (ModernBERT-based)
121
+ 7. **Foundation-Sec-8B**: [arXiv:2508.01059](https://arxiv.org/abs/2508.01059)
122
+ 8. **AnnoCTR**: [arXiv:2404.07765](https://arxiv.org/abs/2404.07765)
123
+ 9. **Adaptive Anonymization**: [arXiv:2602.20743](https://arxiv.org/abs/2602.20743)
124
+ 10. **LLM-in-the-Loop De-identification**: [arXiv:2412.10918](https://arxiv.org/abs/2412.10918)
125
+
126
+ ## Literature Evidence Supporting Hypothesis
127
+
128
+ | Study | Finding | Relevance |
129
+ |-------|---------|-----------|
130
+ | [arXiv:2306.05561](https://arxiv.org/abs/2306.05561) | NER-based pseudonymization drops classification F1 by only 0.27-0.36% | Strongest evidence for Hβ‚€ |
131
+ | [arXiv:2309.03057](https://arxiv.org/abs/2309.03057) | Hide-and-Seek framework maintains translation quality after anonymization | Architectural precedent |
132
+ | [arXiv:2412.10918](https://arxiv.org/abs/2412.10918) | Fine-tuned small NER models achieve F1=0.97+ for de-identification | SLM capability evidence |
133
+ | [arXiv:2411.01073](https://arxiv.org/abs/2411.01073) | RAG over ATT&CK KB achieves context recall ~0.85 | RAG effectiveness for ATT&CK |