Add comprehensive README with experiment design and literature review
Browse files
README.md
ADDED
|
@@ -0,0 +1,133 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SLM+RAG Anonymization for TRAM Threat Reports
|
| 2 |
+
|
| 3 |
+
## Hypothesis
|
| 4 |
+
**Hβ**: Anonymization of CTI threat reports via SLM (Small Language Model) + RAG does NOT significantly decrease the downstream ATT&CK technique classification accuracy.
|
| 5 |
+
|
| 6 |
+
**Hβ**: Anonymization via SLM+RAG causes a statistically significant drop (>2% F1) in ATT&CK classification performance.
|
| 7 |
+
|
| 8 |
+
## Experiment Architecture
|
| 9 |
+
|
| 10 |
+
```
|
| 11 |
+
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 12 |
+
β EXPERIMENT PIPELINE β
|
| 13 |
+
β β
|
| 14 |
+
β ββββββββββββββββ βββββββββββββββββββββββ β
|
| 15 |
+
β β Raw CTI βββββΆβ ATT&CK Classifier ββββΆ F1_original β
|
| 16 |
+
β β Report β β (SecureBERT) β β
|
| 17 |
+
β ββββββββ¬ββββββββ βββββββββββββββββββββββ β
|
| 18 |
+
β β β
|
| 19 |
+
β βΌ β
|
| 20 |
+
β ββββββββββββββββββββββββββββ β
|
| 21 |
+
β β SLM Anonymizer + RAG β β
|
| 22 |
+
β β β β
|
| 23 |
+
β β Step 1: NER Detection β β
|
| 24 |
+
β β - GLiNER / SecBERT NER β β
|
| 25 |
+
β β - Entity types: β β
|
| 26 |
+
β β ORG, THREAT_ACTOR, β β
|
| 27 |
+
β β MALWARE, TOOL, IP, β β
|
| 28 |
+
β β LOC, CVE β β
|
| 29 |
+
β β β β
|
| 30 |
+
β β Step 2: RAG Context β β
|
| 31 |
+
β β - ATT&CK KB embeddings β β
|
| 32 |
+
β β - Guides what to β β
|
| 33 |
+
β β preserve vs. mask β β
|
| 34 |
+
β β β β
|
| 35 |
+
β β Step 3: SLM Replacement β β
|
| 36 |
+
β β - Typed placeholders β β
|
| 37 |
+
β β - [MALWARE_1], etc. β β
|
| 38 |
+
β ββββββββββββ¬ββββββββββββββββ β
|
| 39 |
+
β βΌ β
|
| 40 |
+
β ββββββββββββββββ βββββββββββββββββββββββ β
|
| 41 |
+
β β Anonymized βββββΆβ ATT&CK Classifier ββββΆ F1_anon β
|
| 42 |
+
β β CTI Report β β (same SecureBERT) β β
|
| 43 |
+
β ββββββββββββββββ βββββββββββββββββββββββ β
|
| 44 |
+
β β
|
| 45 |
+
β βββββββββββββββββββββββββββββββββββββββββββββββββββ β
|
| 46 |
+
β β EVALUATION β β
|
| 47 |
+
β β - ΞF1 = F1_original - F1_anon β β
|
| 48 |
+
β β - McNemar's test for statistical significance β β
|
| 49 |
+
β β - Per-technique F1 comparison β β
|
| 50 |
+
β β - Entity leakage rate β β
|
| 51 |
+
β βββββββββββββββββββββββββββββββββββββββββββββββββββ β
|
| 52 |
+
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
## Anonymization Strategies (Ablation)
|
| 56 |
+
|
| 57 |
+
| Strategy ID | Method | Description |
|
| 58 |
+
|-------------|--------|-------------|
|
| 59 |
+
| `baseline` | None | No anonymization (control) |
|
| 60 |
+
| `placeholder` | NER β Typed Placeholder | `APT29` β `[THREAT_ACTOR_1]` |
|
| 61 |
+
| `slm_replace` | SLM generates synthetic replacements | `APT29` β `ThreatGroup-Alpha` |
|
| 62 |
+
| `slm_rag` | SLM + RAG-guided anonymization | RAG retrieves ATT&CK context, SLM preserves behavioral terms |
|
| 63 |
+
| `full_redact` | Full entity redaction | `APT29` β `[REDACTED]` |
|
| 64 |
+
|
| 65 |
+
## Datasets
|
| 66 |
+
|
| 67 |
+
| Dataset | HF ID | Usage |
|
| 68 |
+
|---------|-------|-------|
|
| 69 |
+
| Security-TTP-Mapping | [`tumeteor/Security-TTP-Mapping`](https://huggingface.co/datasets/tumeteor/Security-TTP-Mapping) | Train/eval ATT&CK classifier |
|
| 70 |
+
| CTI-Bench (ATE) | [`AI4Sec/cti-bench`](https://huggingface.co/datasets/AI4Sec/cti-bench) config `cti-ate` | Eval benchmark |
|
| 71 |
+
| CTI-Bench (TAA) | [`AI4Sec/cti-bench`](https://huggingface.co/datasets/AI4Sec/cti-bench) config `cti-taa` | Natural anonymization baseline |
|
| 72 |
+
| AnnoCTR | [`priamai/AnnoCTR`](https://huggingface.co/datasets/priamai/AnnoCTR) | NER training data |
|
| 73 |
+
|
| 74 |
+
## Models
|
| 75 |
+
|
| 76 |
+
| Component | Model | HF ID | Size |
|
| 77 |
+
|-----------|-------|-------|------|
|
| 78 |
+
| ATT&CK Classifier | SecureBERT | [`ehsanaghaei/SecureBERT`](https://huggingface.co/ehsanaghaei/SecureBERT) | 125M |
|
| 79 |
+
| ATT&CK Classifier v2 | SecureBERT 2.0 | [`cisco-ai/SecureBERT2.0-base`](https://huggingface.co/cisco-ai/SecureBERT2.0-base) | 149M |
|
| 80 |
+
| Semantic Ranker | SentSecBert | [`QCRI/SentSecBert_10k`](https://huggingface.co/QCRI/SentSecBert_10k) | ~110M |
|
| 81 |
+
| SLM Anonymizer | Foundation-Sec-8B | [`fdtn-ai/Foundation-Sec-8B-Instruct`](https://huggingface.co/fdtn-ai/Foundation-Sec-8B-Instruct) | 8B |
|
| 82 |
+
| NER Model | GLiNER | [`urchade/gliner_mediumv2.1`](https://huggingface.co/urchade/gliner_mediumv2.1) | 90M |
|
| 83 |
+
|
| 84 |
+
## Quick Start
|
| 85 |
+
|
| 86 |
+
### Phase 1: Regex-only anonymization (no GPU needed)
|
| 87 |
+
```bash
|
| 88 |
+
python experiments/run_experiment.py \
|
| 89 |
+
--classifier-model ehsanaghaei/SecureBERT \
|
| 90 |
+
--epochs 5 \
|
| 91 |
+
--batch-size 16 \
|
| 92 |
+
--hub-model-id Dinegonos/securbert-ttp-classifier
|
| 93 |
+
```
|
| 94 |
+
|
| 95 |
+
### Phase 2: GLiNER NER + anonymization (GPU needed)
|
| 96 |
+
```bash
|
| 97 |
+
python experiments/run_experiment.py \
|
| 98 |
+
--classifier-model ehsanaghaei/SecureBERT \
|
| 99 |
+
--use-gliner \
|
| 100 |
+
--hub-model-id Dinegonos/securbert-ttp-classifier
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
### Phase 3: Full SLM+RAG pipeline (A10G/A100 needed)
|
| 104 |
+
```bash
|
| 105 |
+
python experiments/run_experiment.py \
|
| 106 |
+
--classifier-model ehsanaghaei/SecureBERT \
|
| 107 |
+
--use-gliner \
|
| 108 |
+
--use-slm-rag \
|
| 109 |
+
--slm-model fdtn-ai/Foundation-Sec-8B-Instruct \
|
| 110 |
+
--hub-model-id Dinegonos/securbert-ttp-classifier
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
## Key References
|
| 114 |
+
|
| 115 |
+
1. **TRAM**: [github.com/center-for-threat-informed-defense/tram](https://github.com/center-for-threat-informed-defense/tram)
|
| 116 |
+
2. **NCE Matching for TTP**: [arXiv:2401.10337](https://arxiv.org/abs/2401.10337) β F1@3=0.555 on TRAM
|
| 117 |
+
3. **Privacy-Preserving NLP**: [arXiv:2306.05561](https://arxiv.org/abs/2306.05561) β NER-PS drops <0.4% F1
|
| 118 |
+
4. **CTIBench**: [arXiv:2406.07599](https://arxiv.org/abs/2406.07599) β GPT-4 F1=0.639 on CTI-ATE
|
| 119 |
+
5. **SecureBERT**: [arXiv:2204.02685](https://arxiv.org/abs/2204.02685)
|
| 120 |
+
6. **SecureBERT 2.0**: [arXiv:2510.00240](https://arxiv.org/abs/2510.00240) (ModernBERT-based)
|
| 121 |
+
7. **Foundation-Sec-8B**: [arXiv:2508.01059](https://arxiv.org/abs/2508.01059)
|
| 122 |
+
8. **AnnoCTR**: [arXiv:2404.07765](https://arxiv.org/abs/2404.07765)
|
| 123 |
+
9. **Adaptive Anonymization**: [arXiv:2602.20743](https://arxiv.org/abs/2602.20743)
|
| 124 |
+
10. **LLM-in-the-Loop De-identification**: [arXiv:2412.10918](https://arxiv.org/abs/2412.10918)
|
| 125 |
+
|
| 126 |
+
## Literature Evidence Supporting Hypothesis
|
| 127 |
+
|
| 128 |
+
| Study | Finding | Relevance |
|
| 129 |
+
|-------|---------|-----------|
|
| 130 |
+
| [arXiv:2306.05561](https://arxiv.org/abs/2306.05561) | NER-based pseudonymization drops classification F1 by only 0.27-0.36% | Strongest evidence for Hβ |
|
| 131 |
+
| [arXiv:2309.03057](https://arxiv.org/abs/2309.03057) | Hide-and-Seek framework maintains translation quality after anonymization | Architectural precedent |
|
| 132 |
+
| [arXiv:2412.10918](https://arxiv.org/abs/2412.10918) | Fine-tuned small NER models achieve F1=0.97+ for de-identification | SLM capability evidence |
|
| 133 |
+
| [arXiv:2411.01073](https://arxiv.org/abs/2411.01073) | RAG over ATT&CK KB achieves context recall ~0.85 | RAG effectiveness for ATT&CK |
|