Pankei/soc-narrative-grpo-budget512-qwen3-14b

SOC Narrative GRPO — Qwen3-14B (Step 32, Budget 512 — alternative run)

LoRA adapter for Qwen/Qwen3-14B trained with GRPO LoRA on 512 balanced user/day windows (CERT R4.2) (step 32).

Description

SOC Narrative is a framework for insider threat detection using small open-weight LLMs. A model receives a user/day window of events from the CERT Insider Threat Dataset R4.2 and must produce a structured response with:

• Risk label: normal, suspicious, or malicious
• Evidence: cited event IDs supporting the decision
• Reasoning: brief explanation of the investigation logic

This project explores whether small LLMs (3B–14B) can match or exceed traditional ML baselines for UEBA (User and Entity Behavior Analytics).

Metrics (dev_balanced_50)

Metric	Value
Accuracy	0.74
Macro F1	0.735
Recall Malicious	0.88
Valid Format Rate	0.64
Actionability Rate	0.64

Quick Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-14B"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "Pankei/soc-narrative-grpo-budget512-qwen3-14b")

tokenizer = AutoTokenizer.from_pretrained(base)
inputs = tokenizer("<your prompt>", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0]))

Note: This is a LoRA adapter (~30–160 MB). You need the full base model (Qwen/Qwen3-14B) to load it.

Training Details

• Base model: Qwen/Qwen3-14B
• Method: GRPO LoRA
• Train data: 512 balanced user/day windows (CERT R4.2)
• Checkpoint: step 32
• LoRA rank: 32, alpha: 64, target modules: q_proj, k_proj, v_proj, o_proj
• Format: Structured SOC Narrative (risk + evidence + reasoning)
• Hardware: NVIDIA H100 (80 GB)

Limitations

• Evaluated on a small balanced sample (n=50) — results may not generalize to production distributions
• Alternative GRPO run with different seed/hparams. Matches SFT recall (0.88) but with lower format compliance.
• Dataset is based on synthetic insider threat scenarios from CERT R4.2 (2016) — real-world performance may differ

Citation

@misc{soc-narrative-2026,
  author = {Research project},
  title = {SOC Narrative: Small LLMs for UEBA / Insider Threat Detection},
  year = {2026},
  howpublished = {\url{https://github.com/Pancake2021/research_work_by_a_student}}
}