cti-ner-gliner

GLiNER (Zaratiana et al. 2024, zero-shot span-matching NER) fine-tuned for named entity recognition on Cyber Threat Intelligence (CTI) text — specifically CISA cybersecurity advisories. Part of a capstone project comparing a zero-shot span-matching architecture against token-classification encoders (RoBERTa-base, RoBERTa-large, SecureBERT) on the same clean, LLM-annotated gold corpus.

This is currently the weakest-performing model in that comparison — see Evaluation below. An earlier, uncontrolled comparison (an old GLiNER checkpoint trained on a prior, noisier weak-supervision corpus, scored against this same honest gold) suggested GLiNER might generalize better than the token classifiers. That did not hold once GLiNER was retrained on the identical gold the token classifiers use and scored with the identical harness — the earlier result was a training-data confound, not an architecture effect. Reported here because the negative result is as informative as a positive one would have been.

Code and training pipeline: dandye/nlp_capstone

Labels

16 CTI entity types:

affected software product, attack technique id, attack technique name, cve identifier, cwe identifier, email address, file hash, file name, file path, hacking tool, ip address, malware family, tactic, threat actor group, tlp classification, yara rule name

Training data

The gold corpus was built with an LLM-primary annotation pipeline (Claude Sonnet, exhaustive extraction gated by verbatim + format validators), replacing an earlier MITRE/STIX dictionary-based weak-supervision approach after a human audit found the dictionary labels missed roughly half of real entities and mislabeled about 10%. Source text is public-domain CISA advisories. 66 advisories train / 8 validation / 8 test, split at the document level (same split used for the RoBERTa/SecureBERT comparison models, so results are directly comparable).

Evaluation

Scored on the same 8-document held-out test set as the token-classification models, with a document-level span harness (strict = exact (start, end, label) match; relaxed = overlapping span + matching label).

Metric	Strict	Relaxed
Micro F1	0.56	0.58
Macro F1	0.62	0.64

For context, on the same test set and gold corpus:

Model	Strict Micro F1	Strict Macro F1
roberta-base	0.659	0.711
roberta-large	0.648	0.727
SecureBERT	0.712	0.738
GLiNER (this model)	0.560	0.62

GLiNER is competitive on structured, dictionary-shaped labels (attack technique id 0.99, cve identifier 0.96, cwe identifier 1.00) but noticeably weaker than the token classifiers on most other labels, and collapses on ip address specifically (F1 0.03 vs. 0.85–0.92 for the token classifiers). Part of that gap may be an evaluation-harness artifact rather than a pure architecture effect: unlike the token classifiers (which use overlapping sliding windows), GLiNER truncates input at 384 tokens, and several of the longest test documents were truncated during scoring — this is a documented, unverified caveat, not a confirmed cause. The 8-document test set is small; treat these numbers as indicative, not precise.

Usage

from gliner import GLiNER

model = GLiNER.from_pretrained("pzryathzsdhc/cti-ner-gliner")

labels = [
    "affected software product", "attack technique id", "attack technique name",
    "cve identifier", "cwe identifier", "email address", "file hash", "file name",
    "file path", "hacking tool", "ip address", "malware family", "tactic",
    "threat actor group", "tlp classification", "yara rule name",
]

text = "The threat actor group APT29 deployed a malware family known as WINELOADER, exploiting CVE-2024-21887."
entities = model.predict_entities(text, labels, threshold=0.4)
for e in entities:
    print(e["text"], "=>", e["label"])

Training details

• Base model: urchade/gliner_medium-v2.1
• 5 epochs, batch size 8, gradient accumulation 4 (effective batch 32), LR 5e-5, linear schedule, warmup ratio 0.1
• Best checkpoint selected by validation loss (GLiNER's trainer reports no span-level F1 during training)
• Trained on CPU — MPS leaks memory on this architecture's dynamic shapes and OOMs mid-epoch on Apple Silicon; CPU is slower but reliable
• ~3h45m on Apple Silicon (CPU)