gliner-ettin-68m-ptbr-pii-full-3x-v1

GLiNER fine-tune of jhu-clsp/ettin-encoder-68m (~68M params) for Brazilian-Portuguese PII detection, with cross-source generalization tested on English/multilingual PII and spam corpora.

This is part of a 6-config sweep (32M × 68M) × (top-50k curated, top-100k curated, full ~984k), trained on the same MI300X GPU under identical recipe except for batch size and step budget. See Comparison across the sweep below.

Quick start

from gliner import GLiNER

model = GLiNER.from_pretrained("arthrod/gliner-ettin-68m-ptbr-pii-full-3x-v1", subfolder="checkpoint-41400")

text = (
    "Sou Maria Silva, moro na Rua das Flores, 123, em São Paulo. "
    "Meu CPF é 123.456.789-09 e meu telefone é (11) 91234-5678."
)

labels = [
    "first name", "last name", "middle name",
    "cpf document number", "rg document number", "pis document number",
    "phone number", "email address", "credit card",
    "location street", "location building number", "location neighborhood",
    "location city", "location state", "location state abbreviation",
    "location zip", "location full address",
    "dob",
]

for ent in model.predict_entities(text, labels, threshold=0.3):
    print(ent["text"], "→", ent["label"], f"({ent['score']:.2f})")

Best checkpoint by PT-BR partial F1 = checkpoint-41400. All other checkpoints (every eval_every step) are also kept in this repo for ablation.

Training recipe

• base encoder: jhu-clsp/ettin-encoder-68m

• hidden_size: 512, max_len: 1024, max_width: 100, span_mode: token_level

• training data: data/splits/train.jsonl

• val data: data/splits/val_5k.jsonl

• num_steps: 69000, train_batch_size: 128, eval_batch_size: 128

• lr_encoder: 1.5e-05, lr_others: 5e-5

• weight_decay: encoder=0.01, other=0.01

• scheduler: cosine, warmup_ratio: 0.1, max_grad_norm: 1.0

• focal loss: alpha=0.75, gamma=2.0, reduction=mean

• precision: bf16 (HIPBLASLT_ALLOW_TF32=0 on MI300X)

• dropout: 0.3, fine_tune: True, subtoken_pooling: first

• training data summary: data/splits/train.jsonl (~984k samples, 3x step budget)

The sweep uses the same loss recipe (focal α=0.75, γ=2.0, mean reduction), bf16 with HIPBLASLT_ALLOW_TF32=0 (avoids MI300X NaN intermittency), and a cosine schedule with 10% warmup. The local gliner source is patched against an upstream token_level regression that landed in 0.2.26 — this checkpoint is trained against gliner==0.2.25 plus that fix.

Evaluation protocol

Every eval_every steps a 5,000-sample, proportional, source-mixed holdout (data/holdout_5k_original) is re-scored. The holdout was sliced from the upstream PII / spam datasets before the train split was assembled, so there is no leakage. For each row we build a per-source label superset (canonical 24 PT-BR labels ∪ that row's gold labels, capped at 100, lowercased) and call model.inference(..., flat_ner=True, multi_label=False, threshold=0.3). Per-source F1 is computed by nervaluate (strict / exact / partial / ent_type).

Eleven evaluation sources cover PT-BR PII, EN PII, and negative-evidence spam/phishing:

• gliner2_pii_ptbr_reward_split — PT-BR PII (the target)
• nemotron_pii — nvidia/Nemotron-PII, EN
• open_pii_masking_500k, pii_masking_400k — multilingual PII
• enron_spam_*, phishing_*, sms_spam_multilingual, spam_messages_mshenoda, spamassassin — negative evidence (no PII expected → tests false-positive rate)

Best-checkpoint scores (checkpoint-41400)

Per-source aggregate F1 on the 11-source proportional 5k holdout:

eval source	strict F1	partial F1
PT-BR (gliner2 reward-split)	0.7076	0.7979
EN (nvidia/Nemotron-PII)	0.6149	0.6799
open-pii-masking-500k	0.4902	0.6464
pii-masking-400k	0.4701	0.6024
enron-spam (bvk)	0.0000	0.0000
enron-spam (setfit)	0.0000	0.0000
phishing (darkknight)	0.0000	0.0000
phishing (zefang)	0.0000	0.0000
SMS spam (multilingual)	0.0000	0.0000
spam-messages (mshenoda)	0.0000	0.0000
spamassassin	0.0000	0.0000

Per-entity breakdown (4 PII sources)

Averaged only across sources that contain the label (sources where the label is absent in gold are excluded — including them would falsely deflate the aggregate, e.g. personal description of ethnicity averaged with two sources that lack it would drop from ~0.58 to ~0.29 partial F1). n_src is the number of contributing sources. Sorted by partial F1.

entity	n_src	strict P	strict R	strict F1	partial P	partial R	partial F1
ipv4	1	1.000	1.000	1.000	1.000	1.000	1.000
email address	4	0.975	0.970	0.972	0.982	0.977	0.980
device identifier	1	1.000	0.955	0.977	1.000	0.955	0.977
url	1	0.915	0.896	0.905	0.957	0.938	0.947
pis document number	1	0.974	0.881	0.925	0.974	0.881	0.925
rg document number	1	0.869	0.960	0.912	0.869	0.960	0.912
phone number	4	0.864	0.926	0.892	0.873	0.936	0.901
date time	1	0.952	0.855	0.901	0.952	0.855	0.901
biometric identifier	1	0.944	0.838	0.887	0.944	0.838	0.887
date	2	0.817	0.924	0.860	0.833	0.941	0.876
mac address	1	0.960	0.800	0.873	0.960	0.800	0.873
coordinate	1	0.746	0.800	0.772	0.831	0.891	0.860
bank routing number	1	0.870	0.839	0.855	0.870	0.839	0.855
medical record number	1	0.760	0.973	0.854	0.760	0.973	0.854
cpf document number	1	0.828	0.837	0.832	0.828	0.837	0.832
county	1	0.727	0.899	0.804	0.745	0.921	0.824
vehicle identifier	1	0.698	1.000	0.822	0.698	1.000	0.822
location full address	1	0.736	0.709	0.722	0.835	0.805	0.819
health plan beneficiary number	1	0.716	0.914	0.803	0.730	0.931	0.818
fax number	1	0.964	0.692	0.806	0.964	0.692	0.806
country	1	0.691	0.938	0.795	0.694	0.942	0.799
time	2	0.692	0.893	0.768	0.708	0.918	0.787
swift bic	1	0.625	1.000	0.769	0.625	1.000	0.769
credit card	4	0.818	0.733	0.760	0.824	0.738	0.765
customer id	1	0.663	0.887	0.759	0.663	0.887	0.759
dob	3	0.811	0.761	0.746	0.819	0.774	0.756
pin	1	0.719	0.793	0.754	0.719	0.793	0.754
cvv	1	0.632	0.857	0.727	0.632	0.857	0.727
certificate license number	1	0.652	0.811	0.723	0.652	0.811	0.723
language	1	0.571	0.973	0.720	0.571	0.973	0.720
account number	2	0.707	0.695	0.699	0.707	0.695	0.699
blood type	1	0.561	0.719	0.630	0.622	0.797	0.699
license plate	1	0.553	0.750	0.636	0.605	0.821	0.697
personal description of organizational affiliation	1	0.463	0.500	0.481	0.659	0.711	0.684
location street	3	0.566	0.653	0.603	0.639	0.740	0.682
personal description of sexual information	1	0.450	0.494	0.471	0.648	0.711	0.678
location building number	3	0.535	0.917	0.671	0.536	0.920	0.673
employee id	1	0.510	0.974	0.670	0.510	0.974	0.670
location zip	4	0.570	0.832	0.659	0.576	0.843	0.667
location state	2	0.494	0.905	0.637	0.496	0.909	0.639
ipv6	1	0.643	0.409	0.500	0.821	0.523	0.639
religious belief	1	0.526	0.714	0.606	0.553	0.750	0.636
last name	4	0.526	0.726	0.603	0.552	0.765	0.635
personal description of political opinion	1	0.407	0.495	0.447	0.574	0.698	0.630
company name	1	0.466	0.672	0.551	0.533	0.768	0.629
tax id number	3	0.651	0.568	0.601	0.679	0.584	0.621
personal description of religious convictions	1	0.414	0.538	0.468	0.539	0.701	0.610
sex or gender	2	0.444	0.875	0.587	0.444	0.875	0.587
personal description of ethnicity	2	0.422	0.726	0.527	0.468	0.795	0.582
first name	4	0.438	0.830	0.555	0.456	0.868	0.579
location neighborhood	1	0.435	0.702	0.537	0.462	0.746	0.570
http cookie	1	0.304	0.241	0.269	0.630	0.500	0.558
location city	4	0.410	0.737	0.525	0.428	0.771	0.548
api key	1	0.387	0.763	0.513	0.407	0.803	0.540
id card number	2	0.448	0.660	0.529	0.453	0.668	0.535
social security number	3	0.665	0.496	0.530	0.669	0.498	0.533
user name	2	0.413	0.658	0.507	0.421	0.672	0.517
age	2	0.413	0.791	0.514	0.413	0.791	0.514
personal description of medical conditions	1	0.279	0.439	0.341	0.413	0.652	0.506
password	2	0.282	0.321	0.299	0.453	0.531	0.487
middle name	1	0.304	0.789	0.439	0.304	0.789	0.439
employment status	1	0.304	0.562	0.394	0.315	0.582	0.409
driver license number	2	0.605	0.323	0.368	0.612	0.330	0.375
location state abbreviation	1	0.212	1.000	0.349	0.212	1.000	0.349
education level	1	0.124	0.245	0.165	0.248	0.491	0.329
unique id	1	0.190	0.571	0.286	0.190	0.571	0.286
passport number	1	0.400	0.200	0.267	0.400	0.200	0.267
sexuality	1	0.153	0.667	0.248	0.153	0.667	0.248
political view	1	0.107	0.692	0.186	0.119	0.769	0.206
title	1	0.111	0.582	0.187	0.117	0.609	0.196
occupation	1	0.040	0.232	0.068	0.075	0.435	0.128

Reading guide. Structured tokens (email, phone, doc numbers, IPs, MAC) approach strict ≈ partial because their boundaries are unambiguous. Long natural-language spans (full address, the personal description of … labels) carry a meaningful strict-vs-partial gap because exact boundaries are inherently fuzzy — even two human annotators would disagree. For those labels, partial F1 is the operationally meaningful metric.

Sweep over checkpoints

20 eval points (every eval_every steps). Higher is better.

step	PT-BR strict	PT-BR partial	Nemotron partial	open-pii partial	pii-masking partial
3450	0.0435	0.3185	0.1783	0.3494	0.3067
6900	0.1928	0.4170	0.2571	0.4157	0.3310
10350	0.2774	0.4642	0.3307	0.4036	0.3302
13800	0.3107	0.4872	0.3371	0.3994	0.3343
17250	0.4126	0.6059	0.3688	0.3910	0.3869
20700	0.4838	0.6549	0.4791	0.4424	0.4135
24150	0.4802	0.6348	0.6219	0.4534	0.4592
27600	0.5864	0.7220	0.5603	0.4536	0.4289
31050	0.5480	0.6731	0.5524	0.4509	0.4876
34500	0.5921	0.6957	0.6209	0.4843	0.5082
37950	0.6329	0.7441	0.6189	0.5580	0.5634
41400	0.7076	0.7979	0.6799	0.6464	0.6024
44850	0.6747	0.7687	0.6522	0.6310	0.5855
48300	0.6994	0.7859	0.6205	0.6253	0.5899
51750	0.6312	0.7073	0.6697	0.6222	0.5806
55200	0.6658	0.7415	0.6109	0.6391	0.6015
58650	0.6408	0.7147	0.6031	0.6187	0.5514
62100	0.6765	0.7504	0.6239	0.6448	0.5808
65550	0.6724	0.7475	0.6172	0.6419	0.5827
69000	0.6701	0.7448	0.6147	0.6415	0.5845

Comparison across the sweep (best by PT-BR partial F1 of each run)

ettin-32m top-50k       : PT-BR strict 0.1411  partial 0.3710  Nemotron partial 0.1710  cross-avg 0.2808  (step 8500)
ettin-32m top-100k      : PT-BR strict 0.0946  partial 0.3165  Nemotron partial 0.1455  cross-avg 0.2428  (step 5200)
ettin-68m top-50k       : PT-BR strict 0.2732  partial 0.4802  Nemotron partial 0.2397  cross-avg 0.3117  (step 9945)
ettin-68m top-100k      : PT-BR strict 0.4109  partial 0.6191  Nemotron partial 0.3046  cross-avg 0.4497  (step 8580)
ettin-68m full          : PT-BR strict 0.3663  partial 0.5416  Nemotron partial 0.3563  cross-avg 0.4239  (step 19550)
ettin-32m full          : PT-BR strict 0.1429  partial 0.3536  Nemotron partial 0.2640  cross-avg 0.2867  (step 19550)
ettin-32m full-3x       : PT-BR strict 0.6326  partial 0.7167  Nemotron partial 0.5721  cross-avg 0.6031  (step 65550)
ettin-68m full-3x       : PT-BR strict 0.7076  partial 0.7979  Nemotron partial 0.6799  cross-avg 0.6817  (step 41400)
gliner-ettin-32m-ptbr-pii-easter-egg: PT-BR strict 0.6445  partial 0.7344  Nemotron partial 0.6019  cross-avg 0.6243  (step 100)
gliner-ettin-68m-ptbr-pii-easter-egg: PT-BR strict 0.7109  partial 0.8073  Nemotron partial 0.7182  cross-avg 0.7167  (step 100)  *** PT-BR best  *** cross-source best

Take-away. The 3x re-runs (69 000 steps) blew past the original 23 000-step sweep — the original 32M variants were straightforwardly under-trained, and the longer cosine schedule unlocks 1.5–2× higher F1 on every PII source. Curated subsets (top-50k / top-100k) overfit to PT-BR but trail badly on cross-source generalization; the full-data 3x runs dominate on every metric we care about. For production use, prefer gliner-ettin-{32m,68m}-ptbr-pii-full-3x-v1 — they sit at the Pareto frontier of in-domain F1 and cross-source generalization.

Holdout benchmark (matched protocol on nvidia/Nemotron-PII)

For the ettin-68m full checkpoint at step 17250, evaluated independently with eval_nemotron_prior.py (5000 samples, threshold=0.5, gold-label set, single-label flat NER):

model	strict F1	partial F1
GLiNER multitask-large (prior)	0.5834	0.6490
mmbert-teacher (prior)	0.4574	0.5418
Albertina ckpt-12k (prior)	0.1801	0.3399
ettin-68m (prior)	0.1493	0.3210
ettin-68m full v1 step-17250 (this sweep)	0.2747	0.4306

i.e. +0.125 strict F1 / +0.110 partial F1 over the prior ettin-68m baseline at the same protocol — with this sweep's recipe (per-source label superset + bf16 mean-reduction + token_level fix).

Limitations

• This is a private research checkpoint. Do not use it as a sole source of truth for redaction without human review.
• PT-BR target performance was the optimization objective; spam/phishing partial F1 is a negative-evidence signal and is intentionally not the target.
• The sweep was constrained to ~23k steps for 68M and ~12k–16k for the curated runs by single-MI300X budget. The follow-up triple-step re-run is what should be used in production once it lands.

Files

Each checkpoint-N/ directory contains a self-contained GLiNER snapshot (gliner_config.json, pytorch_model.bin, tokenizer, optimizer state, scheduler state, RNG state). Load any of them with GLiNER.from_pretrained(repo, subfolder="checkpoint-N").