arabert-openiti-1215

A domain-adapted Arabic language model for pre-modern Arabic prose, produced by continued pre-training of aubmindlab/bert-base-arabertv02 on the OpenITI corpus (release 2025.1.9), filtered to authors with death dates up to AH 1215 (1800 CE).

It is intended as a starting point for fine-tuning on pre-modern Arabic — token classification such as named entity recognition and span tagging, and sequence classification — and as a drop-in replacement for bert-base-arabertv02 in pipelines working with this material. Models trained on modern Arabic perform poorly here: the base scored a perplexity of 22.11 on held-out text from this corpus, against 6.80 after adaptation.

It can also be used frozen, to produce embeddings for semantic similarity or text-reuse detection. That works, and is how its predecessor has been used — but it is a weaker application. Masked language modelling optimises per-token representations, so mean-pooling them into a single sentence vector uses the model indirectly. Where a task can be framed as token classification, that is the better fit.

---

Why AH 1215

The cut-off is 1800 CE, a conventional boundary prior to the spread of print culture, journalism and the translation movement, which together reshaped Arabic prose. The aim is a model of pre-modern written Arabic, not of Arabic in general.

This matters because of how the corpus is distributed. Of 8,755 primary-version Arabic texts in OpenITI 2025.1.9:

Group	Period	Texts	Share
Pre-AH-911	to 1505 CE	5,407	62%
AH 912–1215	1505–1800 CE	~700	8%
Past AH 1215	1800 CE onward	~2,650	30%

The largest single bands in the corpus are AH 1300–1399 (1,366 texts) and AH 1400–1499 (1,021) — authors who died from roughly 1882 CE onward. Training without a cut-off would therefore mean training substantially on modern Arabic, which is what the base model already provides and what a pre-modern model is meant to move away from.

Caveat. Death date is an imperfect proxy for register. Nineteenth- and twentieth-century scholars often wrote in a consciously classicising style, and whether AH 1300+ OpenITI material is linguistically distinct from AH 1200s material has not been measured. The boundary is defensible on periodisation grounds; it is not yet empirically established.

Why AraBERT v0.2 rather than v2

aubmindlab/bert-base-arabertv2 expects text pre-segmented with Farasa, which splits prefixes and suffixes off Arabic words before tokenising. The preprocessing used here applies no segmentation. bert-base-arabertv02 is the same model family, trained on the same 77 GB / 8.66 billion words, with the same 64,000-token vocabulary size, but built for unsegmented input — so it is the base that matches this data.

The difference is measurable. On the same Arabic text, v2 requires 23.5% more tokens than v0.2, because its vocabulary was learned on pre-segmented text and it therefore fragments joined forms: الحيوان becomes الح ##يو ##ان under v2, and remains whole under v0.2.

The predecessor model arabert-openiti-911 was built on v2 and trained on unsegmented text, inheriting this mismatch. Continued pre-training cannot repair a vocabulary, so that model reads Classical Arabic through a vocabulary built for segmented modern Arabic.

---

Training data

Source: OpenITI release 2025.1.9, primary-version corpus.

Filter:

• Primary versions only (status = pri)
• Arabic only (language = ara)
• Author death date ≤ AH 1215
• Minimum 200 characters after cleaning

Result: 6,211 files in the manifest; 6,113 documents entered training after the length minimum. 1,227,620,131 characters after cleaning, ~354.6M tokens, packed into 691,448 blocks of 512 tokens.

Normalisation (OpenITI Python library):

• OpenITI markup stripped — metadata header, page references, milestone markers, structural headers
• deNoise — diacritics removed
• normalize_ara_light — alif variants unified (أ إ آ ٱ → ا), alif maqṣūra normalised (ى → ي), Persian letters normalised; tā marbūṭa (ة) and hamzas preserved
• No Farasa segmentation — see above

Any downstream use must apply the same normalisation. Text prepared differently is, to the model, a different language.

---

Training procedure

Parameter	Value
Base model	aubmindlab/bert-base-arabertv02
Task	Masked language modelling
MLM probability	0.15
Sequence length	512, packed
Epochs	2
Effective batch size	64 (16 per device × 4 gradient accumulation steps)
Learning rate	5e-5
LR scheduler	Linear, 5% warmup
Weight decay	0.01
Precision	bf16
Training steps	19,448 (9,724 per epoch)
Hardware	1 × NVIDIA A100 SXM4 80GB
Training time	1 h 38 min
Library versions	torch 2.6.0+cu124, transformers 5.5.0

Validation is the last two of twenty shards, with documents distributed round-robin during preprocessing so the held-out portion is a representative sample across the corpus rather than a chronological tail.

Vocabulary is inherited unchanged from bert-base-arabertv02 at 64,000 tokens, so the model is a drop-in replacement anywhere that base is used.

---

Evaluation

Model	Perplexity on held-out shards
aubmindlab/bert-base-arabertv02 (before adaptation)	22.11
arabert-openiti-1215 (after adaptation)	6.80

A 69% reduction. Both measured on the same held-out shards of this corpus, with masking generated once under a fixed seed so both models saw identical inputs.

What this does and does not establish. It shows that continued pre-training substantially improved fit to pre-modern Arabic relative to the model's own starting point. It says nothing about how this model compares to any other.

In particular, this figure is not comparable to the 5.4 reported for arabert-openiti-911. Perplexity is per-token, and the two models have different vocabularies — arabert-openiti-911 inherits AraBERT v2's, learned on Farasa-segmented text. They also use different evaluation sets. A higher number here does not indicate a worse model.

Comparing the two requires either a length-normalised metric such as bits per character, or a downstream task. See below.

Downstream evaluation of this model is not yet done. Perplexity indicates fit to held-out text of the same kind, not usefulness on any task.

Evidence for the approach — from the predecessor, not this model

The only downstream evidence available concerns arabert-openiti-911, on a Miskawayh–Ṭabarī text-reuse alignment task scored against hand-annotated ground truth for AH year 204:

Model	Precision	Recall	F1
arabert-openiti-911	100.0%	66.7%	80.0%
aubmindlab/bert-base-arabertv2	100.0%	55.6%	71.4%

The adapted model recovered one further annotated pair at identical precision, while producing about a quarter fewer alignments overall across AH 124–302 (3,759 against 4,990) — more selective rather than weaker.

Three caveats, all material:

1. This concerns a different model. arabert-openiti-911 has a different base, a different cut-off and a different corpus. Nothing here transfers automatically.
2. Nine annotated pairs in one year. A single recovered pair moves recall by eleven points. The result is consistent with a real improvement and equally consistent with luck.
3. The baseline was handicapped. bert-base-arabertv2 expects Farasa-segmented input and received none, so part of that gap is format mismatch rather than domain adaptation. The honest comparison would use bert-base-arabertv02, and has not been run.

Taken together: there is preliminary evidence that domain adaptation on OpenITI improves a real task, and it is not yet strong enough to rely on.

Planned evaluation — named entity recognition

The alignment task above is a weak instrument for this model, for a reason worth stating: it freezes the model and mean-pools token vectors into a single sentence vector. But masked language modelling optimises per-token contextual representations, so mean-pooling measures what the model is good at only indirectly.

The intended evaluation is token classification — a classification head over the per-token outputs, which is what the architecture is built for and what the Arabic NLP literature uses as its standard benchmark family.

The specific task is book-title recognition in Arabic bibliographic and historical prose, evaluated against the text of Ibn al-Nadīm's Fihrist as annotated for the KITAB-Transform project by Peter Verkinderen, which carries 7,001 individually delimited book entries and 1,039 author entries. Candidate models — this one, arabert-openiti-911, and aubmindlab/bert-base-arabertv02 — would be fine-tuned with identical heads, hyperparameters and splits, and compared by F1 on held-out spans.

This is also tokenizer-independent in a way perplexity is not, so it is the route to a valid comparison between models built on different bases.

The task has two halves, and they are evaluated differently. Detection — is a book being cited here — is what the tagger measures. Identification — which book — is matched against a title authority file, for which the working resource is an extraction of some 28,000 author/title pairs from Ismāʿīl Pasha al-Baghdādī's Hadiyyat al-ʿĀrifīn, prepared by Lorenz Nigst for the KITAB-Transform project. That work falls past this model's training cut-off and so is not in its training data; it functions purely as a reference list.

Caveat: the Fihrist is a single genre, where titles are dense and framing formulaic. Strong performance there would not guarantee performance on narrative prose.

---

Relation to arabert-openiti-911

The two models differ on four axes, so a difference in performance cannot be attributed to any one of them:

	arabert-openiti-911	arabert-openiti-1215
Base model	bert-base-arabertv2	bert-base-arabertv02
Death-date cap	AH 911 (1505 CE)	AH 1215 (1800 CE)
Corpus	5,361 docs, ~287M tokens	6,113 docs, ~355M tokens
Library	transformers 5.5.0	transformers 5.5.0
Dataset loading	sample_by="document"	sample_by="paragraph"

The last row is a corrected bug: in HuggingFace datasets, sample_by="document" loads the entire file as a single sample rather than splitting on blank lines. On this corpus that exhausted 117 GB of RAM during tokenisation. The predecessor's script carries the same setting.

Isolating the effect of the date cap alone would require a third run — AH 1215 data on v2, or AH 911 data on v0.2. Neither has been done.

arabert-openiti-911 remains published so that work depending on it stays reproducible.

---

How to use

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("SarahSav/arabert-openiti-1215")
model = AutoModel.from_pretrained("SarahSav/arabert-openiti-1215")

Apply deNoise and normalize_ara_light from the openiti library to input text, and do not apply Farasa segmentation.

---

Limitations

• Trained on the OpenITI primary corpus as it stands, not on a genre-selected subset. That corpus is predominantly scholarly and literary prose — chronicles, biographical dictionaries, fiqh, tafsīr — but it also contains poetry collections (dīwāns), especially in the early centuries, alongside other genres. The genre distribution of the training data has not been characterised, so performance on any particular genre is unknown rather than merely untested.
• Documentary material — letters, contracts, archival texts — is barely represented in OpenITI. Non-Arabic material was excluded by the language filter. Performance on either is untested.
• OpenITI coverage is uneven across periods as well as genres; some traditions are much better represented than others.
• The AH 1215 boundary rests on periodisation, not measurement — see the caveat above.
• Light normalisation only, so texts with heavy dialectal or orthographic variation may benefit less.
• No downstream task evaluation yet.

---

Citation

Please cite the OpenITI corpus:

Nigst, Lorenz, Romanov, Maxim, Savant, Sarah Bowen, Seydi, Masoumeh, and Verkinderen, Peter. OpenITI: a Machine-Readable Corpus of Islamicate Texts (Version 2025.1.9). Zenodo, 2025.

Authors

Sarah Bowen Savant (Aga Khan University, ISMC — Centre for Digital Humanities), as part of the KITAB-Transform project.

Acknowledgements

Scripts, diagnostics and documentation for this model were developed with the assistance of Claude (Anthropic), used as a coding and analysis tool. Corpus scope, the AH 1215 cut-off and evaluation design are the author's decisions, and responsibility for the model and its documentation rests with the author.

The corpus is the work of the OpenITI team. Two resources referenced under planned evaluation were prepared by colleagues on the KITAB-Transform project: the structurally annotated text of Ibn al-Nadīm's Fihrist, by Peter Verkinderen, and the extraction of author/title pairs from Hadiyyat al-ʿĀrifīn, by Lorenz Nigst.

License

Model weights inherit the licence of the base model aubmindlab/bert-base-arabertv02. Training data is from OpenITI, openly licensed for research use.