--- language: ar license: apache-2.0 base_model: aubmindlab/bert-base-arabertv02 tags: - arabic - classical-arabic - historical - openiti - fill-mask library_name: transformers pipeline_tag: fill-mask --- # arabert-openiti-1215 A domain-adapted Arabic language model for **pre-modern Arabic prose**, produced by continued pre-training of [`aubmindlab/bert-base-arabertv02`](https://huggingface.co/aubmindlab/bert-base-arabertv02) on the [OpenITI](https://openiti.org) corpus (release 2025.1.9), filtered to authors with death dates up to **AH 1215 (1800 CE)**. It is intended as a **starting point for fine-tuning** on pre-modern Arabic — token classification such as named entity recognition and span tagging, and sequence classification — and as a drop-in replacement for `bert-base-arabertv02` in pipelines working with this material. Models trained on modern Arabic perform poorly here: the base scored a perplexity of 22.11 on held-out text from this corpus, against 6.80 after adaptation. It can also be used frozen, to produce embeddings for semantic similarity or text-reuse detection. That works, and is how its predecessor has been used — but it is a weaker application. Masked language modelling optimises per-token representations, so mean-pooling them into a single sentence vector uses the model indirectly. Where a task can be framed as token classification, that is the better fit. --- ## Why AH 1215 The cut-off is **1800 CE**, a conventional boundary prior to the spread of print culture, journalism and the translation movement, which together reshaped Arabic prose. The aim is a model of pre-modern written Arabic, not of Arabic in general. This matters because of how the corpus is distributed. Of 8,755 primary-version Arabic texts in OpenITI 2025.1.9: | Group | Period | Texts | Share | |---|---|---:|---:| | Pre-AH-911 | to 1505 CE | 5,407 | 62% | | AH 912–1215 | 1505–1800 CE | ~700 | 8% | | Past AH 1215 | 1800 CE onward | ~2,650 | 30% | The largest single bands in the corpus are AH 1300–1399 (1,366 texts) and AH 1400–1499 (1,021) — authors who died from roughly 1882 CE onward. Training without a cut-off would therefore mean training substantially on **modern** Arabic, which is what the base model already provides and what a pre-modern model is meant to move away from. **Caveat.** Death date is an imperfect proxy for register. Nineteenth- and twentieth-century scholars often wrote in a consciously classicising style, and whether AH 1300+ OpenITI material is linguistically distinct from AH 1200s material has not been measured. The boundary is defensible on periodisation grounds; it is not yet empirically established. ## Why AraBERT v0.2 rather than v2 **`aubmindlab/bert-base-arabertv2` expects text pre-segmented with Farasa**, which splits prefixes and suffixes off Arabic words before tokenising. The preprocessing used here applies no segmentation. `bert-base-arabertv02` is the same model family, trained on the same 77 GB / 8.66 billion words, with the same 64,000-token vocabulary size, but built for unsegmented input — so it is the base that matches this data. The difference is measurable. On the same Arabic text, v2 requires **23.5% more tokens** than v0.2, because its vocabulary was learned on pre-segmented text and it therefore fragments joined forms: `الحيوان` becomes `الح` `##يو` `##ان` under v2, and remains whole under v0.2. The predecessor model `arabert-openiti-911` was built on v2 and trained on unsegmented text, inheriting this mismatch. Continued pre-training cannot repair a vocabulary, so that model reads Classical Arabic through a vocabulary built for segmented modern Arabic. --- ## Training data **Source:** [OpenITI](https://openiti.org) release 2025.1.9, primary-version corpus. **Filter:** - Primary versions only (`status = pri`) - Arabic only (`language = ara`) - Author death date ≤ AH 1215 - Minimum 200 characters after cleaning **Result:** 6,211 files in the manifest; **6,113 documents** entered training after the length minimum. **1,227,620,131 characters** after cleaning, **~354.6M tokens**, packed into **691,448 blocks** of 512 tokens. **Normalisation** (OpenITI Python library): - OpenITI markup stripped — metadata header, page references, milestone markers, structural headers - `deNoise` — diacritics removed - `normalize_ara_light` — alif variants unified (أ إ آ ٱ → ا), alif maqṣūra normalised (ى → ي), Persian letters normalised; tā marbūṭa (ة) and hamzas preserved - **No Farasa segmentation** — see above Any downstream use must apply the same normalisation. Text prepared differently is, to the model, a different language. --- ## Training procedure | Parameter | Value | |---|---| | Base model | `aubmindlab/bert-base-arabertv02` | | Task | Masked language modelling | | MLM probability | 0.15 | | Sequence length | 512, packed | | Epochs | 2 | | Effective batch size | 64 (16 per device × 4 gradient accumulation steps) | | Learning rate | 5e-5 | | LR scheduler | Linear, 5% warmup | | Weight decay | 0.01 | | Precision | bf16 | | Training steps | 19,448 (9,724 per epoch) | | Hardware | 1 × NVIDIA A100 SXM4 80GB | | Training time | 1 h 38 min | | Library versions | torch 2.6.0+cu124, transformers 5.5.0 | Validation is the last two of twenty shards, with documents distributed round-robin during preprocessing so the held-out portion is a representative sample across the corpus rather than a chronological tail. Vocabulary is inherited unchanged from `bert-base-arabertv02` at 64,000 tokens, so the model is a drop-in replacement anywhere that base is used. --- ## Evaluation | Model | Perplexity on held-out shards | |---|---:| | `aubmindlab/bert-base-arabertv02` (before adaptation) | 22.11 | | `arabert-openiti-1215` (after adaptation) | **6.80** | A 69% reduction. Both measured on the same held-out shards of this corpus, with masking generated once under a fixed seed so both models saw identical inputs. **What this does and does not establish.** It shows that continued pre-training substantially improved fit to pre-modern Arabic relative to the model's own starting point. It says nothing about how this model compares to any other. In particular, **this figure is not comparable to the 5.4 reported for `arabert-openiti-911`.** Perplexity is per-token, and the two models have different vocabularies — `arabert-openiti-911` inherits AraBERT v2's, learned on Farasa-segmented text. They also use different evaluation sets. A higher number here does not indicate a worse model. Comparing the two requires either a length-normalised metric such as bits per character, or a downstream task. See below. **Downstream evaluation of this model is not yet done.** Perplexity indicates fit to held-out text of the same kind, not usefulness on any task. ### Evidence for the approach — from the predecessor, not this model The only downstream evidence available concerns `arabert-openiti-911`, on a Miskawayh–Ṭabarī text-reuse alignment task scored against hand-annotated ground truth for AH year 204: | Model | Precision | Recall | F1 | |---|---:|---:|---:| | `arabert-openiti-911` | 100.0% | 66.7% | 80.0% | | `aubmindlab/bert-base-arabertv2` | 100.0% | 55.6% | 71.4% | The adapted model recovered one further annotated pair at identical precision, while producing about a quarter fewer alignments overall across AH 124–302 (3,759 against 4,990) — more selective rather than weaker. **Three caveats, all material:** 1. **This concerns a different model.** `arabert-openiti-911` has a different base, a different cut-off and a different corpus. Nothing here transfers automatically. 2. **Nine annotated pairs in one year.** A single recovered pair moves recall by eleven points. The result is consistent with a real improvement and equally consistent with luck. 3. **The baseline was handicapped.** `bert-base-arabertv2` expects Farasa-segmented input and received none, so part of that gap is format mismatch rather than domain adaptation. The honest comparison would use `bert-base-arabertv02`, and has not been run. Taken together: there is preliminary evidence that domain adaptation on OpenITI improves a real task, and it is not yet strong enough to rely on. ### Planned evaluation — named entity recognition The alignment task above is a weak instrument for this model, for a reason worth stating: it freezes the model and mean-pools token vectors into a single sentence vector. But masked language modelling optimises *per-token contextual representations*, so mean-pooling measures what the model is good at only indirectly. The intended evaluation is **token classification** — a classification head over the per-token outputs, which is what the architecture is built for and what the Arabic NLP literature uses as its standard benchmark family. The specific task is **book-title recognition** in Arabic bibliographic and historical prose, evaluated against the text of Ibn al-Nadīm's *Fihrist* as annotated for the KITAB-Transform project by Peter Verkinderen, which carries 7,001 individually delimited book entries and 1,039 author entries. Candidate models — this one, `arabert-openiti-911`, and `aubmindlab/bert-base-arabertv02` — would be fine-tuned with identical heads, hyperparameters and splits, and compared by F1 on held-out spans. This is also tokenizer-independent in a way perplexity is not, so it is the route to a valid comparison between models built on different bases. The task has two halves, and they are evaluated differently. **Detection** — is a book being cited here — is what the tagger measures. **Identification** — which book — is matched against a title authority file, for which the working resource is an extraction of some 28,000 author/title pairs from Ismāʿīl Pasha al-Baghdādī's *Hadiyyat al-ʿĀrifīn*, prepared by Lorenz Nigst for the KITAB-Transform project. That work falls past this model's training cut-off and so is not in its training data; it functions purely as a reference list. Caveat: the *Fihrist* is a single genre, where titles are dense and framing formulaic. Strong performance there would not guarantee performance on narrative prose. --- ## Relation to `arabert-openiti-911` The two models differ on **four** axes, so a difference in performance cannot be attributed to any one of them: | | `arabert-openiti-911` | `arabert-openiti-1215` | |---|---|---| | Base model | `bert-base-arabertv2` | `bert-base-arabertv02` | | Death-date cap | AH 911 (1505 CE) | AH 1215 (1800 CE) | | Corpus | 5,361 docs, ~287M tokens | 6,113 docs, ~355M tokens | | Library | transformers 5.5.0 | transformers 5.5.0 | | Dataset loading | `sample_by="document"` | `sample_by="paragraph"` | The last row is a corrected bug: in HuggingFace `datasets`, `sample_by="document"` loads the entire file as a single sample rather than splitting on blank lines. On this corpus that exhausted 117 GB of RAM during tokenisation. The predecessor's script carries the same setting. Isolating the effect of the date cap alone would require a third run — AH 1215 data on v2, or AH 911 data on v0.2. Neither has been done. `arabert-openiti-911` remains published so that work depending on it stays reproducible. --- ## How to use ```python from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("SarahSav/arabert-openiti-1215") model = AutoModel.from_pretrained("SarahSav/arabert-openiti-1215") ``` Apply `deNoise` and `normalize_ara_light` from the `openiti` library to input text, and do **not** apply Farasa segmentation. --- ## Limitations - Trained on the OpenITI primary corpus **as it stands**, not on a genre-selected subset. That corpus is predominantly scholarly and literary prose — chronicles, biographical dictionaries, fiqh, tafsīr — but it also contains poetry collections (dīwāns), especially in the early centuries, alongside other genres. **The genre distribution of the training data has not been characterised**, so performance on any particular genre is unknown rather than merely untested. - Documentary material — letters, contracts, archival texts — is barely represented in OpenITI. Non-Arabic material was excluded by the language filter. Performance on either is untested. - OpenITI coverage is uneven across periods as well as genres; some traditions are much better represented than others. - The AH 1215 boundary rests on periodisation, not measurement — see the caveat above. - Light normalisation only, so texts with heavy dialectal or orthographic variation may benefit less. - No downstream task evaluation yet. --- ## Citation Please cite the OpenITI corpus: > Nigst, Lorenz, Romanov, Maxim, Savant, Sarah Bowen, Seydi, Masoumeh, and Verkinderen, Peter. > *OpenITI: a Machine-Readable Corpus of Islamicate Texts* (Version 2025.1.9). Zenodo, 2025. ## Authors Sarah Bowen Savant (Aga Khan University, ISMC — Centre for Digital Humanities), as part of the KITAB-Transform project. ## Acknowledgements Scripts, diagnostics and documentation for this model were developed with the assistance of Claude (Anthropic), used as a coding and analysis tool. Corpus scope, the AH 1215 cut-off and evaluation design are the author's decisions, and responsibility for the model and its documentation rests with the author. The corpus is the work of the OpenITI team. Two resources referenced under planned evaluation were prepared by colleagues on the KITAB-Transform project: the structurally annotated text of Ibn al-Nadīm's *Fihrist*, by Peter Verkinderen, and the extraction of author/title pairs from *Hadiyyat al-ʿĀrifīn*, by Lorenz Nigst. ## License Model weights inherit the licence of the base model [`aubmindlab/bert-base-arabertv02`](https://huggingface.co/aubmindlab/bert-base-arabertv02). Training data is from OpenITI, openly licensed for research use.