Update model card (v0.1 fact-checked)

README.md

@@ -26,31 +26,28 @@ model-index:
           type: token-classification
           name: Part-of-Speech Tagging
         dataset:
-          name: Darija POS corpus (validation split, dev+test merged)
+          name: Darija POS corpus (validation split, dev+test merged, in-corpus)
           type: darija-pos-corpus
         metrics:
           - type: f1
-            value: 0.9999
-            name: Seqeval F1 (validation)
-          - type: accuracy
-            value: 0.9999
-            name: Token accuracy (validation)
+            value: 1.0
+            name: Seqeval F1 (in-corpus validation — see caveats)
 widget:
   - text: "شربت أتاي مع صحابي"
-    example_title: "Simple sentence (verb + nouns)"
+    example_title: "Simple sentence"
   - text: "غادي نمشي للسوق غدا"
-    example_title: "Future (غادي + verb)"
+    example_title: "Future construction"
   - text: "ما عند ي ش الفلوس"
     example_title: "Negation, clitic-segmented (ما … ش)"
 ---
 
 # DarijaPOSTagger (v0.1) — Part-of-Speech Tagging for Moroccan Darija
 
-> 🚧 **Preliminary release (v0.1) — for education & research.** This model was trained on a small corpus (1,225 sentences) and released early, with its limitations openly documented, as a teaching artifact and a baseline for the Darija NLP community. Read the [Preliminary Validation](#preliminary-validation-july-2026) and [Known Limitations](#known-limitations--bias) sections before using it. Not intended for production use.
+> 🚧 **Preliminary release (v0.1) — for education & research.** This model was trained on a small corpus (1,225 sentences) and is released early, with its limitations openly documented, as a teaching artifact and a baseline for the Darija NLP community. Read the [Preliminary Validation](#preliminary-validation-july-2026) and [Known Limitations](#known-limitations--bias) sections before using it. Not intended for production use.
 
-**DarijaPOSTagger** is a BERT-based token classification model that assigns **part-of-speech (POS) tags** to **Moroccan Darija** text (Moroccan Arabic dialect, Arabic script). It is fine-tuned from [SI2M-Lab/DarijaBERT](https://huggingface.co/SI2M-Lab/DarijaBERT) on a manually POS-annotated Darija corpus, using a 15-tag tagset adapted to Darija morphology — including dialect-specific particles such as the progressive **كي** (`PROG_PART`), the future marker **غادي** (`FUT_PART`), and the discontinuous negation **ما … ش** (`NEG_PART`).
+**DarijaPOSTagger** is a BERT-based token classification model that assigns **part-of-speech (POS) tags** to **Moroccan Darija** text (Moroccan Arabic dialect, Arabic script). It is fine-tuned from [SI2M-Lab/DarijaBERT](https://huggingface.co/SI2M-Lab/DarijaBERT) on a POS-annotated Darija corpus using a 15-tag scheme that includes Darija-relevant particle categories (`PROG_PART`, `FUT_PART`, `NEG_PART`) alongside standard classes. Notably, the corpus is **morpheme-segmented**: clitics such as the article **ال** and the negation **ش** are annotated as separate tokens (e.g., ما `NEG_PART` + عند `ADV` + هوم `PRON` + ش `NEG_PART`).
 
-The model is released by [Typica.ai](https://typica.ai) as part of its applied research on **culturally localized AI for underserved languages**, and is **open-sourced for educational and research purposes**. It complements Typica.ai's Darija NLP suite (NER, sentiment, toxicity detection).
+The model is released by [Typica.ai](https://typica.ai) as part of its applied research on **culturally localized AI for underserved languages**, and is **open-sourced for educational and research purposes**.
 
 ## Model Details
 
@@ -63,39 +60,44 @@ The model is released by [Typica.ai](https://typica.ai) as part of its applied r
 | **License** | CC BY-NC 4.0 (non-commercial — education & research) |
 | **Version** | v0.1 (preliminary) |
 | **Released** | July 2026 |
-| **Contact** | assoudi@typica.ai |
+| **Contact** | assoudi@typica.ai <!-- TODO: verify preferred contact email --> |
 
 ### Tagset (15 tags)
 
-| Tag | Meaning | Example (Darija) |
-|-----|---------|------------------|
-| `NOUN` | Noun | الولد، السوق، الفلوس |
-| `V` | Verb | مشيت، كيلعب، نمشي |
-| `ADJ` | Adjective | زوين، كبير |
-| `ADV` | Adverb | غدا، بزاف، دابا |
-| `PRON` | Pronoun | هو، ديالي، ها |
-| `DET` | Determiner | ال، هاد، واحد |
-| `PREP` | Preposition | ف، ل، على، من |
-| `CONJ` | Conjunction | و، ولكن، حيت |
-| `PART` | Particle (general) | را، واش |
-| `PROG_PART` | Progressive/aspectual particle | كي، تا (كيلعب) |
-| `FUT_PART` | Future particle | غادي، غا (غادي نمشي) |
-| `NEG_PART` | Negation particle | ما، ش (ما عندي ش) |
-| `NSUFF` | Noun suffix (inflection) | ة، ات، ين |
-| `CASE` | Case marker | — |
-| `O` | Other (punctuation, numbers, URLs, mentions, emoticons, foreign tokens) | — |
+Corpus-attested examples are shown where available; the annotation scheme of the source corpus is authoritative.
+
+| Tag | Category | Corpus-attested examples |
+|-----|----------|--------------------------|
+| `NOUN` | Noun | مغربي، ديبلوم |
+| `V` | Verb | خدام |
+| `ADJ` | Adjective | — |
+| `ADV` | Adverb | عند |
+| `PRON` | Pronoun | لي، هوم |
+| `DET` | Determiner | ال |
+| `PREP` | Preposition | ديال |
+| `CONJ` | Conjunction | — |
+| `PART` | Particle | — |
+| `PROG_PART` | Progressive particle | — |
+| `FUT_PART` | Future particle | — |
+| `NEG_PART` | Negation particle | ما، ش |
+| `NSUFF` | Noun suffix | ات |
+| `CASE` | (per corpus scheme) | — |
+| `O` | Other (punctuation, numbers, URLs, mentions, emoticons, foreign tokens — collapsed) | 0.7 |
+
+> Note: some conventions may differ from other Arabic tagsets — for instance, this corpus annotates عند as `ADV`. Consult the source corpus's annotation guidelines for authoritative definitions.
 
 ## Intended Uses
 
 **Direct intended uses:**
 - Research on morphosyntactic analysis of Moroccan Darija and Arabic dialects.
 - Education: teaching token classification, subword/label alignment, and fine-tuning for low-resource languages.
-- Preprocessing component in Darija NLP pipelines (e.g., feature extraction, chunking, linguistic annotation, corpus building).
+- Baseline / preprocessing component in experimental Darija NLP pipelines.
 
 **Out-of-scope uses:**
 - ❌ Commercial deployment without a separate agreement with Typica.ai (license is non-commercial).
-- ❌ Latin-script Darija (Arabizi), French, or English text: non-Arabic tokens were cleaned from training data.
-- ❌ Modern Standard Arabic or other Arabic dialects — the tagset and training data are Darija-specific.
+- ❌ Latin-script Darija (Arabizi), French, or English text: non-Arabic characters were cleaned from training data.
+- ❌ Modern Standard Arabic or other Arabic dialects — the training data is Darija-specific.
+- ❌ Production use: this is a preliminary v0.1 release.
 
 ## How to Use
 
@@ -110,9 +112,9 @@ pos_tagger = pipeline(
 )
 
 sentences = [
-    "غادي نمشي للسوق غدا",          # I will go to the market tomorrow
-    "شربت أتاي مع صحابي",           # I drank tea with my friends
-    "ما عنديش الفلوس",              # I don't have money
+    "غادي نمشي للسوق غدا",     # I will go to the market tomorrow
+    "شربت أتاي مع صحابي",      # I drank tea with my friends
+    "ما عند ي ش الفلوس",       # I don't have money (clitic-segmented, matching the training format)
 ]
 
 for sentence in sentences:
@@ -121,16 +123,20 @@ for sentence in sentences:
         print(f"  {token['word']:<15} → {token['entity_group']:<10} ({token['score']:.3f})")
 ```
 
-> ℹ️ **Input format matters.** The training corpus is *morpheme-segmented* (clitics like ال، ف، كي، ش appear as separate tokens), while raw Darija text fuses them into surface words (فالزنقة، عنديش). The model is noticeably more reliable on clitic-segmented input; on raw text, expect degraded accuracy on fused forms (see Preliminary Validation below). Also note that `aggregation_strategy` may merge adjacent same-label words into one span — for strict one-tag-per-word output, tokenize with `is_split_into_words=True` and read per-word predictions directly.
+> ℹ️ **Input format matters.** The training corpus is morpheme-segmented (clitics like ال and ش appear as separate tokens), while raw Darija text fuses them into surface words (e.g., عنديش، فالزنقة). Fused surface forms are out-of-distribution for this model; expect lower confidence and less reliable tags on them (see Preliminary Validation). Also note that `aggregation_strategy` may merge adjacent same-label words into one span — for strict one-tag-per-word output, tokenize with `is_split_into_words=True` and read per-word predictions directly.
 
 ## Training Data
 
-The model was trained on a manually POS-annotated Moroccan Darija corpus in CoNLL-style format (one `token  tag` pair per line, sentences separated by `EOS` markers).
+The model was trained on a POS-annotated Moroccan Darija corpus in CoNLL-style format (one `token  tag` pair per line, sentences separated by `EOS` markers), read from local files.
 
-**Preprocessing:**
+<!-- TODO: add source corpus name, provenance, and citation -->
+
+**Preprocessing (as implemented in the training notebook):**
 - Tokens were cleaned with a character filter keeping Arabic script, digits, and basic punctuation.
-- The original 22-tag scheme was simplified to 15 tags: formatting and non-linguistic tags (`WB`, `TB`, `EMOT`, `HASH`, `FOREIGN`, `MENTION`, `PUNC`, `URL`, `NUM`) were collapsed into a single `O` class, keeping the tagset focused on linguistically meaningful categories.
-- Subword/label alignment: only the first subword of each word receives the word's label during tokenization (WordPiece), with special tokens masked at `-100`.
+- The original tag scheme was simplified to 15 tags: formatting and non-linguistic tags (`WB`, `TB`, `EMOT`, `HASH`, `FOREIGN`, `MENTION`, `PUNC`, `URL`, `NUM`) were mapped to a single `O` class (any tag not in the final tagset defaults to `O`).
+- Subword/label alignment: each word's label is propagated to **all** of its WordPiece subwords; special tokens are masked at `-100`.
+
+> ⚠️ **Known v0.1 training artifact:** the label-alignment code inherits a `B-` → `I-` conversion step from NER-style code (`if label % 2 == 1: label += 1`), which is not meaningful for POS tags. For words split into multiple subwords, this shifts odd-id labels on continuation subwords to the next label id (e.g., `DET` → `FUT_PART`, `PRON` → `V`) during training. The impact has not been quantified; it is scheduled to be removed in v0.2.
 
 ### Splits
 
@@ -139,13 +145,14 @@ The model was trained on a manually POS-annotated Moroccan Darija corpus in CoNL
 | Train | 1,225 | Fine-tuning |
 | Dev | 175 | — |
 | Test | 350 | — |
-| Validation (dev + test merged) | 525 | Evaluation during training |
+| Validation (dev + test merged) | 525 | Per-epoch evaluation (monitoring) |
 
 ## Training Procedure
 
 - **Base checkpoint:** `SI2M-Lab/DarijaBERT` with a freshly initialized token classification head (15 labels)
-- **Objective:** token-level cross-entropy (seqeval-based evaluation)
+- **Objective:** token-level cross-entropy; seqeval-based evaluation during training
 - **Hardware:** single T4 GPU (Google Colab)
+- **Checkpoint released:** final model after 10 epochs (no best-checkpoint selection was configured)
 
 **Hyperparameters:**
 
@@ -161,22 +168,15 @@ The model was trained on a manually POS-annotated Moroccan Darija corpus in CoNL
 
 ### Quantitative (in-corpus)
 
-On the validation split (525 sentences, dev + test merged), the model converges to near-perfect seqeval scores:
-
-| Metric | Score |
-|---|---:|
-| Precision | ≈ 0.9999 |
-| Recall | ≈ 0.9999 |
-| F1 | ≈ 0.9999 |
-| Token accuracy | ≈ 0.9999 |
+On the validation split (525 sentences, dev + test merged), training-log seqeval metrics reach **1.00** (precision, recall, F1, and token accuracy) from epoch 6 onward, including the final epoch 10 checkpoint that is released here.
 
-> ⚠️ **Interpret with care.** The validation set is small, merges the original dev and test splits, and was used for checkpoint selection during training — it is not a clean held-out test set. Near-perfect scores reflect in-corpus fit on a small, homogeneous, morpheme-segmented corpus — not expected real-world accuracy. The qualitative validation below gives a more realistic picture.
+> ⚠️ **Interpret with care.** The validation set is small, merges the original dev and test splits, and comes from the same corpus as the training data — it is not an independent held-out test set. A perfect in-corpus score on a small, homogeneous, morpheme-segmented corpus does not translate to real-world accuracy on raw Darija text. The qualitative validation below gives a more realistic picture.
 
 ### Preliminary Validation (July 2026)
 
-Manual spot-checks on **raw (unsegmented) everyday Darija sentences** — a harder condition than the segmented training data — showed a consistent pattern:
+Manual spot-checks on **raw (unsegmented) everyday Darija sentences** — a harder condition than the segmented training format — showed the following:
 
-**What works well** — open-class words in canonical surface forms are tagged reliably, typically at ≥ 0.99 confidence:
+**Reliable on canonical surface forms** — open-class words are tagged consistently, typically at ≥ 0.99 confidence:
 
 | Word | Predicted | Confidence |
 |---|---|---:|
@@ -187,27 +187,32 @@ Manual spot-checks on **raw (unsegmented) everyday Darija sentences** — a hard
 | الفلوس | NOUN | 0.998 |
 | للسوق | NOUN | 0.999 |
 
-**Known error patterns on raw text:**
+**Degraded on fused clitic forms** (out-of-distribution relative to the segmented training data):
 
-| Input | Predicted | Expected | Likely cause |
-|---|---|---|---|
-| الولد | O (0.512) | DET+NOUN | Fused article ال unseen as one token |
-| عنديش | ADV (0.582) | PREP+PRON+NEG_PART | Fused clitics (عند+ي+ش) unseen as one token |
-| فالزنقة | (merged into NOUN span) | PREP+DET+NOUN | Fused preposition ف lost |
-| غادي | PART (0.988) | FUT_PART | Closed-class near-miss |
-| مع | CONJ (0.996) | PREP | Closed-class confusion, confidently wrong |
+| Input (raw, fused) | Predicted | Observation |
+|---|---|---|
+| الولد | O (0.512) | Near coin-flip confidence; fused article form unseen as a single token |
+| عنديش | ADV (0.582) | Low confidence on the fused form. Note: the corpus tags عند as `ADV`, so the predicted class follows the corpus convention |
+| فالزنقة | merged into a NOUN span | The fused preposition ف is not separately tagged (also affected by pipeline span aggregation) |
 
-**Root cause:** a train/inference input mismatch. The corpus is morpheme-segmented (clitics as separate tokens — hence the `NSUFF` tag and clitic-level `NEG_PART`), so fused surface words are out-of-distribution at inference. Low confidence (≈ 0.5) on fused forms is a useful signal of this. Note also that closed-class errors (مع → CONJ) can be high-confidence, so confidence alone is not a reliability guarantee.
+**Closed-class predictions to verify against corpus conventions** — these are confident predictions whose correctness depends on the source corpus's annotation guidelines, which have not yet been checked for these words:
+
+| Word | Predicted | Confidence |
+|---|---|---:|
+| غادي | PART | 0.988 |
+| مع | CONJ | 0.996 |
 
-**Planned for v0.2:** an independent held-out evaluation on raw text, and either a paired rule-based clitic segmenter or retraining on de-segmented surface-word data.
+**Practical takeaways:** low confidence (≈ 0.5–0.6) is a useful signal that an input form is out-of-distribution (typically fused clitics); clitic-segmented input matching the training format is more reliable; and confident predictions on function words should be interpreted relative to the corpus's own annotation scheme rather than assumed tagging conventions from other resources.
+
+**Planned for v0.2:** removal of the label-alignment artifact, an independent held-out evaluation on raw text, verification of closed-class conventions against the source annotation guidelines, and either a paired clitic segmenter or retraining on de-segmented surface-word data.
 
 ## Known Limitations & Bias
 
-- **Preliminary release:** v0.1 is an educational baseline, not a production tagger. Error patterns above are documented, not yet fixed.
-- **Segmentation mismatch:** best results require clitic-segmented input; raw text degrades accuracy on fused forms (fused articles, prepositions, and negation clitics).
+- **Preliminary release:** v0.1 is an educational baseline, not a production tagger. Observations above are documented, not yet fixed.
+- **Segmentation mismatch:** the training data is morpheme-segmented; raw text with fused clitics is out-of-distribution and degrades reliability.
+- **Training artifact:** the subword label-alignment shift described in Training Data affects multi-subword words with odd label ids; impact unquantified.
 - **Small training corpus:** 1,225 training sentences; limited coverage of rare constructions, regional sub-dialects, and domain vocabulary.
-- **In-corpus evaluation:** the quantitative metrics overestimate real-world performance (see Evaluation).
-- **Closed-class confusions:** some function words (e.g., مع، غادي) can be tagged incorrectly with high confidence.
+- **In-corpus evaluation only:** no independent held-out test yet; the quantitative metrics overestimate real-world performance.
 - **Script coverage:** Arabic script only; Arabizi/Latin-script Darija is out of scope.
 - **Tag granularity:** numbers, punctuation, URLs, mentions, and foreign words are collapsed into `O`.
 
@@ -227,22 +232,11 @@ No dedicated paper accompanies this model yet. If you use it, please cite the mo
 }
 ```
 
-Please also cite the base model:
-
-```bibtex
-@article{gaanoun2023darijabert,
-  title   = {DarijaBERT: a step forward in NLP for the written Moroccan dialect},
-  author  = {Gaanoun, Kamel and Naira, Abdou Mohamed and Allak, Anass and Benelallam, Imade},
-  journal = {International Journal of Data Science and Analytics},
-  year    = {2023},
-  doi     = {10.1007/s41060-023-00498-2}
-}
-```
+Please also cite the base model, [SI2M-Lab/DarijaBERT](https://huggingface.co/SI2M-Lab/DarijaBERT) — see its model card for the authors' preferred citation.
 
 ## Related Models by Typica.ai
 
-- [MAGBERT-NER](https://huggingface.co/TypicaAI/magbert-ner) — Named Entity Recognition for Moroccan Darija/French
-- Darija Toxicity Detection — culturally grounded content moderation (see [arXiv:2505.04640](https://arxiv.org/abs/2505.04640))
+See the [TypicaAI organization page](https://huggingface.co/TypicaAI) for the full Darija NLP suite, including MAGBERT-NER (named entity recognition) and the Darija toxicity detection model ([arXiv:2505.04640](https://arxiv.org/abs/2505.04640)).
 
 ## Contact