docs: update model card for Sprint 7.2 (real-world F1=0.975)

README.md

@@ -24,15 +24,18 @@ model-index:
           type: token-classification
           name: Named Entity Recognition
         metrics:
-          - name: F1
+          - name: F1 (real-world)
             type: f1
-            value: 0.956
-          - name: Precision
+            value: 0.975
+          - name: F1 (mixed test)
+            type: f1
+            value: 0.929
+          - name: Precision (real-world)
             type: precision
-            value: 0.939
-          - name: Recall
+            value: 0.960
+          - name: Recall (real-world)
             type: recall
-            value: 0.974
+            value: 0.990
 ---
 
 # Thai Job NER — Fine-tuned PhayaThaiBERT
@@ -69,74 +72,83 @@ for entity in results:
 
 ## Training
 
-- **Base model:** `clicknext/phayathaibert` (CamemBERT architecture, ~122M params, XLM-R-derived vocabulary)
-- **Training data:** 1,253 Thai job posts (synthetic silver labels from GPT-4o, fuzzy-aligned to IOB2) — [Dataset on HuggingFace](https://huggingface.co/datasets/chayuto/thai-job-ner-dataset)
+- **Base model:** `clicknext/phayathaibert` (CamemBERT architecture, ~122M params)
+- **Training data:** 510 style-matched synthetic + 270 real posts (54 real upsampled 5x)
+- **Training strategy:** Mixed synthetic + real with real data upsampling to ensure 34.6% real data proportion
 - **Hardware:** Apple Silicon MPS backend, FP32
-- **Hyperparameters:** LR=3e-5, warmup=0.1, batch=2, grad_accum=8, 15 epochs, gradient checkpointing, frozen embeddings
-- **Training time:** ~10 min
+- **Hyperparameters:** LR=3e-5, warmup=0.1, batch=2, grad_accum=4, 10 epochs, gradient checkpointing, frozen embeddings
+- **Training time:** ~35 min on Apple Silicon
 
 ### Data Pipeline
 
-Raw Thai text + GPT-4o entity extractions → fuzzy alignment with rapidfuzz + pythainlp TCC boundary snapping → subword token mapping via offset_mapping → IOB2-formatted HuggingFace Dataset.
+Raw Thai text + entity annotations -> fuzzy alignment with rapidfuzz + pythainlp TCC boundary snapping -> subword token mapping via offset_mapping -> IOB2-formatted HuggingFace Dataset.
+
+### Key Training Insight
+
+Real data proportion is the dominant factor for real-world performance. Our experiments showed:
+
+| Real Data % | Real-World F1 |
+|------------|---------------|
+| 2.9% | 0.771 |
+| 4.1% | 0.935 |
+| 9.6% | 0.714* |
+| 23.3% | 0.975 |
+| **34.6%** | **0.975** |
+
+(*small dataset, underfitting)
+
+Style-matched synthetic data (matching real-world formatting patterns) is 3x more data-efficient than generic synthetic data.
 
 ## Evaluation
 
-### Overall (Test Set, 126 examples)
+### Real-World Performance (8 held-out real posts, 98 entities)
+
+| Entity | Precision | Recall | F1 |
+|--------|-----------|--------|-----|
+| COMPENSATION | 1.000 | 1.000 | **1.000** |
+| CONTACT | 1.000 | 1.000 | **1.000** |
+| DEMOGRAPHIC | 1.000 | 1.000 | **1.000** |
+| LOCATION | 1.000 | 1.000 | **1.000** |
+| PERSON | 1.000 | 1.000 | **1.000** |
+| HARD_SKILL | 0.947 | 1.000 | **0.973** |
+| EMPLOYMENT_TERMS | 0.846 | 0.917 | **0.880** |
+| **Overall** | **0.960** | **0.990** | **0.975** |
+
+### Mixed Test Performance (78 examples)
 
 | Metric | Score |
 |--------|-------|
-| **F1** | **0.956** |
-| Precision | 0.939 |
-| Recall | 0.974 |
-
-### Per-Entity F1
-
-| Entity | F1 | Precision | Recall |
-|--------|-----|-----------|--------|
-| CONTACT | 0.987 | 0.983 | 0.991 |
-| PERSON | 0.979 | 0.972 | 0.986 |
-| LOCATION | 0.966 | 0.950 | 0.983 |
-| EMPLOYMENT_TERMS | 0.966 | 0.943 | 0.990 |
-| COMPENSATION | 0.965 | 0.956 | 0.973 |
-| HARD_SKILL | 0.946 | 0.919 | 0.974 |
-| DEMOGRAPHIC | 0.915 | 0.897 | 0.935 |
-
-### Comparison vs WangchanBERTa
-
-| Entity | WangchanBERTa | PhayaThaiBERT | Delta |
-|--------|---------------|---------------|-------|
-| **Overall F1** | 0.897 | **0.956** | **+0.059** |
-| COMPENSATION | 0.764 | **0.965** | **+0.200** |
-| PERSON | 0.907 | **0.979** | **+0.072** |
-| HARD_SKILL | 0.903 | **0.946** | **+0.043** |
-| EMPLOYMENT_TERMS | 0.926 | **0.966** | +0.040 |
-| DEMOGRAPHIC | 0.875 | **0.915** | +0.041 |
-| CONTACT | 0.962 | **0.987** | +0.025 |
-| LOCATION | 0.959 | **0.966** | +0.008 |
-
-PhayaThaiBERT improves on every entity type, with the most dramatic gain on COMPENSATION (+0.200 F1).
-
-## Links
-
-- **Model:** [chayuto/thai-job-ner-phayathaibert](https://huggingface.co/chayuto/thai-job-ner-phayathaibert)
-- **WangchanBERTa variant:** [chayuto/thai-job-ner-wangchanberta](https://huggingface.co/chayuto/thai-job-ner-wangchanberta)
-- **Dataset:** [chayuto/thai-job-ner-dataset](https://huggingface.co/datasets/chayuto/thai-job-ner-dataset)
-- **Source Code:** [github.com/chayuto/thai-job-nlp-ner](https://github.com/chayuto/thai-job-nlp-ner)
+| **F1** | **0.929** |
+| Precision | 0.915 |
+| Recall | 0.944 |
 
-## Limitations
+### Model Evolution
 
-- Trained on synthetic data — may underperform on real-world posts with heavy emoji usage, OCR errors, or extreme colloquialism
-- Embeddings were frozen during training (MPS memory constraint) — unfreezing on a larger GPU may yield further gains
-- 512 token max sequence length
-- Larger model file size due to 248K vocabulary (vs WangchanBERTa's 25K)
+| Version | Data | Real-World F1 |
+|---------|------|---------------|
+| v1 (Sprint 5) | 1,253 synthetic only | 0.143 |
+| v2 (Sprint 6) | 1,057 synthetic + real | 0.935 |
+| **v3 (Sprint 7)** | **510 v3 synthetic + real (5x)** | **0.975** |
 
-## Technical Notes
+**Note:** Real-world test set is small (8 posts, 98 entities). Confidence intervals are wide. More real data is needed for robust evaluation.
+
+## Limitations
 
-- **FP16 is broken on MPS** — always use FP32 for Apple Silicon training
-- PhayaThaiBERT's 248K vocab (XLM-R-derived) requires frozen embeddings + gradient checkpointing to fit on 18GB MPS
-- Uses `offset_mapping` for tokenizer-agnostic subword-to-character alignment
-- Thai Character Cluster (TCC) boundary snapping prevents Unicode grapheme splitting during alignment
+- Trained primarily on caregiving/elderly care job posts (~70% of training data). Performance on other domains (tech, office, etc.) may be lower.
+- Real-world evaluation is based on only 8 held-out posts — results should be interpreted with caution.
+- Entity annotations are silver labels (Claude-annotated), not human gold labels.
+- max_length=256 tokens — longer posts may have entities truncated.
 
-## License
+## Citation
 
-MIT
+If you use this model, please cite:
+
+```
+@misc{thai-job-ner-phayathaibert,
+  title={Thai Job NER: Fine-tuned PhayaThaiBERT for Job Post Entity Extraction},
+  author={Chayut O.},
+  year={2026},
+  publisher={HuggingFace},
+  url={https://huggingface.co/chayuto/thai-job-ner-phayathaibert}
+}
+```