Upload fine-tuned Thai Job NER model (v3, F1=0.897)

README.md

@@ -30,18 +30,20 @@ model-index:
           - name: F1 (mixed test)
             type: f1
             value: 0.929
-          - name: Precision (real-world)
+          - name: Precision
             type: precision
             value: 0.960
-          - name: Recall (real-world)
+          - name: Recall
             type: recall
             value: 0.990
 ---
 
-# Thai Job NER — Fine-tuned PhayaThaiBERT
+# Thai Job NER — Fine-tuned PhayaThaiBERT (v2: Mixed Training)
 
 Named Entity Recognition model for extracting structured HR data from informal Thai job postings (e.g., Facebook groups, Line chats). Fine-tuned from [PhayaThaiBERT](https://huggingface.co/clicknext/phayathaibert) (~122M params).
 
+**v2 update:** Trained on style-matched synthetic + real-world data with upsampling. Real-world F1 improved from 0.143 → **0.975**.
+
 ## Model Description
 
 This model extracts 7 entity types from Thai job-related text:
@@ -72,49 +74,46 @@ for entity in results:
 
 ## Training
 
-- **Base model:** `clicknext/phayathaibert` (CamemBERT architecture, ~122M params)
-- **Training data:** 510 style-matched synthetic + 270 real posts (54 real upsampled 5x)
-- **Training strategy:** Mixed synthetic + real with real data upsampling to ensure 34.6% real data proportion
+- **Base model:** `clicknext/phayathaibert` (CamemBERT architecture, ~122M params, XLM-R-derived vocabulary)
+- **Training data:** 619 posts (510 style-matched synthetic + 54 real-world posts upsampled 5x), silver labels from GPT-4o, fuzzy-aligned to IOB2
+- **Real data proportion:** 34.6% (key factor for real-world performance)
 - **Hardware:** Apple Silicon MPS backend, FP32
 - **Hyperparameters:** LR=3e-5, warmup=0.1, batch=2, grad_accum=4, 10 epochs, gradient checkpointing, frozen embeddings
-- **Training time:** ~35 min on Apple Silicon
+- **Training time:** ~35 min
 
 ### Data Pipeline
 
-Raw Thai text + entity annotations -> fuzzy alignment with rapidfuzz + pythainlp TCC boundary snapping -> subword token mapping via offset_mapping -> IOB2-formatted HuggingFace Dataset.
+Raw Thai text + GPT-4o entity extractions → fuzzy alignment with rapidfuzz + pythainlp TCC boundary snapping → subword token mapping via offset_mapping → IOB2-formatted HuggingFace Dataset.
 
-### Key Training Insight
+### Training Strategy
 
-Real data proportion is the dominant factor for real-world performance. Our experiments showed:
+Style-matched synthetic data (v3) was generated to mimic real-world post characteristics: varied formatting, code-switching, informal language. Real-world posts were upsampled 5x to achieve a 20-35% real data proportion — the sweet spot identified through systematic experiments (see below).
 
-| Real Data % | Real-World F1 |
-|------------|---------------|
-| 2.9% | 0.771 |
-| 4.1% | 0.935 |
-| 9.6% | 0.714* |
-| 23.3% | 0.975 |
-| **34.6%** | **0.975** |
+## Evaluation
 
-(*small dataset, underfitting)
+### Real-World Test Set (8 held-out real posts, 98 entities)
 
-Style-matched synthetic data (matching real-world formatting patterns) is 3x more data-efficient than generic synthetic data.
+| Metric | Score |
+|--------|-------|
+| **F1** | **0.975** |
+| Precision | 0.960 |
+| Recall | 0.990 |
 
-## Evaluation
+### Real-World Per-Entity F1
 
-### Real-World Performance (8 held-out real posts, 98 entities)
+| Entity | F1 | Precision | Recall | Support |
+|--------|-----|-----------|--------|---------|
+| COMPENSATION | 1.000 | 1.000 | 1.000 | 15 |
+| CONTACT | 1.000 | 1.000 | 1.000 | 10 |
+| DEMOGRAPHIC | 1.000 | 1.000 | 1.000 | 14 |
+| LOCATION | 1.000 | 1.000 | 1.000 | 8 |
+| PERSON | 1.000 | 1.000 | 1.000 | 3 |
+| HARD_SKILL | 0.973 | 0.947 | 1.000 | 36 |
+| EMPLOYMENT_TERMS | 0.880 | 0.846 | 0.917 | 12 |
 
-| Entity | Precision | Recall | F1 |
-|--------|-----------|--------|-----|
-| COMPENSATION | 1.000 | 1.000 | **1.000** |
-| CONTACT | 1.000 | 1.000 | **1.000** |
-| DEMOGRAPHIC | 1.000 | 1.000 | **1.000** |
-| LOCATION | 1.000 | 1.000 | **1.000** |
-| PERSON | 1.000 | 1.000 | **1.000** |
-| HARD_SKILL | 0.947 | 1.000 | **0.973** |
-| EMPLOYMENT_TERMS | 0.846 | 0.917 | **0.880** |
-| **Overall** | **0.960** | **0.990** | **0.975** |
+5 of 7 entity types achieve **perfect F1** on real-world data.
 
-### Mixed Test Performance (78 examples)
+### Mixed Test Set (78 examples)
 
 | Metric | Score |
 |--------|-------|
@@ -122,33 +121,40 @@ Style-matched synthetic data (matching real-world formatting patterns) is 3x mor
 | Precision | 0.915 |
 | Recall | 0.944 |
 
-### Model Evolution
+### Experiment History: Closing the Real-World Gap
 
-| Version | Data | Real-World F1 |
-|---------|------|---------------|
-| v1 (Sprint 5) | 1,253 synthetic only | 0.143 |
-| v2 (Sprint 6) | 1,057 synthetic + real | 0.935 |
-| **v3 (Sprint 7)** | **510 v3 synthetic + real (5x)** | **0.975** |
+| Version | Training Data | Real-World F1 | Key Insight |
+|---------|--------------|---------------|-------------|
+| v1 (synthetic-only) | 1,253 synthetic | 0.143 | Synthetic alone fails on real data |
+| v1 (real-only) | 37 real posts | 0.558 | Small but 4x more effective per sample |
+| v1 (mixed, 4.1% real) | 1,057 mixed | 0.935 | Even a few real posts help dramatically |
+| **v2 (v3+real5x, 34.6% real)** | **619 mixed** | **0.975** | **Style-matched synthetic + upsampling** |
 
-**Note:** Real-world test set is small (8 posts, 98 entities). Confidence intervals are wide. More real data is needed for robust evaluation.
+Key finding: real data proportion is the #1 factor. Style-matched synthetic data is 3x more efficient than generic synthetic data.
+
+## Links
+
+- **Model:** [chayuto/thai-job-ner-phayathaibert](https://huggingface.co/chayuto/thai-job-ner-phayathaibert)
+- **WangchanBERTa variant:** [chayuto/thai-job-ner-wangchanberta](https://huggingface.co/chayuto/thai-job-ner-wangchanberta)
+- **Dataset:** [chayuto/thai-job-ner-dataset](https://huggingface.co/datasets/chayuto/thai-job-ner-dataset)
+- **Source Code:** [github.com/chayuto/thai-job-nlp-ner](https://github.com/chayuto/thai-job-nlp-ner)
 
 ## Limitations
 
-- Trained primarily on caregiving/elderly care job posts (~70% of training data). Performance on other domains (tech, office, etc.) may be lower.
-- Real-world evaluation is based on only 8 held-out posts — results should be interpreted with caution.
-- Entity annotations are silver labels (Claude-annotated), not human gold labels.
-- max_length=256 tokens — longer posts may have entities truncated.
+- Real-world evaluation is based on 8 held-out posts (98 entities) — confidence intervals are wide
+- EMPLOYMENT_TERMS remains the weakest entity (F1=0.880) — boundary ambiguity in schedule/contract terms
+- Embeddings were frozen during training (MPS memory constraint) — unfreezing on a larger GPU may yield further gains
+- 256 token max sequence length (covers >95% of real posts)
+- Larger model file size due to 248K vocabulary (vs WangchanBERTa's 25K)
 
-## Citation
+## Technical Notes
 
-If you use this model, please cite:
+- **FP16 is broken on MPS** — always use FP32 for Apple Silicon training
+- PhayaThaiBERT's 248K vocab (XLM-R-derived) requires frozen embeddings + gradient checkpointing to fit on 18GB MPS
+- Uses `offset_mapping` for tokenizer-agnostic subword-to-character alignment
+- Thai Character Cluster (TCC) boundary snapping prevents Unicode grapheme splitting during alignment
+- Real data upsampling (5-10x) is a simple, effective technique for low-resource scenarios
 
-```
-@misc{thai-job-ner-phayathaibert,
-  title={Thai Job NER: Fine-tuned PhayaThaiBERT for Job Post Entity Extraction},
-  author={Chayut O.},
-  year={2026},
-  publisher={HuggingFace},
-  url={https://huggingface.co/chayuto/thai-job-ner-phayathaibert}
-}
-```
+## License
+
+MIT