Publish nllb200-spm8k-directional-v5 best checkpoint (20260816-193900)

Browse files

Files changed (13) hide show

README.md +57 -54
added_tokens.json +100 -307
config.json +1 -1
eval/metrics.json +0 -0
experiment_metadata.json +129 -113
model.safetensors +2 -2
mt_standardization.py +8 -4
mt_standardization_profile.json +1 -1
publication.json +10 -10
sentencepiece.bpe.model +2 -2
special_tokens_map.json +6 -213
tokenizer_config.json +146 -2009
training_profile.json +23 -31

README.md CHANGED Viewed

@@ -42,41 +42,46 @@ model-index:
     metrics:
     - type: bleu
       name: sacreBLEU
-      value: 8.215640
     - type: chrf
       name: chrF2
-      value: 32.608422
     - type: ter
       name: TER
-      value: 84.327590
 ---
 # nllb200-zh-formosan-spm8k
 **Direction:** Traditional Chinese to Formosan<br>
 **Base model:** [`facebook/nllb-200-distilled-600M`](https://huggingface.co/facebook/nllb-200-distilled-600M)<br>
-**Recipe:** `nllb200-spm8k-directional-v3`<br>
-**Release:** `20260809-210523`, validation-selected step 270,000
 This is a directional model for 15 Formosan languages. It uses the
 `private_no_bible` leakage-controlled corpus, Formosan-aware 8k SentencePiece extension, balanced
-language/source sampling, and direction/domain/dialect control tags. The model
-weights are public, but the private training corpus is not included.
 ## Model details
 | Item | Value |
 |---|---|
 | Base revision | `f8d333a098d19b4fd9a8b18f94170487ad3f821d` |
-| Training rows | 702,520 |
 | Effective batch size | 64 |
 | Maximum sequence length | 384 |
 | Learning rate | 2e-05 |
 | Precision | `bf16` |
-| Checkpoint selection | Human validation `chrF2` |
 | Formosan text | `kindOf=standard`, `formosan-mt-standard-v3` |
-| Corpus SHA-256 | `238b96ea87716a749391d7157ce99f1611c1d4a489c4c108c315553505ca1f29` |
-| Training profile SHA-256 | `34f45832bdedc1b8322b26936dd88667dfecb360d941a7f54f60ae31a1fb4e10` |
 ## Usage
@@ -94,9 +99,15 @@ NLLB_LIDS = {'ami': 'ami_Latn', 'bnn': 'bnn_Latn', 'ckv': 'ckv_Latn', 'dru': 'dr
 def translate(text, lang_code, source_bucket="unknown", dialect="default"):
     tokenizer.src_lang = 'zho_Hant'
     prompt = (
-        f"<to_{lang_code}> <src_zh> <dom_{source_bucket}> "
-        f"<dialect_{dialect}> {text}"
     )
     inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
     output = model.generate(
@@ -116,63 +127,55 @@ when source bucket or dialect metadata is unavailable.
 ## Evaluation
-The best checkpoint was selected on human validation chrF2. Test references
-are human sentence pairs; synthetic pivots and lexical entries are train-only.
 The headline result uses `default` metadata controls, so it does not
 assume access to test-set domain or dialect labels.
 | Split | Rows |
 |---|---:|
-| Train | 702,520 |
-| Test | 64,356 |
-| Validate | 21,450 |
 | Scope | BLEU | chrF2 | TER |
 |---|---:|---:|---:|
-| Hard test | 8.22 | 32.61 | 84.33 |
-| Selection validation | 7.91 | 32.95 | 83.55 |
-Test empty-output rate: 0.0062%.
-### Confidence intervals
-Stratified bootstrap, 200 samples, 95% confidence.
-| Metric | Lower | Upper |
-|---|---:|---:|
-| BLEU | 8.09 | 8.31 |
-| chrF2 | 32.49 | 32.73 |
-| TER | 83.79 | 84.89 |
 | Language | Samples | BLEU | chrF2 | TER |
 |---|---:|---:|---:|---:|
-| `ami` | 11,271 | 7.13 | 31.39 | 82.76 |
-| `bnn` | 6,112 | 3.23 | 30.38 | 87.07 |
-| `ckv` | 3,038 | 16.38 | 42.28 | 67.39 |
-| `dru` | 6,109 | 1.75 | 23.37 | 105.90 |
-| `pwn` | 5,718 | 5.60 | 32.84 | 88.50 |
-| `pyu` | 4,280 | 9.65 | 34.37 | 76.53 |
-| `ssf` | 1,673 | 14.63 | 45.44 | 61.42 |
-| `sxr` | 1,781 | 4.92 | 43.63 | 78.33 |
-| `szy` | 2,055 | 13.24 | 38.38 | 70.71 |
-| `tao` | 1,838 | 9.86 | 33.73 | 73.43 |
-| `tay` | 6,723 | 0.49 | 12.72 | 133.93 |
-| `trv` | 6,983 | 8.86 | 30.64 | 77.67 |
-| `tsu` | 2,164 | 8.08 | 35.52 | 79.20 |
-| `xnb` | 2,728 | 16.40 | 48.69 | 62.64 |
-| `xsy` | 1,883 | 15.59 | 42.03 | 67.94 |
-The corpus gate enforces standard-tier Formosan text, at least 7.5% test and
-2.5% validation per language, human sentence-only evaluation, and zero exact,
-skeleton, one-edit, configured high character n-gram, or document
-train/evaluation conflicts. This release passed all gates: exact
 0, skeleton
 0, one-edit
 0, character n-gram
-0, and document
-0.
 See `eval/metrics.json` for sacreBLEU signatures, per-language, source,
 dialect, and length diagnostics. `publication.json` records the corpus,

     metrics:
     - type: bleu
       name: sacreBLEU
+      value: 10.520457
     - type: chrf
       name: chrF2
+      value: 35.064121
     - type: ter
       name: TER
+      value: 80.016573
 ---
 # nllb200-zh-formosan-spm8k
 **Direction:** Traditional Chinese to Formosan<br>
 **Base model:** [`facebook/nllb-200-distilled-600M`](https://huggingface.co/facebook/nllb-200-distilled-600M)<br>
+**Recipe:** `nllb200-spm8k-directional-v5`<br>
+**Release:** `20260816-193900`, validation-selected step 300,000
 This is a directional model for 15 Formosan languages. It uses the
 `private_no_bible` leakage-controlled corpus, Formosan-aware 8k SentencePiece extension, balanced
+language/source sampling, direction/language controls, fixed coarse domain
+tags, and dialect tags. Domain and dialect metadata use independent training
+dropout so `unknown` and `default` are learned inference conditions. The model
+weights are public. The training corpus is distributed separately to
+authorized FormosanBank members through the access-controlled
+[`FormosanBank/formosan-mt-private`](https://huggingface.co/datasets/FormosanBank/formosan-mt-private)
+dataset and is not included with the weights.
 ## Model details
 | Item | Value |
 |---|---|
 | Base revision | `f8d333a098d19b4fd9a8b18f94170487ad3f821d` |
+| Training rows | 625,460 |
 | Effective batch size | 64 |
 | Maximum sequence length | 384 |
 | Learning rate | 2e-05 |
 | Precision | `bf16` |
+| Checkpoint selection | Validation `chrF2` |
 | Formosan text | `kindOf=standard`, `formosan-mt-standard-v3` |
+| Corpus SHA-256 | `0024e7fcd35958455a2a060b20999e94b5e8a4364e7f54ba97ef27810746d932` |
+| Training profile SHA-256 | `4253b1b136fedebc589da942b80df71945212d1f4b92a8e98ec6fb160712f14b` |
 ## Usage
 def translate(text, lang_code, source_bucket="unknown", dialect="default"):
     tokenizer.src_lang = 'zho_Hant'
+    source_bucket = source_bucket if source_bucket in ('dictionary', 'classroom', 'narrative', 'linguistic', 'education', 'media', 'culture', 'religious', 'unknown') else "unknown"
+    domain_tag = f"<dom_{source_bucket}>"
+    if tokenizer.convert_tokens_to_ids(domain_tag) == tokenizer.unk_token_id:
+        domain_tag = "<dom_unknown>"
+    dialect_tag = f"<dialect_{dialect}>"
+    if tokenizer.convert_tokens_to_ids(dialect_tag) == tokenizer.unk_token_id:
+        dialect_tag = "<dialect_default>"
     prompt = (
+        f"<to_{lang_code}> <src_zh> {domain_tag} {dialect_tag} {text}"
     )
     inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
     output = model.generate(
 ## Evaluation
+The best checkpoint was selected on validation chrF2. Test and validation
+contain only eligible, human-translated sentence pairs. Synthetic pivots and
+lexical entries are train-only.
 The headline result uses `default` metadata controls, so it does not
 assume access to test-set domain or dialect labels.
 | Split | Rows |
 |---|---:|
+| Train | 625,460 |
+| Test | 73,591 |
+| Validate | 36,797 |
 | Scope | BLEU | chrF2 | TER |
 |---|---:|---:|---:|
+| Hard test | 10.52 | 35.06 | 80.02 |
+| Selection validation | 9.46 | 35.82 | 83.08 |
+Test empty-output rate: 0.0000%.
 | Language | Samples | BLEU | chrF2 | TER |
 |---|---:|---:|---:|---:|
+| `ami` | 14,348 | 8.40 | 30.97 | 83.12 |
+| `bnn` | 6,048 | 7.78 | 36.24 | 84.72 |
+| `ckv` | 2,871 | 22.62 | 48.09 | 60.42 |
+| `dru` | 6,588 | 4.03 | 28.21 | 97.14 |
+| `pwn` | 5,945 | 6.09 | 33.16 | 90.98 |
+| `pyu` | 4,334 | 11.81 | 37.37 | 76.99 |
+| `ssf` | 2,150 | 21.53 | 50.02 | 56.10 |
+| `sxr` | 1,968 | 7.19 | 41.72 | 86.80 |
+| `szy` | 2,300 | 17.42 | 43.22 | 67.77 |
+| `tao` | 2,360 | 12.30 | 36.10 | 72.84 |
+| `tay` | 7,161 | 4.62 | 24.52 | 92.61 |
+| `trv` | 10,728 | 10.91 | 33.51 | 72.50 |
+| `tsu` | 2,048 | 12.91 | 38.60 | 73.06 |
+| `xnb` | 2,590 | 18.48 | 50.69 | 61.18 |
+| `xsy` | 2,152 | 21.81 | 47.25 | 63.76 |
+The corpus gate enforces standard-tier Formosan text, 5/10
+validation/test proportions from all deduplicated pairs, capacity-aware source
+balance, sentence-only evaluation, and
+zero exact, skeleton, one-edit, or configured high character n-gram
+train/evaluation conflicts. Document overlap is diagnostic. This release
+passed all leakage gates: exact
 0, skeleton
 0, one-edit
 0, character n-gram
+0. Document overlap:
+1439.
 See `eval/metrics.json` for sacreBLEU signatures, per-language, source,
 dialect, and length diagnostics. `publication.json` records the corpus,

added_tokens.json CHANGED Viewed

@@ -1,309 +1,102 @@
 {
-  "<dialect_central>": 261372,
-  "<dialect_coastal>": 261373,
-  "<dialect_dawu>": 261374,
-  "<dialect_default>": 261375,
-  "<dialect_deluvalley>": 261376,
-  "<dialect_dona>": 261377,
-  "<dialect_duda>": 261378,
-  "<dialect_eastern>": 261379,
-  "<dialect_fourseasons>": 261380,
-  "<dialect_hengchun>": 261381,
-  "<dialect_jianhe>": 261382,
-  "<dialect_junqun>": 261383,
-  "<dialect_kanakanavu>": 261384,
-  "<dialect_kaqun>": 261385,
-  "<dialect_kavalan>": 261386,
-  "<dialect_luanqun>": 261387,
-  "<dialect_malan>": 261388,
-  "<dialect_maolin>": 261389,
-  "<dialect_nanwang>": 261390,
-  "<dialect_northern>": 261391,
-  "<dialect_saaroa>": 261392,
-  "<dialect_saisiyat>": 261393,
-  "<dialect_sakizaya>": 261394,
-  "<dialect_sekolik>": 261395,
-  "<dialect_southern>": 261396,
-  "<dialect_tanqun>": 261397,
-  "<dialect_tegudaya>": 261398,
-  "<dialect_thao>": 261399,
-  "<dialect_truku>": 261400,
-  "<dialect_tsou>": 261401,
-  "<dialect_unknown>": 261402,
-  "<dialect_wanda>": 261403,
-  "<dialect_wanshan>": 261404,
-  "<dialect_wenshui>": 261405,
-  "<dialect_wutai>": 261406,
-  "<dialect_xiqun>": 261407,
-  "<dialect_xiuguluan>": 261408,
-  "<dialect_yami>": 261409,
-  "<dialect_yilanzeaol>": 261410,
-  "<dialect_zeaol>": 261411,
-  "<dialect_zhiben>": 261412,
-  "<dialect_zhuoqun>": 261413,
-  "<dom_classroom_context>": 261414,
-  "<dom_culture>": 261415,
-  "<dom_dictionary>": 261416,
-  "<dom_essays>": 261417,
-  "<dom_formosan_100_paiwan_texts>": 261418,
-  "<dom_formosan_academia_sinica_oral_legends>": 261419,
-  "<dom_formosan_adlay_kun_long_liu_syntactic_interactio>": 261420,
-  "<dom_formosan_adlay_liu_lfg_relativisation_jianshi_sq>": 261421,
-  "<dom_formosan_amis_adversative_constructions>": 261422,
-  "<dom_formosan_amis_huang_english_phonology_grammar>": 261423,
-  "<dom_formosan_amis_kavalan_lin_interrogative_verbs>": 261424,
-  "<dom_formosan_amis_kuo_sung_comparative_constructions>": 261425,
-  "<dom_formosan_amis_myths_and_customs>": 261426,
-  "<dom_formosan_amis_pa_verbs>": 261427,
-  "<dom_formosan_amis_serial_verb_constructions>": 261428,
-  "<dom_formosan_amis_tung_chiou>": 261429,
-  "<dom_formosan_amy_lee_body_part_nomenclature_seediq>": 261430,
-  "<dom_formosan_anna_hsiou_chuan_chang_reference_gramma>": 261431,
-  "<dom_formosan_anton_quack_puyuma_pulingaw>": 261432,
-  "<dom_formosan_apay_tang_anger_happiness_truku_seediq>": 261433,
-  "<dom_formosan_asai_sedik_language>": 261434,
-  "<dom_formosan_atayal_you_nao_savak_raxayal>": 261435,
-  "<dom_formosan_bien_daniel_chiang_house_social_hierarc>": 261436,
-  "<dom_formosan_boyizhenu_narrative_oral_literature_tap>": 261437,
-  "<dom_formosan_bunun_debusser_dissertation>": 261438,
-  "<dom_formosan_bunun_topic_focus>": 261439,
-  "<dom_formosan_c_douglas_chretien_oceanic_material_in>": 261440,
-  "<dom_formosan_catherine_tseng_seediq_atayal_foods_the>": 261441,
-  "<dom_formosan_chang_kwo_tan_syncretic_objects_materia>": 261442,
-  "<dom_formosan_chaolin_li_motion_verb_grammaticalizati>": 261443,
-  "<dom_formosan_chen_fukuda_relabeling_ergative>": 261444,
-  "<dom_formosan_chen_fukuda_three_ways_steal>": 261445,
-  "<dom_formosan_cheng_sung_modality_kanakanavu>": 261446,
-  "<dom_formosan_chi_lu_chen_and_micheal_d_coe_an_invest>": 261447,
-  "<dom_formosan_chuming_wu_linking_constructions_mayrin>": 261448,
-  "<dom_formosan_chun_mei_chen_comparative_phonology_pai>": 261449,
-  "<dom_formosan_chunming_wu_two_types_of_noun_incorpora>": 261450,
-  "<dom_formosan_cip_atayal_grammar_overview>": 261451,
-  "<dom_formosan_claire_mcgill_a_brief_tayal_vocabulary>": 261452,
-  "<dom_formosan_david_blundell_ed_austronesian_taiwan>": 261453,
-  "<dom_formosan_dean_johnson_year_clouds_smangus>": 261454,
-  "<dom_formosan_der_hwa_victoria_rau_grammar_atayal>": 261455,
-  "<dom_formosan_der_hwa_victoria_rau_lexical_similarity>": 261456,
-  "<dom_formosan_dong_manv_yami_ode_to_taro>": 261457,
-  "<dom_formosan_dong_yi_lin_universal_quantifier_kavala>": 261458,
-  "<dom_formosan_dong_yijia_tjuwabar_paiwan_ritual_langu>": 261459,
-  "<dom_formosan_dorinda_tsai_hsiu_liu_neutral_imperfect>": 261460,
-  "<dom_formosan_ed_torjesen_amis_taiwan_word_list>": 261461,
-  "<dom_formosan_egerod_origin_headhunting_atayal_text_v>": 261462,
-  "<dom_formosan_egerod_soren_word_order_word_classes_at>": 261463,
-  "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>": 261464,
-  "<dom_formosan_elizabeth_zeitoun_lillian_huang_anna_ch>": 261465,
-  "<dom_formosan_elizabeth_zeitoun_lillian_huang_marie_y>": 261466,
-  "<dom_formosan_elizabeth_zeitoun_pronominal_system_man>": 261467,
-  "<dom_formosan_elizabeth_zeitoun_squib_dynamic_vs_stat>": 261468,
-  "<dom_formosan_epark>": 261469,
-  "<dom_formosan_ferrell_taiwan_aboriginal_groups_proble>": 261470,
-  "<dom_formosan_george_leroy_shelley_vudai_rukai>": 261471,
-  "<dom_formosan_gitbook_translations>": 261472,
-  "<dom_formosan_glosbe>": 261473,
-  "<dom_formosan_gujing_lin_tsou_serial_verb_constructio>": 261474,
-  "<dom_formosan_henry_chang_nominal_aspect_tsou>": 261475,
-  "<dom_formosan_henry_chang_tsou_causative_applicative>": 261476,
-  "<dom_formosan_henry_y_chang_formosan_speech_act_mood>": 261477,
-  "<dom_formosan_ho_dan_phonological_system_butonglu_pai>": 261478,
-  "<dom_formosan_holmer_seediq>": 261479,
-  "<dom_formosan_hsiu_chuan_liao_transitivity_ergativity>": 261480,
-  "<dom_formosan_huang_dongqiu_atayal_bunun_amis_languag>": 261481,
-  "<dom_formosan_huang_grammaticalization_squliq_atayal>": 261482,
-  "<dom_formosan_huang_hui_chuan_glide_formation_takitud>": 261483,
-  "<dom_formosan_huang_mei_chin_atayal_reference_grammar>": 261484,
-  "<dom_formosan_hui_chuan_j_huang_syllable_types_in_bun>": 261485,
-  "<dom_formosan_hui_shan_lin_squliq_atayal_reduplicatio>": 261486,
-  "<dom_formosan_huteson_rukai_survey>": 261487,
-  "<dom_formosan_ian_maddieson_and_richard_wright_the_vo>": 261488,
-  "<dom_formosan_ilrdf_42_language_practice_word_lists>": 261489,
-  "<dom_formosan_ilrdf_tousvusvutu_kita>": 261490,
-  "<dom_formosan_indigenous_education_resource_center_ke>": 261491,
-  "<dom_formosan_indigenous_language_literary_awards>": 261492,
-  "<dom_formosan_isabelle_bril_roots_and_stems_lexical_a>": 261493,
-  "<dom_formosan_jian_iwan_guo_qingliu_life_history>": 261494,
-  "<dom_formosan_jian_shilang_introduction_thao_grammar>": 261495,
-  "<dom_formosan_john_tu_er_wei_notes_on_the_religious_b>": 261496,
-  "<dom_formosan_john_u_wolff_the_proto_austronesian_pho>": 261497,
-  "<dom_formosan_jonathan_kuo_amis_manner_result_verbs>": 261498,
-  "<dom_formosan_josiane_cauquelin_aborigines_taiwan_puy>": 261499,
-  "<dom_formosan_joy_wu_clausal_modifiers_in_amis>": 261500,
-  "<dom_formosan_joy_wu_verb_classification_case_marking>": 261501,
-  "<dom_formosan_kanakanavu_texts>": 261502,
-  "<dom_formosan_kavalan_zhang_reference_grammar>": 261503,
-  "<dom_formosan_kumu_tapas_tribal_memory_oral_history_w>": 261504,
-  "<dom_formosan_kuo_j_c_argument_alternation_and_argume>": 261505,
-  "<dom_formosan_lei_shih_family_system_paiwan_at_su_pai>": 261506,
-  "<dom_formosan_li_chen_yeh_developing_a_hla_alua_learn>": 261507,
-  "<dom_formosan_li_may_sung_budai_rukai_exclamatives>": 261508,
-  "<dom_formosan_li_may_sung_clausal_nominalization_buda>": 261509,
-  "<dom_formosan_lillian_huang_atayal_participants_cross>": 261510,
-  "<dom_formosan_lillian_huang_ergativity_atayal>": 261511,
-  "<dom_formosan_liu_manyi_kaskun_ata_matas_i_ki_quaz>": 261512,
-  "<dom_formosan_malcom_d_ross_the_history_and_transitiv>": 261513,
-  "<dom_formosan_malcom_ross_proto_austronesian_verbal_m>": 261514,
-  "<dom_formosan_man_ni_chu_the_measurement_of_final_i_v>": 261515,
-  "<dom_formosan_matsuzawa_kazuko_the_social_and_ritual>": 261516,
-  "<dom_formosan_maya_yeh_blaq_uv_construction_atayal>": 261517,
-  "<dom_formosan_mayumi_oiwa_adverbial_verb_construction>": 261518,
-  "<dom_formosan_mayumi_oiwa_bungard_morphology_and_synt>": 261519,
-  "<dom_formosan_mayumi_oiwa_possessor_raising_truku_see>": 261520,
-  "<dom_formosan_mei_chin_huang_verb_classification_in_m>": 261521,
-  "<dom_formosan_melissa_shih_hui_lin_sakizaya_or_amis_a>": 261522,
-  "<dom_formosan_nanang_tadaw_naci_mowna_pgagu>": 261523,
-  "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>": 261524,
-  "<dom_formosan_naomi_tsukida_ditransitive_causative_se>": 261525,
-  "<dom_formosan_nowbucyang_truku_thesis>": 261526,
-  "<dom_formosan_ochiai_izumi_2016_comparative_linguisti>": 261527,
-  "<dom_formosan_ochiai_izumi_bu_hwan_vocabulary_recorde>": 261528,
-  "<dom_formosan_ochiai_izumi_numerals_paran_seediq_rela>": 261529,
-  "<dom_formosan_ochiai_izumi_on_a_seediq_glossary_recor>": 261530,
-  "<dom_formosan_old_texts>": 261531,
-  "<dom_formosan_paiwan_ho_five_dialects>": 261532,
-  "<dom_formosan_paiwanstories>": 261533,
-  "<dom_formosan_pang_hsin_ting_reconstruction_proto_puy>": 261534,
-  "<dom_formosan_patricia_stanley_morphophonemics_of_ver>": 261535,
-  "<dom_formosan_paul_jen_kuei_li_comparative_bunun_dial>": 261536,
-  "<dom_formosan_paul_jen_kuei_li_conjunction_thao>": 261537,
-  "<dom_formosan_paul_jen_kuei_li_internal_relationships>": 261538,
-  "<dom_formosan_paul_jen_kuei_li_origins_of_the_east_fo>": 261539,
-  "<dom_formosan_paul_jen_kuei_li_review_of_taiwan_abori>": 261540,
-  "<dom_formosan_paul_jen_kuei_li_rukai_structure>": 261541,
-  "<dom_formosan_paul_jen_kuei_li_some_plant_names_in_fo>": 261542,
-  "<dom_formosan_paul_jen_kuei_li_the_preglottalised_sto>": 261543,
-  "<dom_formosan_paul_li_atayalic_final_voiced_stops>": 261544,
-  "<dom_formosan_paul_li_case_marking_four_formosan_lang>": 261545,
-  "<dom_formosan_paul_li_position_atayal_austronesian>": 261546,
-  "<dom_formosan_pingdong_county_government_kai_na_sepuc>": 261547,
-  "<dom_formosan_puyuma_chen_raising_to_object>": 261548,
-  "<dom_formosan_puyuma_katipol_kopfjagdriten>": 261549,
-  "<dom_formosan_puyuma_teng_pronominal_systems>": 261550,
-  "<dom_formosan_puyuma_teng_reference_grammar>": 261551,
-  "<dom_formosan_raleigh_ferrell_construction_markers_fo>": 261552,
-  "<dom_formosan_raleigh_ferrell_paiwan_phonology_and_pr>": 261553,
-  "<dom_formosan_raleigh_j_ferrell_intent_and_volition_i>": 261554,
-  "<dom_formosan_rik_bunun>": 261555,
-  "<dom_formosan_rik_de_busser_the_influence_of_christia>": 261556,
-  "<dom_formosan_robert_blust_a_note_on_covert_structure>": 261557,
-  "<dom_formosan_robert_blust_austronesian_homeland_ling>": 261558,
-  "<dom_formosan_robert_blust_some_remarks_linguistic_po>": 261559,
-  "<dom_formosan_robert_blust_thao_triplication>": 261560,
-  "<dom_formosan_robert_blust_the_austronesian_languages>": 261561,
-  "<dom_formosan_rukai_mantauran_stories>": 261562,
-  "<dom_formosan_rukai_texts>": 261563,
-  "<dom_formosan_rukai_zeitoun_saying>": 261564,
-  "<dom_formosan_sakizaya_affixes>": 261565,
-  "<dom_formosan_schetelig_mittheilungen_uber_die_sprach>": 261566,
-  "<dom_formosan_seals>": 261567,
-  "<dom_formosan_seediq_zhang_reference_grammar>": 261568,
-  "<dom_formosan_shigeru_tsuchida_and_isidore_dyen_proto>": 261569,
-  "<dom_formosan_shigeru_tsuchida_kanakanavu_texts>": 261570,
-  "<dom_formosan_shigeru_tsuchida_preliminary_reports_on>": 261571,
-  "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>": 261572,
-  "<dom_formosan_shih_chi_stella_yeh_stress_and_quantity>": 261573,
-  "<dom_formosan_shih_rukai_adverbial>": 261574,
-  "<dom_formosan_sihwei_chen_and_haowen_jiang_ways_of_ta>": 261575,
-  "<dom_formosan_sihwei_chen_lexical_aspect_atayal>": 261576,
-  "<dom_formosan_soren_egerod_statement_atayal_phonology>": 261577,
-  "<dom_formosan_soren_egerod_verb_inflection_atayal>": 261578,
-  "<dom_formosan_stacy_teng_malcom_ross_is_puyuma_primar>": 261579,
-  "<dom_formosan_stacy_wan_tin_huang_functions_yami_verb>": 261580,
-  "<dom_formosan_sung_kuo_descriptive_comparative_constr>": 261581,
-  "<dom_formosan_taoshan_elementary_school_atelier_hui_k>": 261582,
-  "<dom_formosan_teresa_m_chen_verbal_construction_and_v>": 261583,
-  "<dom_formosan_ting_chi_wei_parallelism_in_amis_sluici>": 261584,
-  "<dom_formosan_tingchun_chen_successive_cyclic_case_as>": 261585,
-  "<dom_formosan_truku_lowking_demonstratives>": 261586,
-  "<dom_formosan_tsai_hsui_liu_complementation_three_lan>": 261587,
-  "<dom_formosan_tsai_wei_tien_dylan_conjunctive_reducti>": 261588,
-  "<dom_formosan_tsou_descriptive_study>": 261589,
-  "<dom_formosan_tsukida_naomi_verb_classification_amis>": 261590,
-  "<dom_formosan_tu_wen_chiu_a_synchronic_classification>": 261591,
-  "<dom_formosan_tung_tsuchida_ting_li_pan_saaroa_texts>": 261592,
-  "<dom_formosan_victoria_rau_yi_hsin_wu_and_meng_chien>": 261593,
-  "<dom_formosan_wei_huilin_liu_social_structure_yami>": 261594,
-  "<dom_formosan_wei_matri_clan_lineage_system_ami>": 261595,
-  "<dom_formosan_wei_tien_tsai_subjecthood_temporal_adju>": 261596,
-  "<dom_formosan_wiedfelt_formosan_aborigines_atayal>": 261597,
-  "<dom_formosan_wilang_yutas_videos>": 261598,
-  "<dom_formosan_wu_chunming_adverbials_in_paiwan>": 261599,
-  "<dom_formosan_xie_fuhui_introduction_kavalan_grammar>": 261600,
-  "<dom_formosan_yedda_palemeq_blog>": 261601,
-  "<dom_formosan_yeddas_blog>": 261602,
-  "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>": 261603,
-  "<dom_formosan_yi_ming_chou_object_control_saisiyat>": 261604,
-  "<dom_formosan_yi_ting_chen_a_minimalist_approach_to_a>": 261605,
-  "<dom_formosan_yi_ting_chen_amis_left_periphery>": 261606,
-  "<dom_formosan_yi_ting_chen_language_use_code_mixing_a>": 261607,
-  "<dom_formosan_yi_yang_cheng_kanakanavu_word_level_pro>": 261608,
-  "<dom_formosan_yi_yang_cheng_tense_aspect_agent_markin>": 261609,
-  "<dom_formosan_yoshiro_nihira_bunun_vocabulary>": 261610,
-  "<dom_formosan_yuehchien_chien_the_lexical_system_of_y>": 261611,
-  "<dom_formosan_yukihiro_yamada_ying_chu_liao_phonology>": 261612,
-  "<dom_formosan_zeng_shifen_reduplication_affixation_pa>": 261613,
-  "<dom_formosan_zhan_sujuan_taiwan_indigenous_peoples_h>": 261614,
-  "<dom_formosan_zhang_xiujuan_introduction_paiwan_gramm>": 261615,
-  "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>": 261616,
-  "<dom_formosan_zhao_shanhe_li_taiyuan_zhu_qingyi_abori>": 261617,
-  "<dom_formosan_zheng_acl_2024>": 261618,
-  "<dom_formosan_zheng_data>": 261619,
-  "<dom_formosan_zheng_yumei_zhu_qingyi_bo_hongming_abor>": 261620,
-  "<dom_learning_vocab>": 261621,
-  "<dom_nine_level>": 261622,
-  "<dom_ntu>": 261623,
-  "<dom_picture_book>": 261624,
-  "<dom_picture_story>": 261625,
-  "<dom_presidential_apology>": 261626,
-  "<dom_reading_writing>": 261627,
-  "<dom_unknown>": 261628,
-  "<dom_youtube>": 261629,
-  "<src_ami>": 261630,
-  "<src_bnn>": 261631,
-  "<src_ckv>": 261632,
-  "<src_dru>": 261633,
-  "<src_eng>": 261634,
-  "<src_pwn>": 261635,
-  "<src_pyu>": 261636,
-  "<src_ssf>": 261637,
-  "<src_sxr>": 261638,
-  "<src_szy>": 261639,
-  "<src_tao>": 261640,
-  "<src_tay>": 261641,
-  "<src_trv>": 261642,
-  "<src_tsu>": 261643,
-  "<src_xnb>": 261644,
-  "<src_xsy>": 261645,
-  "<src_zh>": 261646,
-  "<to_ami>": 261647,
-  "<to_bnn>": 261648,
-  "<to_ckv>": 261649,
-  "<to_dru>": 261650,
-  "<to_eng>": 261651,
-  "<to_pwn>": 261652,
-  "<to_pyu>": 261653,
-  "<to_ssf>": 261654,
-  "<to_sxr>": 261655,
-  "<to_szy>": 261656,
-  "<to_tao>": 261657,
-  "<to_tay>": 261658,
-  "<to_trv>": 261659,
-  "<to_tsu>": 261660,
-  "<to_xnb>": 261661,
-  "<to_xsy>": 261662,
-  "<to_zh>": 261663,
-  "ami_Latn": 261357,
-  "bnn_Latn": 261358,
-  "ckv_Latn": 261359,
-  "dru_Latn": 261360,
-  "pwn_Latn": 261361,
-  "pyu_Latn": 261362,
-  "ssf_Latn": 261363,
-  "sxr_Latn": 261364,
-  "szy_Latn": 261365,
-  "tao_Latn": 261366,
-  "tay_Latn": 261367,
-  "trv_Latn": 261368,
-  "tsu_Latn": 261369,
-  "xnb_Latn": 261370,
-  "xsy_Latn": 261371
 }

 {
+  "<dialect_central>": 261440,
+  "<dialect_coastal>": 261441,
+  "<dialect_dawu>": 261442,
+  "<dialect_default>": 261443,
+  "<dialect_deluvalley>": 261444,
+  "<dialect_dona>": 261445,
+  "<dialect_duda>": 261446,
+  "<dialect_eastern>": 261447,
+  "<dialect_fourseasons>": 261448,
+  "<dialect_hengchun>": 261449,
+  "<dialect_jianhe>": 261450,
+  "<dialect_junqun>": 261451,
+  "<dialect_kanakanavu>": 261452,
+  "<dialect_kaqun>": 261453,
+  "<dialect_kavalan>": 261454,
+  "<dialect_luanqun>": 261455,
+  "<dialect_malan>": 261456,
+  "<dialect_maolin>": 261457,
+  "<dialect_nanwang>": 261458,
+  "<dialect_northern>": 261459,
+  "<dialect_saaroa>": 261460,
+  "<dialect_saisiyat>": 261461,
+  "<dialect_sakizaya>": 261462,
+  "<dialect_sekolik>": 261463,
+  "<dialect_southern>": 261464,
+  "<dialect_tanqun>": 261465,
+  "<dialect_tegudaya>": 261466,
+  "<dialect_thao>": 261467,
+  "<dialect_truku>": 261468,
+  "<dialect_tsou>": 261469,
+  "<dialect_unknown>": 261470,
+  "<dialect_wanda>": 261471,
+  "<dialect_wanshan>": 261472,
+  "<dialect_wenshui>": 261473,
+  "<dialect_wutai>": 261474,
+  "<dialect_xiqun>": 261475,
+  "<dialect_xiuguluan>": 261476,
+  "<dialect_yami>": 261477,
+  "<dialect_yilanzeaol>": 261478,
+  "<dialect_zeaol>": 261479,
+  "<dialect_zhiben>": 261480,
+  "<dialect_zhuoqun>": 261481,
+  "<dom_classroom>": 261482,
+  "<dom_culture>": 261483,
+  "<dom_dictionary>": 261484,
+  "<dom_education>": 261485,
+  "<dom_linguistic>": 261486,
+  "<dom_media>": 261487,
+  "<dom_narrative>": 261488,
+  "<dom_religious>": 261489,
+  "<dom_unknown>": 261490,
+  "<src_ami>": 261491,
+  "<src_bnn>": 261492,
+  "<src_ckv>": 261493,
+  "<src_dru>": 261494,
+  "<src_eng>": 261495,
+  "<src_pwn>": 261496,
+  "<src_pyu>": 261497,
+  "<src_ssf>": 261498,
+  "<src_sxr>": 261499,
+  "<src_szy>": 261500,
+  "<src_tao>": 261501,
+  "<src_tay>": 261502,
+  "<src_trv>": 261503,
+  "<src_tsu>": 261504,
+  "<src_xnb>": 261505,
+  "<src_xsy>": 261506,
+  "<src_zh>": 261507,
+  "<to_ami>": 261508,
+  "<to_bnn>": 261509,
+  "<to_ckv>": 261510,
+  "<to_dru>": 261511,
+  "<to_eng>": 261512,
+  "<to_pwn>": 261513,
+  "<to_pyu>": 261514,
+  "<to_ssf>": 261515,
+  "<to_sxr>": 261516,
+  "<to_szy>": 261517,
+  "<to_tao>": 261518,
+  "<to_tay>": 261519,
+  "<to_trv>": 261520,
+  "<to_tsu>": 261521,
+  "<to_xnb>": 261522,
+  "<to_xsy>": 261523,
+  "<to_zh>": 261524,
+  "ami_Latn": 261425,
+  "bnn_Latn": 261426,
+  "ckv_Latn": 261427,
+  "dru_Latn": 261428,
+  "pwn_Latn": 261429,
+  "pyu_Latn": 261430,
+  "ssf_Latn": 261431,
+  "sxr_Latn": 261432,
+  "szy_Latn": 261433,
+  "tao_Latn": 261434,
+  "tay_Latn": 261435,
+  "trv_Latn": 261436,
+  "tsu_Latn": 261437,
+  "xnb_Latn": 261438,
+  "xsy_Latn": 261439
 }

config.json CHANGED Viewed

@@ -30,5 +30,5 @@
   "tokenizer_class": "NllbTokenizer",
   "transformers_version": "4.56.1",
   "use_cache": true,
-  "vocab_size": 261664
 }

   "tokenizer_class": "NllbTokenizer",
   "transformers_version": "4.56.1",
   "use_cache": true,
+  "vocab_size": 261525
 }

eval/metrics.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

experiment_metadata.json CHANGED Viewed

@@ -1,271 +1,287 @@
 {
   "model_family": "nllb",
-  "recipe_id": "nllb200-spm8k-directional-v3",
   "mt_standardization": {
     "id": "formosan-mt-standard-v3",
-    "sha256": "4bbded87eb4833b2d1cd5a88a1f2b059560416e0e69ecfa55433a77f418c903e",
     "namespace": "formosan-mt"
   },
-  "step": 270000,
   "best_metric": "chrF2",
-  "best_value": 32.94932709039752,
   "direction": "zh2f",
-  "run_contract_sha256": "e2006ab32a0dc5494687f7247e4eca1c15ede5b1565fa5d853d6aea8b3f901b3",
   "validation": {
-    "mean_token_loss": 3.722221616097202,
-    "ppl": 41.356169658075665,
     "by_language": {
-      "ami": 4.227757943682411,
-      "bnn": 4.41658405355141,
-      "ckv": 3.316785193118075,
-      "dru": 4.270603088675659,
-      "pwn": 3.852267307491185,
-      "pyu": 2.6538681161539395,
-      "ssf": 3.8765744227177934,
-      "sxr": 2.887809524761298,
-      "szy": 3.772529477844695,
-      "tao": 4.109648863775561,
-      "tay": 3.8526229924424076,
-      "trv": 3.817608572327577,
-      "tsu": 4.129769822751514,
-      "xnb": 1.5115395753407543,
-      "xsy": 3.574612971363336
     },
     "generation": {
       "samples": 1920,
       "bleu_tokenize": "13a",
       "global": {
-        "BLEU": 7.909769982143852,
-        "chrF2": 32.94932709039752,
-        "TER": 83.55092837457896,
-        "exact_match_rate": 0.0067708333333333336,
-        "empty_output_rate": 0.0005208333333333333,
-        "character_length_ratio": 0.8571467578282855,
         "signatures": {
           "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
           "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
           "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-        }
       },
       "by_language": {
         "ami": {
           "samples": 128,
-          "BLEU": 3.2466653393626124,
-          "chrF2": 24.480302322491998,
-          "TER": 85.56701030927834,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.7305336832895888,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "bnn": {
           "samples": 128,
-          "BLEU": 4.118774930501576,
-          "chrF2": 30.584133368500645,
-          "TER": 86.47029455816275,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8470179925107316,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "ckv": {
           "samples": 128,
-          "BLEU": 14.771329924800888,
-          "chrF2": 40.18559503002575,
-          "TER": 68.66558616963803,
-          "exact_match_rate": 0.0234375,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8627385336457984,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "dru": {
           "samples": 128,
-          "BLEU": 4.992638352657695,
-          "chrF2": 29.360221486825978,
-          "TER": 91.61729857819904,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.7757179759914906,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "pwn": {
           "samples": 128,
-          "BLEU": 2.918474231442962,
-          "chrF2": 27.445614146838217,
-          "TER": 95.16797054763,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8880145278450363,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "pyu": {
           "samples": 128,
-          "BLEU": 14.412041793204803,
-          "chrF2": 42.43115177228584,
-          "TER": 64.53407510431154,
-          "exact_match_rate": 0.0078125,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8500426015336552,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "ssf": {
           "samples": 128,
-          "BLEU": 8.729724338324955,
-          "chrF2": 36.98782330150638,
-          "TER": 78.41666666666667,
-          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.888234394124847,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "sxr": {
           "samples": 128,
-          "BLEU": 3.888034502159744,
-          "chrF2": 43.87195350169566,
-          "TER": 77.5524981897176,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8962462628723287,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "szy": {
           "samples": 128,
-          "BLEU": 11.476377973476632,
-          "chrF2": 33.101270259045116,
-          "TER": 74.21660252886201,
           "exact_match_rate": 0.0234375,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.686102147546456,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "tao": {
           "samples": 128,
-          "BLEU": 11.563453709773146,
-          "chrF2": 34.534905111358746,
-          "TER": 77.18073970690858,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9344822232572582,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "tay": {
           "samples": 128,
-          "BLEU": 3.5225862239374814,
-          "chrF2": 25.613045414217467,
-          "TER": 115.18828451882845,
           "exact_match_rate": 0.0,
-          "empty_output_rate": 0.0078125,
-          "character_length_ratio": 1.027499310217971,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "trv": {
           "samples": 128,
-          "BLEU": 9.419553928223658,
-          "chrF2": 32.22859707980392,
-          "TER": 75.03660322108345,
-          "exact_match_rate": 0.0078125,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8475278070958837,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "tsu": {
           "samples": 128,
-          "BLEU": 3.731604123794413,
-          "chrF2": 24.065562206452924,
-          "TER": 101.12576391122548,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.937484292535813,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "xnb": {
           "samples": 128,
-          "BLEU": 22.814365243298425,
-          "chrF2": 56.07023167294509,
-          "TER": 52.06684256816183,
-          "exact_match_rate": 0.0390625,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9557964970809008,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         },
         "xsy": {
           "samples": 128,
-          "BLEU": 8.402288460536255,
-          "chrF2": 34.99291589155046,
-          "TER": 81.05857865001487,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8809847901994436,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
-          }
         }
       }
     },
-    "step": 270000,
     "selection": {
       "metric": "chrF2",
-      "value": 32.94932709039752,
       "improved": true,
-      "best_value_before_eval": 32.63613273062232
     }
   }
 }

 {
   "model_family": "nllb",
+  "recipe_id": "nllb200-spm8k-directional-v5",
   "mt_standardization": {
     "id": "formosan-mt-standard-v3",
+    "sha256": "3c93c1abecfc68c72e3bb1d547fbe0bdebb14bb74e7a5d3c5c519bd18d34bf6f",
     "namespace": "formosan-mt"
   },
+  "step": 300000,
   "best_metric": "chrF2",
+  "best_value": 35.82202484658809,
   "direction": "zh2f",
+  "run_contract_sha256": "cc251cf00ec88f29e0d8407267f5f6d00ec19fe2fc5adfd7ab8f61c20c4ec897",
   "validation": {
+    "mean_token_loss": 3.3005945952051094,
+    "ppl": 27.12876475945609,
     "by_language": {
+      "ami": 2.9977749379532206,
+      "bnn": 3.690821646233282,
+      "ckv": 2.94499888262739,
+      "dru": 4.181627018271752,
+      "pwn": 3.4254762849185942,
+      "pyu": 3.304404493729452,
+      "ssf": 2.497812805786773,
+      "sxr": 3.7283250016987934,
+      "szy": 3.4226339202524496,
+      "tao": 4.027004377501526,
+      "tay": 3.8187424449188474,
+      "trv": 3.3320100487357895,
+      "tsu": 3.1443220605594697,
+      "xnb": 2.505188786058678,
+      "xsy": 2.4550011498098887
     },
     "generation": {
       "samples": 1920,
       "bleu_tokenize": "13a",
       "global": {
+        "BLEU": 9.455191646228812,
+        "chrF2": 35.82202484658809,
+        "TER": 83.08488862131246,
+        "exact_match_rate": 0.0020833333333333333,
+        "empty_output_rate": 0.0,
+        "character_length_ratio": 0.9482003611684212,
         "signatures": {
           "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
           "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
           "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+        },
+        "source_copy_rate": 0.0
       },
       "by_language": {
         "ami": {
           "samples": 128,
+          "BLEU": 6.471396361006349,
+          "chrF2": 30.301236626781282,
+          "TER": 90.9090909090909,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.8739165329052969,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "bnn": {
           "samples": 128,
+          "BLEU": 5.466768069013215,
+          "chrF2": 33.750602012686926,
+          "TER": 88.137321549966,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9296519493297257,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "ckv": {
           "samples": 128,
+          "BLEU": 15.527114881978997,
+          "chrF2": 41.66076326407682,
+          "TER": 74.10375369042598,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9610020876826723,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "dru": {
           "samples": 128,
+          "BLEU": 2.209520710659227,
+          "chrF2": 27.983762152508092,
+          "TER": 101.71972191730698,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.0066437007874016,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "pwn": {
           "samples": 128,
+          "BLEU": 4.6023247601103465,
+          "chrF2": 33.737940135980466,
+          "TER": 89.06685236768801,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9638833451361617,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "pyu": {
           "samples": 128,
+          "BLEU": 9.007496571772867,
+          "chrF2": 35.476914949292,
+          "TER": 84.61979913916787,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9999442431000837,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "ssf": {
           "samples": 128,
+          "BLEU": 18.004249116567237,
+          "chrF2": 47.24722590802964,
+          "TER": 59.93303571428571,
+          "exact_match_rate": 0.0078125,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.8978708791208792,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "sxr": {
           "samples": 128,
+          "BLEU": 3.610289433997478,
+          "chrF2": 38.23203639485925,
+          "TER": 91.54639175257732,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.957137966449846,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "szy": {
           "samples": 128,
+          "BLEU": 12.018448268563786,
+          "chrF2": 38.38498932105344,
+          "TER": 79.93158494868872,
           "exact_match_rate": 0.0234375,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.0317768627146298,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "tao": {
           "samples": 128,
+          "BLEU": 9.388220471677311,
+          "chrF2": 32.52694722566377,
+          "TER": 73.00040600893219,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.8613728129205922,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "tay": {
           "samples": 128,
+          "BLEU": 2.7297561079820287,
+          "chrF2": 21.796275475634115,
+          "TER": 112.19386439804073,
           "exact_match_rate": 0.0,
+          "empty_output_rate": 0.0,
+          "character_length_ratio": 1.0032571521632918,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "trv": {
           "samples": 128,
+          "BLEU": 8.33659966162617,
+          "chrF2": 32.0446575794997,
+          "TER": 74.22053231939162,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.8481531004512787,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "tsu": {
           "samples": 128,
+          "BLEU": 10.036666281229088,
+          "chrF2": 35.833768781078454,
+          "TER": 81.94852941176471,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9632515066882258,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "xnb": {
           "samples": 128,
+          "BLEU": 15.073988584777457,
+          "chrF2": 48.23658323417333,
+          "TER": 63.860369609856264,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9253807476499307,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         },
         "xsy": {
           "samples": 128,
+          "BLEU": 16.437505767524137,
+          "chrF2": 43.27827590109571,
+          "TER": 68.47792528558196,
           "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9321233674658164,
           "signatures": {
             "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
             "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
             "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          },
+          "source_copy_rate": 0.0
         }
       }
     },
+    "step": 300000,
     "selection": {
       "metric": "chrF2",
+      "value": 35.82202484658809,
       "improved": true,
+      "best_value_before_eval": 35.57269991348219
     }
   }
 }

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f6ea7229090f47f38ea260a58c086702682d10da590318fd2f83df93926beb84
-size 2482710880

 version https://git-lfs.github.com/spec/v1
+oid sha256:0f602a99072474db5b481d9abad0d9ae73985e7b2d26681a65cd292509cea53f
+size 2482141536

mt_standardization.py CHANGED Viewed

@@ -170,12 +170,16 @@ def include_optional_segments(run: _StandardizationRun) -> None:
     total = 0
     while True:
         count = 0
-        def replacement(match: re.Match[str]) -> str:
             nonlocal count
             start, end = match.span()
-            previous = working[start - 1] if start else ""
-            following = working[end] if end < len(working) else ""
             content = match.group(1)
             previous_is_boundary = (
                 is_orthographic_character(previous)
@@ -194,7 +198,7 @@ def include_optional_segments(run: _StandardizationRun) -> None:
             count += 1
             return content
-        updated = OPTIONAL_CONTENT_RE.sub(replacement, working)
         if not count:
             break
         total += count

     total = 0
     while True:
         count = 0
+        current = working
+        def replacement(
+            match: re.Match[str],
+            current_text: str = current,
+        ) -> str:
             nonlocal count
             start, end = match.span()
+            previous = current_text[start - 1] if start else ""
+            following = current_text[end] if end < len(current_text) else ""
             content = match.group(1)
             previous_is_boundary = (
                 is_orthographic_character(previous)
             count += 1
             return content
+        updated = OPTIONAL_CONTENT_RE.sub(replacement, current)
         if not count:
             break
         total += count

mt_standardization_profile.json CHANGED Viewed

@@ -1,7 +1,7 @@
 {
   "schema_version": 1,
   "profile_id": "formosan-mt-standard-v3",
-  "implementation_sha256": "c78f092f826aed70468bb3d4276eb27bab482dc5dc36dedf9ed5726596e0dd31",
   "unicode_normalization": "NFC",
   "policy": {
     "unwrap_letter_infixes": true,

 {
   "schema_version": 1,
   "profile_id": "formosan-mt-standard-v3",
+  "implementation_sha256": "2062106642f27053616728f6f962f7d3ffea022bf737dbe98a8d4af1e6de3831",
   "unicode_normalization": "NFC",
   "policy": {
     "unwrap_letter_infixes": true,

publication.json CHANGED Viewed

@@ -2,29 +2,29 @@
   "schema_version": 1,
   "repo_id": "FormosanBank/nllb200-zh-formosan-spm8k",
   "direction": "zh2f",
-  "run_stamp": "20260809-210523",
   "checkpoint": "best",
-  "checkpoint_step": 270000,
   "corpus": {
     "name": "private_no_bible",
     "target_lang": "chinese",
-    "rows": 788326,
     "splits": {
-      "test": 64356,
-      "train": 702520,
-      "validate": 21450
     },
-    "sha256": "238b96ea87716a749391d7157ce99f1611c1d4a489c4c108c315553505ca1f29"
   },
-  "profile_sha256": "34f45832bdedc1b8322b26936dd88667dfecb360d941a7f54f60ae31a1fb4e10",
   "base_model": {
     "name": "facebook/nllb-200-distilled-600M",
     "revision": "f8d333a098d19b4fd9a8b18f94170487ad3f821d"
   },
   "mt_standardization": {
     "id": "formosan-mt-standard-v3",
-    "sha256": "4bbded87eb4833b2d1cd5a88a1f2b059560416e0e69ecfa55433a77f418c903e",
     "namespace": "formosan-mt"
   },
-  "run_contract_sha256": "e2006ab32a0dc5494687f7247e4eca1c15ede5b1565fa5d853d6aea8b3f901b3"
 }

   "schema_version": 1,
   "repo_id": "FormosanBank/nllb200-zh-formosan-spm8k",
   "direction": "zh2f",
+  "run_stamp": "20260816-193900",
   "checkpoint": "best",
+  "checkpoint_step": 300000,
   "corpus": {
     "name": "private_no_bible",
     "target_lang": "chinese",
+    "rows": 735848,
     "splits": {
+      "test": 73591,
+      "train": 625460,
+      "validate": 36797
     },
+    "sha256": "0024e7fcd35958455a2a060b20999e94b5e8a4364e7f54ba97ef27810746d932"
   },
+  "profile_sha256": "4253b1b136fedebc589da942b80df71945212d1f4b92a8e98ec6fb160712f14b",
   "base_model": {
     "name": "facebook/nllb-200-distilled-600M",
     "revision": "f8d333a098d19b4fd9a8b18f94170487ad3f821d"
   },
   "mt_standardization": {
     "id": "formosan-mt-standard-v3",
+    "sha256": "3c93c1abecfc68c72e3bb1d547fbe0bdebb14bb74e7a5d3c5c519bd18d34bf6f",
     "namespace": "formosan-mt"
   },
+  "run_contract_sha256": "cc251cf00ec88f29e0d8407267f5f6d00ec19fe2fc5adfd7ab8f61c20c4ec897"
 }

sentencepiece.bpe.model CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d8ffca4dcbcf1655c9d4c9c4f3cd0ed2eaecf37fd689618fb9702a899021bc03
-size 4948342

 version https://git-lfs.github.com/spec/v1
+oid sha256:c622fc2eeba77b2253540f877e5c304f93e3f2a607d88bb777a0dccc8a5b36f2
+size 4949444

special_tokens_map.json CHANGED Viewed

@@ -259,222 +259,15 @@
     "<dialect_zeaol>",
     "<dialect_zhiben>",
     "<dialect_zhuoqun>",
-    "<dom_classroom_context>",
     "<dom_culture>",
     "<dom_dictionary>",
-    "<dom_essays>",
-    "<dom_formosan_100_paiwan_texts>",
-    "<dom_formosan_academia_sinica_oral_legends>",
-    "<dom_formosan_adlay_kun_long_liu_syntactic_interactio>",
-    "<dom_formosan_adlay_liu_lfg_relativisation_jianshi_sq>",
-    "<dom_formosan_amis_adversative_constructions>",
-    "<dom_formosan_amis_huang_english_phonology_grammar>",
-    "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
-    "<dom_formosan_amis_kuo_sung_comparative_constructions>",
-    "<dom_formosan_amis_myths_and_customs>",
-    "<dom_formosan_amis_pa_verbs>",
-    "<dom_formosan_amis_serial_verb_constructions>",
-    "<dom_formosan_amis_tung_chiou>",
-    "<dom_formosan_amy_lee_body_part_nomenclature_seediq>",
-    "<dom_formosan_anna_hsiou_chuan_chang_reference_gramma>",
-    "<dom_formosan_anton_quack_puyuma_pulingaw>",
-    "<dom_formosan_apay_tang_anger_happiness_truku_seediq>",
-    "<dom_formosan_asai_sedik_language>",
-    "<dom_formosan_atayal_you_nao_savak_raxayal>",
-    "<dom_formosan_bien_daniel_chiang_house_social_hierarc>",
-    "<dom_formosan_boyizhenu_narrative_oral_literature_tap>",
-    "<dom_formosan_bunun_debusser_dissertation>",
-    "<dom_formosan_bunun_topic_focus>",
-    "<dom_formosan_c_douglas_chretien_oceanic_material_in>",
-    "<dom_formosan_catherine_tseng_seediq_atayal_foods_the>",
-    "<dom_formosan_chang_kwo_tan_syncretic_objects_materia>",
-    "<dom_formosan_chaolin_li_motion_verb_grammaticalizati>",
-    "<dom_formosan_chen_fukuda_relabeling_ergative>",
-    "<dom_formosan_chen_fukuda_three_ways_steal>",
-    "<dom_formosan_cheng_sung_modality_kanakanavu>",
-    "<dom_formosan_chi_lu_chen_and_micheal_d_coe_an_invest>",
-    "<dom_formosan_chuming_wu_linking_constructions_mayrin>",
-    "<dom_formosan_chun_mei_chen_comparative_phonology_pai>",
-    "<dom_formosan_chunming_wu_two_types_of_noun_incorpora>",
-    "<dom_formosan_cip_atayal_grammar_overview>",
-    "<dom_formosan_claire_mcgill_a_brief_tayal_vocabulary>",
-    "<dom_formosan_david_blundell_ed_austronesian_taiwan>",
-    "<dom_formosan_dean_johnson_year_clouds_smangus>",
-    "<dom_formosan_der_hwa_victoria_rau_grammar_atayal>",
-    "<dom_formosan_der_hwa_victoria_rau_lexical_similarity>",
-    "<dom_formosan_dong_manv_yami_ode_to_taro>",
-    "<dom_formosan_dong_yi_lin_universal_quantifier_kavala>",
-    "<dom_formosan_dong_yijia_tjuwabar_paiwan_ritual_langu>",
-    "<dom_formosan_dorinda_tsai_hsiu_liu_neutral_imperfect>",
-    "<dom_formosan_ed_torjesen_amis_taiwan_word_list>",
-    "<dom_formosan_egerod_origin_headhunting_atayal_text_v>",
-    "<dom_formosan_egerod_soren_word_order_word_classes_at>",
-    "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
-    "<dom_formosan_elizabeth_zeitoun_lillian_huang_anna_ch>",
-    "<dom_formosan_elizabeth_zeitoun_lillian_huang_marie_y>",
-    "<dom_formosan_elizabeth_zeitoun_pronominal_system_man>",
-    "<dom_formosan_elizabeth_zeitoun_squib_dynamic_vs_stat>",
-    "<dom_formosan_epark>",
-    "<dom_formosan_ferrell_taiwan_aboriginal_groups_proble>",
-    "<dom_formosan_george_leroy_shelley_vudai_rukai>",
-    "<dom_formosan_gitbook_translations>",
-    "<dom_formosan_glosbe>",
-    "<dom_formosan_gujing_lin_tsou_serial_verb_constructio>",
-    "<dom_formosan_henry_chang_nominal_aspect_tsou>",
-    "<dom_formosan_henry_chang_tsou_causative_applicative>",
-    "<dom_formosan_henry_y_chang_formosan_speech_act_mood>",
-    "<dom_formosan_ho_dan_phonological_system_butonglu_pai>",
-    "<dom_formosan_holmer_seediq>",
-    "<dom_formosan_hsiu_chuan_liao_transitivity_ergativity>",
-    "<dom_formosan_huang_dongqiu_atayal_bunun_amis_languag>",
-    "<dom_formosan_huang_grammaticalization_squliq_atayal>",
-    "<dom_formosan_huang_hui_chuan_glide_formation_takitud>",
-    "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
-    "<dom_formosan_hui_chuan_j_huang_syllable_types_in_bun>",
-    "<dom_formosan_hui_shan_lin_squliq_atayal_reduplicatio>",
-    "<dom_formosan_huteson_rukai_survey>",
-    "<dom_formosan_ian_maddieson_and_richard_wright_the_vo>",
-    "<dom_formosan_ilrdf_42_language_practice_word_lists>",
-    "<dom_formosan_ilrdf_tousvusvutu_kita>",
-    "<dom_formosan_indigenous_education_resource_center_ke>",
-    "<dom_formosan_indigenous_language_literary_awards>",
-    "<dom_formosan_isabelle_bril_roots_and_stems_lexical_a>",
-    "<dom_formosan_jian_iwan_guo_qingliu_life_history>",
-    "<dom_formosan_jian_shilang_introduction_thao_grammar>",
-    "<dom_formosan_john_tu_er_wei_notes_on_the_religious_b>",
-    "<dom_formosan_john_u_wolff_the_proto_austronesian_pho>",
-    "<dom_formosan_jonathan_kuo_amis_manner_result_verbs>",
-    "<dom_formosan_josiane_cauquelin_aborigines_taiwan_puy>",
-    "<dom_formosan_joy_wu_clausal_modifiers_in_amis>",
-    "<dom_formosan_joy_wu_verb_classification_case_marking>",
-    "<dom_formosan_kanakanavu_texts>",
-    "<dom_formosan_kavalan_zhang_reference_grammar>",
-    "<dom_formosan_kumu_tapas_tribal_memory_oral_history_w>",
-    "<dom_formosan_kuo_j_c_argument_alternation_and_argume>",
-    "<dom_formosan_lei_shih_family_system_paiwan_at_su_pai>",
-    "<dom_formosan_li_chen_yeh_developing_a_hla_alua_learn>",
-    "<dom_formosan_li_may_sung_budai_rukai_exclamatives>",
-    "<dom_formosan_li_may_sung_clausal_nominalization_buda>",
-    "<dom_formosan_lillian_huang_atayal_participants_cross>",
-    "<dom_formosan_lillian_huang_ergativity_atayal>",
-    "<dom_formosan_liu_manyi_kaskun_ata_matas_i_ki_quaz>",
-    "<dom_formosan_malcom_d_ross_the_history_and_transitiv>",
-    "<dom_formosan_malcom_ross_proto_austronesian_verbal_m>",
-    "<dom_formosan_man_ni_chu_the_measurement_of_final_i_v>",
-    "<dom_formosan_matsuzawa_kazuko_the_social_and_ritual>",
-    "<dom_formosan_maya_yeh_blaq_uv_construction_atayal>",
-    "<dom_formosan_mayumi_oiwa_adverbial_verb_construction>",
-    "<dom_formosan_mayumi_oiwa_bungard_morphology_and_synt>",
-    "<dom_formosan_mayumi_oiwa_possessor_raising_truku_see>",
-    "<dom_formosan_mei_chin_huang_verb_classification_in_m>",
-    "<dom_formosan_melissa_shih_hui_lin_sakizaya_or_amis_a>",
-    "<dom_formosan_nanang_tadaw_naci_mowna_pgagu>",
-    "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
-    "<dom_formosan_naomi_tsukida_ditransitive_causative_se>",
-    "<dom_formosan_nowbucyang_truku_thesis>",
-    "<dom_formosan_ochiai_izumi_2016_comparative_linguisti>",
-    "<dom_formosan_ochiai_izumi_bu_hwan_vocabulary_recorde>",
-    "<dom_formosan_ochiai_izumi_numerals_paran_seediq_rela>",
-    "<dom_formosan_ochiai_izumi_on_a_seediq_glossary_recor>",
-    "<dom_formosan_old_texts>",
-    "<dom_formosan_paiwan_ho_five_dialects>",
-    "<dom_formosan_paiwanstories>",
-    "<dom_formosan_pang_hsin_ting_reconstruction_proto_puy>",
-    "<dom_formosan_patricia_stanley_morphophonemics_of_ver>",
-    "<dom_formosan_paul_jen_kuei_li_comparative_bunun_dial>",
-    "<dom_formosan_paul_jen_kuei_li_conjunction_thao>",
-    "<dom_formosan_paul_jen_kuei_li_internal_relationships>",
-    "<dom_formosan_paul_jen_kuei_li_origins_of_the_east_fo>",
-    "<dom_formosan_paul_jen_kuei_li_review_of_taiwan_abori>",
-    "<dom_formosan_paul_jen_kuei_li_rukai_structure>",
-    "<dom_formosan_paul_jen_kuei_li_some_plant_names_in_fo>",
-    "<dom_formosan_paul_jen_kuei_li_the_preglottalised_sto>",
-    "<dom_formosan_paul_li_atayalic_final_voiced_stops>",
-    "<dom_formosan_paul_li_case_marking_four_formosan_lang>",
-    "<dom_formosan_paul_li_position_atayal_austronesian>",
-    "<dom_formosan_pingdong_county_government_kai_na_sepuc>",
-    "<dom_formosan_puyuma_chen_raising_to_object>",
-    "<dom_formosan_puyuma_katipol_kopfjagdriten>",
-    "<dom_formosan_puyuma_teng_pronominal_systems>",
-    "<dom_formosan_puyuma_teng_reference_grammar>",
-    "<dom_formosan_raleigh_ferrell_construction_markers_fo>",
-    "<dom_formosan_raleigh_ferrell_paiwan_phonology_and_pr>",
-    "<dom_formosan_raleigh_j_ferrell_intent_and_volition_i>",
-    "<dom_formosan_rik_bunun>",
-    "<dom_formosan_rik_de_busser_the_influence_of_christia>",
-    "<dom_formosan_robert_blust_a_note_on_covert_structure>",
-    "<dom_formosan_robert_blust_austronesian_homeland_ling>",
-    "<dom_formosan_robert_blust_some_remarks_linguistic_po>",
-    "<dom_formosan_robert_blust_thao_triplication>",
-    "<dom_formosan_robert_blust_the_austronesian_languages>",
-    "<dom_formosan_rukai_mantauran_stories>",
-    "<dom_formosan_rukai_texts>",
-    "<dom_formosan_rukai_zeitoun_saying>",
-    "<dom_formosan_sakizaya_affixes>",
-    "<dom_formosan_schetelig_mittheilungen_uber_die_sprach>",
-    "<dom_formosan_seals>",
-    "<dom_formosan_seediq_zhang_reference_grammar>",
-    "<dom_formosan_shigeru_tsuchida_and_isidore_dyen_proto>",
-    "<dom_formosan_shigeru_tsuchida_kanakanavu_texts>",
-    "<dom_formosan_shigeru_tsuchida_preliminary_reports_on>",
-    "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
-    "<dom_formosan_shih_chi_stella_yeh_stress_and_quantity>",
-    "<dom_formosan_shih_rukai_adverbial>",
-    "<dom_formosan_sihwei_chen_and_haowen_jiang_ways_of_ta>",
-    "<dom_formosan_sihwei_chen_lexical_aspect_atayal>",
-    "<dom_formosan_soren_egerod_statement_atayal_phonology>",
-    "<dom_formosan_soren_egerod_verb_inflection_atayal>",
-    "<dom_formosan_stacy_teng_malcom_ross_is_puyuma_primar>",
-    "<dom_formosan_stacy_wan_tin_huang_functions_yami_verb>",
-    "<dom_formosan_sung_kuo_descriptive_comparative_constr>",
-    "<dom_formosan_taoshan_elementary_school_atelier_hui_k>",
-    "<dom_formosan_teresa_m_chen_verbal_construction_and_v>",
-    "<dom_formosan_ting_chi_wei_parallelism_in_amis_sluici>",
-    "<dom_formosan_tingchun_chen_successive_cyclic_case_as>",
-    "<dom_formosan_truku_lowking_demonstratives>",
-    "<dom_formosan_tsai_hsui_liu_complementation_three_lan>",
-    "<dom_formosan_tsai_wei_tien_dylan_conjunctive_reducti>",
-    "<dom_formosan_tsou_descriptive_study>",
-    "<dom_formosan_tsukida_naomi_verb_classification_amis>",
-    "<dom_formosan_tu_wen_chiu_a_synchronic_classification>",
-    "<dom_formosan_tung_tsuchida_ting_li_pan_saaroa_texts>",
-    "<dom_formosan_victoria_rau_yi_hsin_wu_and_meng_chien>",
-    "<dom_formosan_wei_huilin_liu_social_structure_yami>",
-    "<dom_formosan_wei_matri_clan_lineage_system_ami>",
-    "<dom_formosan_wei_tien_tsai_subjecthood_temporal_adju>",
-    "<dom_formosan_wiedfelt_formosan_aborigines_atayal>",
-    "<dom_formosan_wilang_yutas_videos>",
-    "<dom_formosan_wu_chunming_adverbials_in_paiwan>",
-    "<dom_formosan_xie_fuhui_introduction_kavalan_grammar>",
-    "<dom_formosan_yedda_palemeq_blog>",
-    "<dom_formosan_yeddas_blog>",
-    "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
-    "<dom_formosan_yi_ming_chou_object_control_saisiyat>",
-    "<dom_formosan_yi_ting_chen_a_minimalist_approach_to_a>",
-    "<dom_formosan_yi_ting_chen_amis_left_periphery>",
-    "<dom_formosan_yi_ting_chen_language_use_code_mixing_a>",
-    "<dom_formosan_yi_yang_cheng_kanakanavu_word_level_pro>",
-    "<dom_formosan_yi_yang_cheng_tense_aspect_agent_markin>",
-    "<dom_formosan_yoshiro_nihira_bunun_vocabulary>",
-    "<dom_formosan_yuehchien_chien_the_lexical_system_of_y>",
-    "<dom_formosan_yukihiro_yamada_ying_chu_liao_phonology>",
-    "<dom_formosan_zeng_shifen_reduplication_affixation_pa>",
-    "<dom_formosan_zhan_sujuan_taiwan_indigenous_peoples_h>",
-    "<dom_formosan_zhang_xiujuan_introduction_paiwan_gramm>",
-    "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
-    "<dom_formosan_zhao_shanhe_li_taiyuan_zhu_qingyi_abori>",
-    "<dom_formosan_zheng_acl_2024>",
-    "<dom_formosan_zheng_data>",
-    "<dom_formosan_zheng_yumei_zhu_qingyi_bo_hongming_abor>",
-    "<dom_learning_vocab>",
-    "<dom_nine_level>",
-    "<dom_ntu>",
-    "<dom_picture_book>",
-    "<dom_picture_story>",
-    "<dom_presidential_apology>",
-    "<dom_reading_writing>",
     "<dom_unknown>",
-    "<dom_youtube>",
     "<src_ami>",
     "<src_bnn>",
     "<src_ckv>",

     "<dialect_zeaol>",
     "<dialect_zhiben>",
     "<dialect_zhuoqun>",
+    "<dom_classroom>",
     "<dom_culture>",
     "<dom_dictionary>",
+    "<dom_education>",
+    "<dom_linguistic>",
+    "<dom_media>",
+    "<dom_narrative>",
+    "<dom_religious>",
     "<dom_unknown>",
     "<src_ami>",
     "<src_bnn>",
     "<src_ckv>",

tokenizer_config.json CHANGED Viewed

@@ -1656,7 +1656,7 @@
       "single_word": false,
       "special": true
     },
-    "261357": {
       "content": "ami_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1664,7 +1664,7 @@
       "single_word": false,
       "special": true
     },
-    "261358": {
       "content": "bnn_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1672,7 +1672,7 @@
       "single_word": false,
       "special": true
     },
-    "261359": {
       "content": "ckv_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1680,7 +1680,7 @@
       "single_word": false,
       "special": true
     },
-    "261360": {
       "content": "dru_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1688,7 +1688,7 @@
       "single_word": false,
       "special": true
     },
-    "261361": {
       "content": "pwn_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1696,7 +1696,7 @@
       "single_word": false,
       "special": true
     },
-    "261362": {
       "content": "pyu_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1704,7 +1704,7 @@
       "single_word": false,
       "special": true
     },
-    "261363": {
       "content": "ssf_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1712,7 +1712,7 @@
       "single_word": false,
       "special": true
     },
-    "261364": {
       "content": "sxr_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1720,7 +1720,7 @@
       "single_word": false,
       "special": true
     },
-    "261365": {
       "content": "szy_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1728,7 +1728,7 @@
       "single_word": false,
       "special": true
     },
-    "261366": {
       "content": "tao_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1736,7 +1736,7 @@
       "single_word": false,
       "special": true
     },
-    "261367": {
       "content": "tay_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1744,7 +1744,7 @@
       "single_word": false,
       "special": true
     },
-    "261368": {
       "content": "trv_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1752,7 +1752,7 @@
       "single_word": false,
       "special": true
     },
-    "261369": {
       "content": "tsu_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1760,7 +1760,7 @@
       "single_word": false,
       "special": true
     },
-    "261370": {
       "content": "xnb_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1768,7 +1768,7 @@
       "single_word": false,
       "special": true
     },
-    "261371": {
       "content": "xsy_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1776,7 +1776,7 @@
       "single_word": false,
       "special": true
     },
-    "261372": {
       "content": "<dialect_central>",
       "lstrip": false,
       "normalized": false,
@@ -1784,7 +1784,7 @@
       "single_word": false,
       "special": true
     },
-    "261373": {
       "content": "<dialect_coastal>",
       "lstrip": false,
       "normalized": false,
@@ -1792,7 +1792,7 @@
       "single_word": false,
       "special": true
     },
-    "261374": {
       "content": "<dialect_dawu>",
       "lstrip": false,
       "normalized": false,
@@ -1800,7 +1800,7 @@
       "single_word": false,
       "special": true
     },
-    "261375": {
       "content": "<dialect_default>",
       "lstrip": false,
       "normalized": false,
@@ -1808,7 +1808,7 @@
       "single_word": false,
       "special": true
     },
-    "261376": {
       "content": "<dialect_deluvalley>",
       "lstrip": false,
       "normalized": false,
@@ -1816,7 +1816,7 @@
       "single_word": false,
       "special": true
     },
-    "261377": {
       "content": "<dialect_dona>",
       "lstrip": false,
       "normalized": false,
@@ -1824,7 +1824,7 @@
       "single_word": false,
       "special": true
     },
-    "261378": {
       "content": "<dialect_duda>",
       "lstrip": false,
       "normalized": false,
@@ -1832,7 +1832,7 @@
       "single_word": false,
       "special": true
     },
-    "261379": {
       "content": "<dialect_eastern>",
       "lstrip": false,
       "normalized": false,
@@ -1840,7 +1840,7 @@
       "single_word": false,
       "special": true
     },
-    "261380": {
       "content": "<dialect_fourseasons>",
       "lstrip": false,
       "normalized": false,
@@ -1848,7 +1848,7 @@
       "single_word": false,
       "special": true
     },
-    "261381": {
       "content": "<dialect_hengchun>",
       "lstrip": false,
       "normalized": false,
@@ -1856,7 +1856,7 @@
       "single_word": false,
       "special": true
     },
-    "261382": {
       "content": "<dialect_jianhe>",
       "lstrip": false,
       "normalized": false,
@@ -1864,7 +1864,7 @@
       "single_word": false,
       "special": true
     },
-    "261383": {
       "content": "<dialect_junqun>",
       "lstrip": false,
       "normalized": false,
@@ -1872,7 +1872,7 @@
       "single_word": false,
       "special": true
     },
-    "261384": {
       "content": "<dialect_kanakanavu>",
       "lstrip": false,
       "normalized": false,
@@ -1880,7 +1880,7 @@
       "single_word": false,
       "special": true
     },
-    "261385": {
       "content": "<dialect_kaqun>",
       "lstrip": false,
       "normalized": false,
@@ -1888,7 +1888,7 @@
       "single_word": false,
       "special": true
     },
-    "261386": {
       "content": "<dialect_kavalan>",
       "lstrip": false,
       "normalized": false,
@@ -1896,7 +1896,7 @@
       "single_word": false,
       "special": true
     },
-    "261387": {
       "content": "<dialect_luanqun>",
       "lstrip": false,
       "normalized": false,
@@ -1904,7 +1904,7 @@
       "single_word": false,
       "special": true
     },
-    "261388": {
       "content": "<dialect_malan>",
       "lstrip": false,
       "normalized": false,
@@ -1912,7 +1912,7 @@
       "single_word": false,
       "special": true
     },
-    "261389": {
       "content": "<dialect_maolin>",
       "lstrip": false,
       "normalized": false,
@@ -1920,7 +1920,7 @@
       "single_word": false,
       "special": true
     },
-    "261390": {
       "content": "<dialect_nanwang>",
       "lstrip": false,
       "normalized": false,
@@ -1928,7 +1928,7 @@
       "single_word": false,
       "special": true
     },
-    "261391": {
       "content": "<dialect_northern>",
       "lstrip": false,
       "normalized": false,
@@ -1936,7 +1936,7 @@
       "single_word": false,
       "special": true
     },
-    "261392": {
       "content": "<dialect_saaroa>",
       "lstrip": false,
       "normalized": false,
@@ -1944,7 +1944,7 @@
       "single_word": false,
       "special": true
     },
-    "261393": {
       "content": "<dialect_saisiyat>",
       "lstrip": false,
       "normalized": false,
@@ -1952,7 +1952,7 @@
       "single_word": false,
       "special": true
     },
-    "261394": {
       "content": "<dialect_sakizaya>",
       "lstrip": false,
       "normalized": false,
@@ -1960,7 +1960,7 @@
       "single_word": false,
       "special": true
     },
-    "261395": {
       "content": "<dialect_sekolik>",
       "lstrip": false,
       "normalized": false,
@@ -1968,7 +1968,7 @@
       "single_word": false,
       "special": true
     },
-    "261396": {
       "content": "<dialect_southern>",
       "lstrip": false,
       "normalized": false,
@@ -1976,7 +1976,7 @@
       "single_word": false,
       "special": true
     },
-    "261397": {
       "content": "<dialect_tanqun>",
       "lstrip": false,
       "normalized": false,
@@ -1984,7 +1984,7 @@
       "single_word": false,
       "special": true
     },
-    "261398": {
       "content": "<dialect_tegudaya>",
       "lstrip": false,
       "normalized": false,
@@ -1992,7 +1992,7 @@
       "single_word": false,
       "special": true
     },
-    "261399": {
       "content": "<dialect_thao>",
       "lstrip": false,
       "normalized": false,
@@ -2000,7 +2000,7 @@
       "single_word": false,
       "special": true
     },
-    "261400": {
       "content": "<dialect_truku>",
       "lstrip": false,
       "normalized": false,
@@ -2008,7 +2008,7 @@
       "single_word": false,
       "special": true
     },
-    "261401": {
       "content": "<dialect_tsou>",
       "lstrip": false,
       "normalized": false,
@@ -2016,7 +2016,7 @@
       "single_word": false,
       "special": true
     },
-    "261402": {
       "content": "<dialect_unknown>",
       "lstrip": false,
       "normalized": false,
@@ -2024,7 +2024,7 @@
       "single_word": false,
       "special": true
     },
-    "261403": {
       "content": "<dialect_wanda>",
       "lstrip": false,
       "normalized": false,
@@ -2032,7 +2032,7 @@
       "single_word": false,
       "special": true
     },
-    "261404": {
       "content": "<dialect_wanshan>",
       "lstrip": false,
       "normalized": false,
@@ -2040,7 +2040,7 @@
       "single_word": false,
       "special": true
     },
-    "261405": {
       "content": "<dialect_wenshui>",
       "lstrip": false,
       "normalized": false,
@@ -2048,7 +2048,7 @@
       "single_word": false,
       "special": true
     },
-    "261406": {
       "content": "<dialect_wutai>",
       "lstrip": false,
       "normalized": false,
@@ -2056,7 +2056,7 @@
       "single_word": false,
       "special": true
     },
-    "261407": {
       "content": "<dialect_xiqun>",
       "lstrip": false,
       "normalized": false,
@@ -2064,7 +2064,7 @@
       "single_word": false,
       "special": true
     },
-    "261408": {
       "content": "<dialect_xiuguluan>",
       "lstrip": false,
       "normalized": false,
@@ -2072,7 +2072,7 @@
       "single_word": false,
       "special": true
     },
-    "261409": {
       "content": "<dialect_yami>",
       "lstrip": false,
       "normalized": false,
@@ -2080,7 +2080,7 @@
       "single_word": false,
       "special": true
     },
-    "261410": {
       "content": "<dialect_yilanzeaol>",
       "lstrip": false,
       "normalized": false,
@@ -2088,7 +2088,7 @@
       "single_word": false,
       "special": true
     },
-    "261411": {
       "content": "<dialect_zeaol>",
       "lstrip": false,
       "normalized": false,
@@ -2096,7 +2096,7 @@
       "single_word": false,
       "special": true
     },
-    "261412": {
       "content": "<dialect_zhiben>",
       "lstrip": false,
       "normalized": false,
@@ -2104,7 +2104,7 @@
       "single_word": false,
       "special": true
     },
-    "261413": {
       "content": "<dialect_zhuoqun>",
       "lstrip": false,
       "normalized": false,
@@ -2112,15 +2112,15 @@
       "single_word": false,
       "special": true
     },
-    "261414": {
-      "content": "<dom_classroom_context>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261415": {
       "content": "<dom_culture>",
       "lstrip": false,
       "normalized": false,
@@ -2128,7 +2128,7 @@
       "single_word": false,
       "special": true
     },
-    "261416": {
       "content": "<dom_dictionary>",
       "lstrip": false,
       "normalized": false,
@@ -2136,1975 +2136,319 @@
       "single_word": false,
       "special": true
     },
-    "261417": {
-      "content": "<dom_essays>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261418": {
-      "content": "<dom_formosan_100_paiwan_texts>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261419": {
-      "content": "<dom_formosan_academia_sinica_oral_legends>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261420": {
-      "content": "<dom_formosan_adlay_kun_long_liu_syntactic_interactio>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261421": {
-      "content": "<dom_formosan_adlay_liu_lfg_relativisation_jianshi_sq>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261422": {
-      "content": "<dom_formosan_amis_adversative_constructions>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261423": {
-      "content": "<dom_formosan_amis_huang_english_phonology_grammar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261424": {
-      "content": "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261425": {
-      "content": "<dom_formosan_amis_kuo_sung_comparative_constructions>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261426": {
-      "content": "<dom_formosan_amis_myths_and_customs>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261427": {
-      "content": "<dom_formosan_amis_pa_verbs>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261428": {
-      "content": "<dom_formosan_amis_serial_verb_constructions>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261429": {
-      "content": "<dom_formosan_amis_tung_chiou>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261430": {
-      "content": "<dom_formosan_amy_lee_body_part_nomenclature_seediq>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261431": {
-      "content": "<dom_formosan_anna_hsiou_chuan_chang_reference_gramma>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261432": {
-      "content": "<dom_formosan_anton_quack_puyuma_pulingaw>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261433": {
-      "content": "<dom_formosan_apay_tang_anger_happiness_truku_seediq>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261434": {
-      "content": "<dom_formosan_asai_sedik_language>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261435": {
-      "content": "<dom_formosan_atayal_you_nao_savak_raxayal>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261436": {
-      "content": "<dom_formosan_bien_daniel_chiang_house_social_hierarc>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261437": {
-      "content": "<dom_formosan_boyizhenu_narrative_oral_literature_tap>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261438": {
-      "content": "<dom_formosan_bunun_debusser_dissertation>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261439": {
-      "content": "<dom_formosan_bunun_topic_focus>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261440": {
-      "content": "<dom_formosan_c_douglas_chretien_oceanic_material_in>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261441": {
-      "content": "<dom_formosan_catherine_tseng_seediq_atayal_foods_the>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261442": {
-      "content": "<dom_formosan_chang_kwo_tan_syncretic_objects_materia>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261443": {
-      "content": "<dom_formosan_chaolin_li_motion_verb_grammaticalizati>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261444": {
-      "content": "<dom_formosan_chen_fukuda_relabeling_ergative>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261445": {
-      "content": "<dom_formosan_chen_fukuda_three_ways_steal>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261446": {
-      "content": "<dom_formosan_cheng_sung_modality_kanakanavu>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261447": {
-      "content": "<dom_formosan_chi_lu_chen_and_micheal_d_coe_an_invest>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261448": {
-      "content": "<dom_formosan_chuming_wu_linking_constructions_mayrin>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261449": {
-      "content": "<dom_formosan_chun_mei_chen_comparative_phonology_pai>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261450": {
-      "content": "<dom_formosan_chunming_wu_two_types_of_noun_incorpora>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261451": {
-      "content": "<dom_formosan_cip_atayal_grammar_overview>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261452": {
-      "content": "<dom_formosan_claire_mcgill_a_brief_tayal_vocabulary>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261453": {
-      "content": "<dom_formosan_david_blundell_ed_austronesian_taiwan>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261454": {
-      "content": "<dom_formosan_dean_johnson_year_clouds_smangus>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261455": {
-      "content": "<dom_formosan_der_hwa_victoria_rau_grammar_atayal>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261456": {
-      "content": "<dom_formosan_der_hwa_victoria_rau_lexical_similarity>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261457": {
-      "content": "<dom_formosan_dong_manv_yami_ode_to_taro>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261458": {
-      "content": "<dom_formosan_dong_yi_lin_universal_quantifier_kavala>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261459": {
-      "content": "<dom_formosan_dong_yijia_tjuwabar_paiwan_ritual_langu>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261460": {
-      "content": "<dom_formosan_dorinda_tsai_hsiu_liu_neutral_imperfect>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261461": {
-      "content": "<dom_formosan_ed_torjesen_amis_taiwan_word_list>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261462": {
-      "content": "<dom_formosan_egerod_origin_headhunting_atayal_text_v>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261463": {
-      "content": "<dom_formosan_egerod_soren_word_order_word_classes_at>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261464": {
-      "content": "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261465": {
-      "content": "<dom_formosan_elizabeth_zeitoun_lillian_huang_anna_ch>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261466": {
-      "content": "<dom_formosan_elizabeth_zeitoun_lillian_huang_marie_y>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261467": {
-      "content": "<dom_formosan_elizabeth_zeitoun_pronominal_system_man>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261468": {
-      "content": "<dom_formosan_elizabeth_zeitoun_squib_dynamic_vs_stat>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "261469": {
-      "content": "<dom_formosan_epark>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261470": {
-      "content": "<dom_formosan_ferrell_taiwan_aboriginal_groups_proble>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261471": {
-      "content": "<dom_formosan_george_leroy_shelley_vudai_rukai>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261472": {
-      "content": "<dom_formosan_gitbook_translations>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261473": {
-      "content": "<dom_formosan_glosbe>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261474": {
-      "content": "<dom_formosan_gujing_lin_tsou_serial_verb_constructio>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261475": {
-      "content": "<dom_formosan_henry_chang_nominal_aspect_tsou>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261476": {
-      "content": "<dom_formosan_henry_chang_tsou_causative_applicative>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261477": {
-      "content": "<dom_formosan_henry_y_chang_formosan_speech_act_mood>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261478": {
-      "content": "<dom_formosan_ho_dan_phonological_system_butonglu_pai>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261479": {
-      "content": "<dom_formosan_holmer_seediq>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261480": {
-      "content": "<dom_formosan_hsiu_chuan_liao_transitivity_ergativity>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261481": {
-      "content": "<dom_formosan_huang_dongqiu_atayal_bunun_amis_languag>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261482": {
-      "content": "<dom_formosan_huang_grammaticalization_squliq_atayal>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261483": {
-      "content": "<dom_formosan_huang_hui_chuan_glide_formation_takitud>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261484": {
-      "content": "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261485": {
-      "content": "<dom_formosan_hui_chuan_j_huang_syllable_types_in_bun>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261486": {
-      "content": "<dom_formosan_hui_shan_lin_squliq_atayal_reduplicatio>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261487": {
-      "content": "<dom_formosan_huteson_rukai_survey>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261488": {
-      "content": "<dom_formosan_ian_maddieson_and_richard_wright_the_vo>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261489": {
-      "content": "<dom_formosan_ilrdf_42_language_practice_word_lists>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261490": {
-      "content": "<dom_formosan_ilrdf_tousvusvutu_kita>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261491": {
-      "content": "<dom_formosan_indigenous_education_resource_center_ke>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261492": {
-      "content": "<dom_formosan_indigenous_language_literary_awards>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261493": {
-      "content": "<dom_formosan_isabelle_bril_roots_and_stems_lexical_a>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261494": {
-      "content": "<dom_formosan_jian_iwan_guo_qingliu_life_history>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261495": {
-      "content": "<dom_formosan_jian_shilang_introduction_thao_grammar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261496": {
-      "content": "<dom_formosan_john_tu_er_wei_notes_on_the_religious_b>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261497": {
-      "content": "<dom_formosan_john_u_wolff_the_proto_austronesian_pho>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261498": {
-      "content": "<dom_formosan_jonathan_kuo_amis_manner_result_verbs>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261499": {
-      "content": "<dom_formosan_josiane_cauquelin_aborigines_taiwan_puy>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261500": {
-      "content": "<dom_formosan_joy_wu_clausal_modifiers_in_amis>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261501": {
-      "content": "<dom_formosan_joy_wu_verb_classification_case_marking>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261502": {
-      "content": "<dom_formosan_kanakanavu_texts>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261503": {
-      "content": "<dom_formosan_kavalan_zhang_reference_grammar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261504": {
-      "content": "<dom_formosan_kumu_tapas_tribal_memory_oral_history_w>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261505": {
-      "content": "<dom_formosan_kuo_j_c_argument_alternation_and_argume>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261506": {
-      "content": "<dom_formosan_lei_shih_family_system_paiwan_at_su_pai>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261507": {
-      "content": "<dom_formosan_li_chen_yeh_developing_a_hla_alua_learn>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261508": {
-      "content": "<dom_formosan_li_may_sung_budai_rukai_exclamatives>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261509": {
-      "content": "<dom_formosan_li_may_sung_clausal_nominalization_buda>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261510": {
-      "content": "<dom_formosan_lillian_huang_atayal_participants_cross>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261511": {
-      "content": "<dom_formosan_lillian_huang_ergativity_atayal>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261512": {
-      "content": "<dom_formosan_liu_manyi_kaskun_ata_matas_i_ki_quaz>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261513": {
-      "content": "<dom_formosan_malcom_d_ross_the_history_and_transitiv>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261514": {
-      "content": "<dom_formosan_malcom_ross_proto_austronesian_verbal_m>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261515": {
-      "content": "<dom_formosan_man_ni_chu_the_measurement_of_final_i_v>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261516": {
-      "content": "<dom_formosan_matsuzawa_kazuko_the_social_and_ritual>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261517": {
-      "content": "<dom_formosan_maya_yeh_blaq_uv_construction_atayal>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261518": {
-      "content": "<dom_formosan_mayumi_oiwa_adverbial_verb_construction>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261519": {
-      "content": "<dom_formosan_mayumi_oiwa_bungard_morphology_and_synt>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261520": {
-      "content": "<dom_formosan_mayumi_oiwa_possessor_raising_truku_see>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261521": {
-      "content": "<dom_formosan_mei_chin_huang_verb_classification_in_m>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261522": {
-      "content": "<dom_formosan_melissa_shih_hui_lin_sakizaya_or_amis_a>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261523": {
-      "content": "<dom_formosan_nanang_tadaw_naci_mowna_pgagu>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261524": {
-      "content": "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261525": {
-      "content": "<dom_formosan_naomi_tsukida_ditransitive_causative_se>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261526": {
-      "content": "<dom_formosan_nowbucyang_truku_thesis>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261527": {
-      "content": "<dom_formosan_ochiai_izumi_2016_comparative_linguisti>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261528": {
-      "content": "<dom_formosan_ochiai_izumi_bu_hwan_vocabulary_recorde>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261529": {
-      "content": "<dom_formosan_ochiai_izumi_numerals_paran_seediq_rela>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261530": {
-      "content": "<dom_formosan_ochiai_izumi_on_a_seediq_glossary_recor>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261531": {
-      "content": "<dom_formosan_old_texts>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261532": {
-      "content": "<dom_formosan_paiwan_ho_five_dialects>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261533": {
-      "content": "<dom_formosan_paiwanstories>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261534": {
-      "content": "<dom_formosan_pang_hsin_ting_reconstruction_proto_puy>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261535": {
-      "content": "<dom_formosan_patricia_stanley_morphophonemics_of_ver>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261536": {
-      "content": "<dom_formosan_paul_jen_kuei_li_comparative_bunun_dial>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261537": {
-      "content": "<dom_formosan_paul_jen_kuei_li_conjunction_thao>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261538": {
-      "content": "<dom_formosan_paul_jen_kuei_li_internal_relationships>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261539": {
-      "content": "<dom_formosan_paul_jen_kuei_li_origins_of_the_east_fo>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261540": {
-      "content": "<dom_formosan_paul_jen_kuei_li_review_of_taiwan_abori>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261541": {
-      "content": "<dom_formosan_paul_jen_kuei_li_rukai_structure>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261542": {
-      "content": "<dom_formosan_paul_jen_kuei_li_some_plant_names_in_fo>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261543": {
-      "content": "<dom_formosan_paul_jen_kuei_li_the_preglottalised_sto>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261544": {
-      "content": "<dom_formosan_paul_li_atayalic_final_voiced_stops>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261545": {
-      "content": "<dom_formosan_paul_li_case_marking_four_formosan_lang>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261546": {
-      "content": "<dom_formosan_paul_li_position_atayal_austronesian>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261547": {
-      "content": "<dom_formosan_pingdong_county_government_kai_na_sepuc>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261548": {
-      "content": "<dom_formosan_puyuma_chen_raising_to_object>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261549": {
-      "content": "<dom_formosan_puyuma_katipol_kopfjagdriten>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261550": {
-      "content": "<dom_formosan_puyuma_teng_pronominal_systems>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261551": {
-      "content": "<dom_formosan_puyuma_teng_reference_grammar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261552": {
-      "content": "<dom_formosan_raleigh_ferrell_construction_markers_fo>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261553": {
-      "content": "<dom_formosan_raleigh_ferrell_paiwan_phonology_and_pr>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261554": {
-      "content": "<dom_formosan_raleigh_j_ferrell_intent_and_volition_i>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261555": {
-      "content": "<dom_formosan_rik_bunun>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261556": {
-      "content": "<dom_formosan_rik_de_busser_the_influence_of_christia>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261557": {
-      "content": "<dom_formosan_robert_blust_a_note_on_covert_structure>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261558": {
-      "content": "<dom_formosan_robert_blust_austronesian_homeland_ling>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261559": {
-      "content": "<dom_formosan_robert_blust_some_remarks_linguistic_po>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261560": {
-      "content": "<dom_formosan_robert_blust_thao_triplication>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261561": {
-      "content": "<dom_formosan_robert_blust_the_austronesian_languages>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261562": {
-      "content": "<dom_formosan_rukai_mantauran_stories>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261563": {
-      "content": "<dom_formosan_rukai_texts>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261564": {
-      "content": "<dom_formosan_rukai_zeitoun_saying>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261565": {
-      "content": "<dom_formosan_sakizaya_affixes>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261566": {
-      "content": "<dom_formosan_schetelig_mittheilungen_uber_die_sprach>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261567": {
-      "content": "<dom_formosan_seals>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261568": {
-      "content": "<dom_formosan_seediq_zhang_reference_grammar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261569": {
-      "content": "<dom_formosan_shigeru_tsuchida_and_isidore_dyen_proto>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261570": {
-      "content": "<dom_formosan_shigeru_tsuchida_kanakanavu_texts>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261571": {
-      "content": "<dom_formosan_shigeru_tsuchida_preliminary_reports_on>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261572": {
-      "content": "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261573": {
-      "content": "<dom_formosan_shih_chi_stella_yeh_stress_and_quantity>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261574": {
-      "content": "<dom_formosan_shih_rukai_adverbial>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261575": {
-      "content": "<dom_formosan_sihwei_chen_and_haowen_jiang_ways_of_ta>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261576": {
-      "content": "<dom_formosan_sihwei_chen_lexical_aspect_atayal>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261577": {
-      "content": "<dom_formosan_soren_egerod_statement_atayal_phonology>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261578": {
-      "content": "<dom_formosan_soren_egerod_verb_inflection_atayal>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261579": {
-      "content": "<dom_formosan_stacy_teng_malcom_ross_is_puyuma_primar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261580": {
-      "content": "<dom_formosan_stacy_wan_tin_huang_functions_yami_verb>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261581": {
-      "content": "<dom_formosan_sung_kuo_descriptive_comparative_constr>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261582": {
-      "content": "<dom_formosan_taoshan_elementary_school_atelier_hui_k>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261583": {
-      "content": "<dom_formosan_teresa_m_chen_verbal_construction_and_v>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261584": {
-      "content": "<dom_formosan_ting_chi_wei_parallelism_in_amis_sluici>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261585": {
-      "content": "<dom_formosan_tingchun_chen_successive_cyclic_case_as>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261586": {
-      "content": "<dom_formosan_truku_lowking_demonstratives>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261587": {
-      "content": "<dom_formosan_tsai_hsui_liu_complementation_three_lan>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261588": {
-      "content": "<dom_formosan_tsai_wei_tien_dylan_conjunctive_reducti>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261589": {
-      "content": "<dom_formosan_tsou_descriptive_study>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261590": {
-      "content": "<dom_formosan_tsukida_naomi_verb_classification_amis>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261591": {
-      "content": "<dom_formosan_tu_wen_chiu_a_synchronic_classification>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261592": {
-      "content": "<dom_formosan_tung_tsuchida_ting_li_pan_saaroa_texts>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261593": {
-      "content": "<dom_formosan_victoria_rau_yi_hsin_wu_and_meng_chien>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261594": {
-      "content": "<dom_formosan_wei_huilin_liu_social_structure_yami>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261595": {
-      "content": "<dom_formosan_wei_matri_clan_lineage_system_ami>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261596": {
-      "content": "<dom_formosan_wei_tien_tsai_subjecthood_temporal_adju>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261597": {
-      "content": "<dom_formosan_wiedfelt_formosan_aborigines_atayal>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261598": {
-      "content": "<dom_formosan_wilang_yutas_videos>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261599": {
-      "content": "<dom_formosan_wu_chunming_adverbials_in_paiwan>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261600": {
-      "content": "<dom_formosan_xie_fuhui_introduction_kavalan_grammar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261601": {
-      "content": "<dom_formosan_yedda_palemeq_blog>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261602": {
-      "content": "<dom_formosan_yeddas_blog>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261603": {
-      "content": "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261604": {
-      "content": "<dom_formosan_yi_ming_chou_object_control_saisiyat>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261605": {
-      "content": "<dom_formosan_yi_ting_chen_a_minimalist_approach_to_a>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261606": {
-      "content": "<dom_formosan_yi_ting_chen_amis_left_periphery>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261607": {
-      "content": "<dom_formosan_yi_ting_chen_language_use_code_mixing_a>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261608": {
-      "content": "<dom_formosan_yi_yang_cheng_kanakanavu_word_level_pro>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261609": {
-      "content": "<dom_formosan_yi_yang_cheng_tense_aspect_agent_markin>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261610": {
-      "content": "<dom_formosan_yoshiro_nihira_bunun_vocabulary>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261611": {
-      "content": "<dom_formosan_yuehchien_chien_the_lexical_system_of_y>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261612": {
-      "content": "<dom_formosan_yukihiro_yamada_ying_chu_liao_phonology>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261613": {
-      "content": "<dom_formosan_zeng_shifen_reduplication_affixation_pa>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261614": {
-      "content": "<dom_formosan_zhan_sujuan_taiwan_indigenous_peoples_h>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261615": {
-      "content": "<dom_formosan_zhang_xiujuan_introduction_paiwan_gramm>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261616": {
-      "content": "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261617": {
-      "content": "<dom_formosan_zhao_shanhe_li_taiyuan_zhu_qingyi_abori>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261618": {
-      "content": "<dom_formosan_zheng_acl_2024>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261619": {
-      "content": "<dom_formosan_zheng_data>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261620": {
-      "content": "<dom_formosan_zheng_yumei_zhu_qingyi_bo_hongming_abor>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261621": {
-      "content": "<dom_learning_vocab>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261622": {
-      "content": "<dom_nine_level>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261623": {
-      "content": "<dom_ntu>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261624": {
-      "content": "<dom_picture_book>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261625": {
-      "content": "<dom_picture_story>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261626": {
-      "content": "<dom_presidential_apology>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261627": {
-      "content": "<dom_reading_writing>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261628": {
-      "content": "<dom_unknown>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261629": {
-      "content": "<dom_youtube>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261630": {
-      "content": "<src_ami>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261631": {
-      "content": "<src_bnn>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261632": {
-      "content": "<src_ckv>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261633": {
-      "content": "<src_dru>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261634": {
-      "content": "<src_eng>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261635": {
-      "content": "<src_pwn>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261636": {
-      "content": "<src_pyu>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261637": {
-      "content": "<src_ssf>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261638": {
-      "content": "<src_sxr>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261639": {
-      "content": "<src_szy>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261640": {
-      "content": "<src_tao>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261641": {
-      "content": "<src_tay>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261642": {
-      "content": "<src_trv>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261643": {
-      "content": "<src_tsu>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261644": {
-      "content": "<src_xnb>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261645": {
-      "content": "<src_xsy>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261646": {
-      "content": "<src_zh>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261647": {
-      "content": "<to_ami>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261648": {
-      "content": "<to_bnn>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261649": {
-      "content": "<to_ckv>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261650": {
-      "content": "<to_dru>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261651": {
-      "content": "<to_eng>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261652": {
-      "content": "<to_pwn>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261653": {
-      "content": "<to_pyu>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261654": {
-      "content": "<to_ssf>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261655": {
-      "content": "<to_sxr>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261656": {
-      "content": "<to_szy>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261657": {
-      "content": "<to_tao>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261658": {
-      "content": "<to_tay>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261659": {
-      "content": "<to_trv>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261660": {
-      "content": "<to_tsu>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261661": {
-      "content": "<to_xnb>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261662": {
-      "content": "<to_xsy>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "261663": {
       "content": "<to_zh>",
       "lstrip": false,
       "normalized": false,
@@ -4373,222 +2717,15 @@
     "<dialect_zeaol>",
     "<dialect_zhiben>",
     "<dialect_zhuoqun>",
-    "<dom_classroom_context>",
     "<dom_culture>",
     "<dom_dictionary>",
-    "<dom_essays>",
-    "<dom_formosan_100_paiwan_texts>",
-    "<dom_formosan_academia_sinica_oral_legends>",
-    "<dom_formosan_adlay_kun_long_liu_syntactic_interactio>",
-    "<dom_formosan_adlay_liu_lfg_relativisation_jianshi_sq>",
-    "<dom_formosan_amis_adversative_constructions>",
-    "<dom_formosan_amis_huang_english_phonology_grammar>",
-    "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
-    "<dom_formosan_amis_kuo_sung_comparative_constructions>",
-    "<dom_formosan_amis_myths_and_customs>",
-    "<dom_formosan_amis_pa_verbs>",
-    "<dom_formosan_amis_serial_verb_constructions>",
-    "<dom_formosan_amis_tung_chiou>",
-    "<dom_formosan_amy_lee_body_part_nomenclature_seediq>",
-    "<dom_formosan_anna_hsiou_chuan_chang_reference_gramma>",
-    "<dom_formosan_anton_quack_puyuma_pulingaw>",
-    "<dom_formosan_apay_tang_anger_happiness_truku_seediq>",
-    "<dom_formosan_asai_sedik_language>",
-    "<dom_formosan_atayal_you_nao_savak_raxayal>",
-    "<dom_formosan_bien_daniel_chiang_house_social_hierarc>",
-    "<dom_formosan_boyizhenu_narrative_oral_literature_tap>",
-    "<dom_formosan_bunun_debusser_dissertation>",
-    "<dom_formosan_bunun_topic_focus>",
-    "<dom_formosan_c_douglas_chretien_oceanic_material_in>",
-    "<dom_formosan_catherine_tseng_seediq_atayal_foods_the>",
-    "<dom_formosan_chang_kwo_tan_syncretic_objects_materia>",
-    "<dom_formosan_chaolin_li_motion_verb_grammaticalizati>",
-    "<dom_formosan_chen_fukuda_relabeling_ergative>",
-    "<dom_formosan_chen_fukuda_three_ways_steal>",
-    "<dom_formosan_cheng_sung_modality_kanakanavu>",
-    "<dom_formosan_chi_lu_chen_and_micheal_d_coe_an_invest>",
-    "<dom_formosan_chuming_wu_linking_constructions_mayrin>",
-    "<dom_formosan_chun_mei_chen_comparative_phonology_pai>",
-    "<dom_formosan_chunming_wu_two_types_of_noun_incorpora>",
-    "<dom_formosan_cip_atayal_grammar_overview>",
-    "<dom_formosan_claire_mcgill_a_brief_tayal_vocabulary>",
-    "<dom_formosan_david_blundell_ed_austronesian_taiwan>",
-    "<dom_formosan_dean_johnson_year_clouds_smangus>",
-    "<dom_formosan_der_hwa_victoria_rau_grammar_atayal>",
-    "<dom_formosan_der_hwa_victoria_rau_lexical_similarity>",
-    "<dom_formosan_dong_manv_yami_ode_to_taro>",
-    "<dom_formosan_dong_yi_lin_universal_quantifier_kavala>",
-    "<dom_formosan_dong_yijia_tjuwabar_paiwan_ritual_langu>",
-    "<dom_formosan_dorinda_tsai_hsiu_liu_neutral_imperfect>",
-    "<dom_formosan_ed_torjesen_amis_taiwan_word_list>",
-    "<dom_formosan_egerod_origin_headhunting_atayal_text_v>",
-    "<dom_formosan_egerod_soren_word_order_word_classes_at>",
-    "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
-    "<dom_formosan_elizabeth_zeitoun_lillian_huang_anna_ch>",
-    "<dom_formosan_elizabeth_zeitoun_lillian_huang_marie_y>",
-    "<dom_formosan_elizabeth_zeitoun_pronominal_system_man>",
-    "<dom_formosan_elizabeth_zeitoun_squib_dynamic_vs_stat>",
-    "<dom_formosan_epark>",
-    "<dom_formosan_ferrell_taiwan_aboriginal_groups_proble>",
-    "<dom_formosan_george_leroy_shelley_vudai_rukai>",
-    "<dom_formosan_gitbook_translations>",
-    "<dom_formosan_glosbe>",
-    "<dom_formosan_gujing_lin_tsou_serial_verb_constructio>",
-    "<dom_formosan_henry_chang_nominal_aspect_tsou>",
-    "<dom_formosan_henry_chang_tsou_causative_applicative>",
-    "<dom_formosan_henry_y_chang_formosan_speech_act_mood>",
-    "<dom_formosan_ho_dan_phonological_system_butonglu_pai>",
-    "<dom_formosan_holmer_seediq>",
-    "<dom_formosan_hsiu_chuan_liao_transitivity_ergativity>",
-    "<dom_formosan_huang_dongqiu_atayal_bunun_amis_languag>",
-    "<dom_formosan_huang_grammaticalization_squliq_atayal>",
-    "<dom_formosan_huang_hui_chuan_glide_formation_takitud>",
-    "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
-    "<dom_formosan_hui_chuan_j_huang_syllable_types_in_bun>",
-    "<dom_formosan_hui_shan_lin_squliq_atayal_reduplicatio>",
-    "<dom_formosan_huteson_rukai_survey>",
-    "<dom_formosan_ian_maddieson_and_richard_wright_the_vo>",
-    "<dom_formosan_ilrdf_42_language_practice_word_lists>",
-    "<dom_formosan_ilrdf_tousvusvutu_kita>",
-    "<dom_formosan_indigenous_education_resource_center_ke>",
-    "<dom_formosan_indigenous_language_literary_awards>",
-    "<dom_formosan_isabelle_bril_roots_and_stems_lexical_a>",
-    "<dom_formosan_jian_iwan_guo_qingliu_life_history>",
-    "<dom_formosan_jian_shilang_introduction_thao_grammar>",
-    "<dom_formosan_john_tu_er_wei_notes_on_the_religious_b>",
-    "<dom_formosan_john_u_wolff_the_proto_austronesian_pho>",
-    "<dom_formosan_jonathan_kuo_amis_manner_result_verbs>",
-    "<dom_formosan_josiane_cauquelin_aborigines_taiwan_puy>",
-    "<dom_formosan_joy_wu_clausal_modifiers_in_amis>",
-    "<dom_formosan_joy_wu_verb_classification_case_marking>",
-    "<dom_formosan_kanakanavu_texts>",
-    "<dom_formosan_kavalan_zhang_reference_grammar>",
-    "<dom_formosan_kumu_tapas_tribal_memory_oral_history_w>",
-    "<dom_formosan_kuo_j_c_argument_alternation_and_argume>",
-    "<dom_formosan_lei_shih_family_system_paiwan_at_su_pai>",
-    "<dom_formosan_li_chen_yeh_developing_a_hla_alua_learn>",
-    "<dom_formosan_li_may_sung_budai_rukai_exclamatives>",
-    "<dom_formosan_li_may_sung_clausal_nominalization_buda>",
-    "<dom_formosan_lillian_huang_atayal_participants_cross>",
-    "<dom_formosan_lillian_huang_ergativity_atayal>",
-    "<dom_formosan_liu_manyi_kaskun_ata_matas_i_ki_quaz>",
-    "<dom_formosan_malcom_d_ross_the_history_and_transitiv>",
-    "<dom_formosan_malcom_ross_proto_austronesian_verbal_m>",
-    "<dom_formosan_man_ni_chu_the_measurement_of_final_i_v>",
-    "<dom_formosan_matsuzawa_kazuko_the_social_and_ritual>",
-    "<dom_formosan_maya_yeh_blaq_uv_construction_atayal>",
-    "<dom_formosan_mayumi_oiwa_adverbial_verb_construction>",
-    "<dom_formosan_mayumi_oiwa_bungard_morphology_and_synt>",
-    "<dom_formosan_mayumi_oiwa_possessor_raising_truku_see>",
-    "<dom_formosan_mei_chin_huang_verb_classification_in_m>",
-    "<dom_formosan_melissa_shih_hui_lin_sakizaya_or_amis_a>",
-    "<dom_formosan_nanang_tadaw_naci_mowna_pgagu>",
-    "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
-    "<dom_formosan_naomi_tsukida_ditransitive_causative_se>",
-    "<dom_formosan_nowbucyang_truku_thesis>",
-    "<dom_formosan_ochiai_izumi_2016_comparative_linguisti>",
-    "<dom_formosan_ochiai_izumi_bu_hwan_vocabulary_recorde>",
-    "<dom_formosan_ochiai_izumi_numerals_paran_seediq_rela>",
-    "<dom_formosan_ochiai_izumi_on_a_seediq_glossary_recor>",
-    "<dom_formosan_old_texts>",
-    "<dom_formosan_paiwan_ho_five_dialects>",
-    "<dom_formosan_paiwanstories>",
-    "<dom_formosan_pang_hsin_ting_reconstruction_proto_puy>",
-    "<dom_formosan_patricia_stanley_morphophonemics_of_ver>",
-    "<dom_formosan_paul_jen_kuei_li_comparative_bunun_dial>",
-    "<dom_formosan_paul_jen_kuei_li_conjunction_thao>",
-    "<dom_formosan_paul_jen_kuei_li_internal_relationships>",
-    "<dom_formosan_paul_jen_kuei_li_origins_of_the_east_fo>",
-    "<dom_formosan_paul_jen_kuei_li_review_of_taiwan_abori>",
-    "<dom_formosan_paul_jen_kuei_li_rukai_structure>",
-    "<dom_formosan_paul_jen_kuei_li_some_plant_names_in_fo>",
-    "<dom_formosan_paul_jen_kuei_li_the_preglottalised_sto>",
-    "<dom_formosan_paul_li_atayalic_final_voiced_stops>",
-    "<dom_formosan_paul_li_case_marking_four_formosan_lang>",
-    "<dom_formosan_paul_li_position_atayal_austronesian>",
-    "<dom_formosan_pingdong_county_government_kai_na_sepuc>",
-    "<dom_formosan_puyuma_chen_raising_to_object>",
-    "<dom_formosan_puyuma_katipol_kopfjagdriten>",
-    "<dom_formosan_puyuma_teng_pronominal_systems>",
-    "<dom_formosan_puyuma_teng_reference_grammar>",
-    "<dom_formosan_raleigh_ferrell_construction_markers_fo>",
-    "<dom_formosan_raleigh_ferrell_paiwan_phonology_and_pr>",
-    "<dom_formosan_raleigh_j_ferrell_intent_and_volition_i>",
-    "<dom_formosan_rik_bunun>",
-    "<dom_formosan_rik_de_busser_the_influence_of_christia>",
-    "<dom_formosan_robert_blust_a_note_on_covert_structure>",
-    "<dom_formosan_robert_blust_austronesian_homeland_ling>",
-    "<dom_formosan_robert_blust_some_remarks_linguistic_po>",
-    "<dom_formosan_robert_blust_thao_triplication>",
-    "<dom_formosan_robert_blust_the_austronesian_languages>",
-    "<dom_formosan_rukai_mantauran_stories>",
-    "<dom_formosan_rukai_texts>",
-    "<dom_formosan_rukai_zeitoun_saying>",
-    "<dom_formosan_sakizaya_affixes>",
-    "<dom_formosan_schetelig_mittheilungen_uber_die_sprach>",
-    "<dom_formosan_seals>",
-    "<dom_formosan_seediq_zhang_reference_grammar>",
-    "<dom_formosan_shigeru_tsuchida_and_isidore_dyen_proto>",
-    "<dom_formosan_shigeru_tsuchida_kanakanavu_texts>",
-    "<dom_formosan_shigeru_tsuchida_preliminary_reports_on>",
-    "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
-    "<dom_formosan_shih_chi_stella_yeh_stress_and_quantity>",
-    "<dom_formosan_shih_rukai_adverbial>",
-    "<dom_formosan_sihwei_chen_and_haowen_jiang_ways_of_ta>",
-    "<dom_formosan_sihwei_chen_lexical_aspect_atayal>",
-    "<dom_formosan_soren_egerod_statement_atayal_phonology>",
-    "<dom_formosan_soren_egerod_verb_inflection_atayal>",
-    "<dom_formosan_stacy_teng_malcom_ross_is_puyuma_primar>",
-    "<dom_formosan_stacy_wan_tin_huang_functions_yami_verb>",
-    "<dom_formosan_sung_kuo_descriptive_comparative_constr>",
-    "<dom_formosan_taoshan_elementary_school_atelier_hui_k>",
-    "<dom_formosan_teresa_m_chen_verbal_construction_and_v>",
-    "<dom_formosan_ting_chi_wei_parallelism_in_amis_sluici>",
-    "<dom_formosan_tingchun_chen_successive_cyclic_case_as>",
-    "<dom_formosan_truku_lowking_demonstratives>",
-    "<dom_formosan_tsai_hsui_liu_complementation_three_lan>",
-    "<dom_formosan_tsai_wei_tien_dylan_conjunctive_reducti>",
-    "<dom_formosan_tsou_descriptive_study>",
-    "<dom_formosan_tsukida_naomi_verb_classification_amis>",
-    "<dom_formosan_tu_wen_chiu_a_synchronic_classification>",
-    "<dom_formosan_tung_tsuchida_ting_li_pan_saaroa_texts>",
-    "<dom_formosan_victoria_rau_yi_hsin_wu_and_meng_chien>",
-    "<dom_formosan_wei_huilin_liu_social_structure_yami>",
-    "<dom_formosan_wei_matri_clan_lineage_system_ami>",
-    "<dom_formosan_wei_tien_tsai_subjecthood_temporal_adju>",
-    "<dom_formosan_wiedfelt_formosan_aborigines_atayal>",
-    "<dom_formosan_wilang_yutas_videos>",
-    "<dom_formosan_wu_chunming_adverbials_in_paiwan>",
-    "<dom_formosan_xie_fuhui_introduction_kavalan_grammar>",
-    "<dom_formosan_yedda_palemeq_blog>",
-    "<dom_formosan_yeddas_blog>",
-    "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
-    "<dom_formosan_yi_ming_chou_object_control_saisiyat>",
-    "<dom_formosan_yi_ting_chen_a_minimalist_approach_to_a>",
-    "<dom_formosan_yi_ting_chen_amis_left_periphery>",
-    "<dom_formosan_yi_ting_chen_language_use_code_mixing_a>",
-    "<dom_formosan_yi_yang_cheng_kanakanavu_word_level_pro>",
-    "<dom_formosan_yi_yang_cheng_tense_aspect_agent_markin>",
-    "<dom_formosan_yoshiro_nihira_bunun_vocabulary>",
-    "<dom_formosan_yuehchien_chien_the_lexical_system_of_y>",
-    "<dom_formosan_yukihiro_yamada_ying_chu_liao_phonology>",
-    "<dom_formosan_zeng_shifen_reduplication_affixation_pa>",
-    "<dom_formosan_zhan_sujuan_taiwan_indigenous_peoples_h>",
-    "<dom_formosan_zhang_xiujuan_introduction_paiwan_gramm>",
-    "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
-    "<dom_formosan_zhao_shanhe_li_taiyuan_zhu_qingyi_abori>",
-    "<dom_formosan_zheng_acl_2024>",
-    "<dom_formosan_zheng_data>",
-    "<dom_formosan_zheng_yumei_zhu_qingyi_bo_hongming_abor>",
-    "<dom_learning_vocab>",
-    "<dom_nine_level>",
-    "<dom_ntu>",
-    "<dom_picture_book>",
-    "<dom_picture_story>",
-    "<dom_presidential_apology>",
-    "<dom_reading_writing>",
     "<dom_unknown>",
-    "<dom_youtube>",
     "<src_ami>",
     "<src_bnn>",
     "<src_ckv>",

       "single_word": false,
       "special": true
     },
+    "261425": {
       "content": "ami_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261426": {
       "content": "bnn_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261427": {
       "content": "ckv_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261428": {
       "content": "dru_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261429": {
       "content": "pwn_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261430": {
       "content": "pyu_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261431": {
       "content": "ssf_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261432": {
       "content": "sxr_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261433": {
       "content": "szy_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261434": {
       "content": "tao_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261435": {
       "content": "tay_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261436": {
       "content": "trv_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261437": {
       "content": "tsu_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261438": {
       "content": "xnb_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261439": {
       "content": "xsy_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261440": {
       "content": "<dialect_central>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261441": {
       "content": "<dialect_coastal>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261442": {
       "content": "<dialect_dawu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261443": {
       "content": "<dialect_default>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261444": {
       "content": "<dialect_deluvalley>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261445": {
       "content": "<dialect_dona>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261446": {
       "content": "<dialect_duda>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261447": {
       "content": "<dialect_eastern>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261448": {
       "content": "<dialect_fourseasons>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261449": {
       "content": "<dialect_hengchun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261450": {
       "content": "<dialect_jianhe>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261451": {
       "content": "<dialect_junqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261452": {
       "content": "<dialect_kanakanavu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261453": {
       "content": "<dialect_kaqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261454": {
       "content": "<dialect_kavalan>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261455": {
       "content": "<dialect_luanqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261456": {
       "content": "<dialect_malan>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261457": {
       "content": "<dialect_maolin>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261458": {
       "content": "<dialect_nanwang>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261459": {
       "content": "<dialect_northern>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261460": {
       "content": "<dialect_saaroa>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261461": {
       "content": "<dialect_saisiyat>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261462": {
       "content": "<dialect_sakizaya>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261463": {
       "content": "<dialect_sekolik>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261464": {
       "content": "<dialect_southern>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261465": {
       "content": "<dialect_tanqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261466": {
       "content": "<dialect_tegudaya>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261467": {
       "content": "<dialect_thao>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261468": {
       "content": "<dialect_truku>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261469": {
       "content": "<dialect_tsou>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261470": {
       "content": "<dialect_unknown>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261471": {
       "content": "<dialect_wanda>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261472": {
       "content": "<dialect_wanshan>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261473": {
       "content": "<dialect_wenshui>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261474": {
       "content": "<dialect_wutai>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261475": {
       "content": "<dialect_xiqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261476": {
       "content": "<dialect_xiuguluan>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261477": {
       "content": "<dialect_yami>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261478": {
       "content": "<dialect_yilanzeaol>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261479": {
       "content": "<dialect_zeaol>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261480": {
       "content": "<dialect_zhiben>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261481": {
       "content": "<dialect_zhuoqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261482": {
+      "content": "<dom_classroom>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261483": {
       "content": "<dom_culture>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261484": {
       "content": "<dom_dictionary>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261485": {
+      "content": "<dom_education>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261486": {
+      "content": "<dom_linguistic>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261487": {
+      "content": "<dom_media>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261488": {
+      "content": "<dom_narrative>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261489": {
+      "content": "<dom_religious>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261490": {
+      "content": "<dom_unknown>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261491": {
+      "content": "<src_ami>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261492": {
+      "content": "<src_bnn>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261493": {
+      "content": "<src_ckv>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261494": {
+      "content": "<src_dru>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261495": {
+      "content": "<src_eng>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261496": {
+      "content": "<src_pwn>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261497": {
+      "content": "<src_pyu>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261498": {
+      "content": "<src_ssf>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261499": {
+      "content": "<src_sxr>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261500": {
+      "content": "<src_szy>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261501": {
+      "content": "<src_tao>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261502": {
+      "content": "<src_tay>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261503": {
+      "content": "<src_trv>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261504": {
+      "content": "<src_tsu>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261505": {
+      "content": "<src_xnb>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261506": {
+      "content": "<src_xsy>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261507": {
+      "content": "<src_zh>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261508": {
+      "content": "<to_ami>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261509": {
+      "content": "<to_bnn>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261510": {
+      "content": "<to_ckv>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261511": {
+      "content": "<to_dru>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261512": {
+      "content": "<to_eng>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261513": {
+      "content": "<to_pwn>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261514": {
+      "content": "<to_pyu>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261515": {
+      "content": "<to_ssf>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261516": {
+      "content": "<to_sxr>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261517": {
+      "content": "<to_szy>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261518": {
+      "content": "<to_tao>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261519": {
+      "content": "<to_tay>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261520": {
+      "content": "<to_trv>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261521": {
+      "content": "<to_tsu>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261522": {
+      "content": "<to_xnb>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261523": {
+      "content": "<to_xsy>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261524": {
       "content": "<to_zh>",
       "lstrip": false,
       "normalized": false,
     "<dialect_zeaol>",
     "<dialect_zhiben>",
     "<dialect_zhuoqun>",
+    "<dom_classroom>",
     "<dom_culture>",
     "<dom_dictionary>",
+    "<dom_education>",
+    "<dom_linguistic>",
+    "<dom_media>",
+    "<dom_narrative>",
+    "<dom_religious>",
     "<dom_unknown>",
     "<src_ami>",
     "<src_bnn>",
     "<src_ckv>",

training_profile.json CHANGED Viewed

@@ -1,43 +1,32 @@
 {
   "schema_version": 3,
-  "recipe_id": "nllb200-spm8k-directional-v3",
   "model_family": "nllb",
   "corpus_pipeline_version": "formosan-mt-corpus-v3",
   "mt_standardization": {
     "id": "formosan-mt-standard-v3",
-    "sha256": "4bbded87eb4833b2d1cd5a88a1f2b059560416e0e69ecfa55433a77f418c903e",
     "namespace": "formosan-mt"
   },
-  "project_root": ".",
-  "corpus_variants": ["public_no_bible", "private_no_bible"],
   "base_model": {
     "name": "facebook/nllb-200-distilled-600M",
     "revision": "f8d333a098d19b4fd9a8b18f94170487ad3f821d"
   },
-  "primary_input_pattern": "corpus_builds/{corpus_name}/pivot_corpora_final/big_corpus_{target}_in_domain_hard.csv",
-  "targets": {
-    "english": {
-      "input_pattern": "corpus_builds/{corpus_name}/pivot_corpora_final/big_corpus_en_in_domain_hard.csv",
-      "target_col": "english_sentence",
-      "directions": ["f2en", "en2f"],
-      "selected_recipe": "NLLB-200 SPM8k directional"
-    },
-    "chinese": {
-      "input_pattern": "corpus_builds/{corpus_name}/pivot_corpora_final/big_corpus_zh_in_domain_hard.csv",
-      "target_col": "chinese_sentence",
-      "directions": ["f2zh", "zh2f"],
-      "selected_recipe": "NLLB-200 SPM8k directional"
-    }
-  },
   "splits": {
-    "english_output_dir": "formosan_mt_experiments/data/splits_en_v1",
-    "chinese_output_dir": "formosan_mt_experiments/data/splits_zh_v1",
-    "train_ratio": 0.9,
-    "validate_ratio": 0.025,
-    "test_ratio": 0.075,
-    "min_test_rows": 500,
-    "min_validate_rows": 150,
-    "character_ngram_jaccard_threshold": 0.82,
     "tiers": ["in_domain_hard"],
     "headline_tier": "in_domain_hard"
   },
@@ -54,14 +43,16 @@
   },
   "training_defaults": {
     "steps": 300000,
-    "batch_size": 16,
-    "grad_accum_steps": 4,
     "effective_batch_size": 64,
     "max_length": 384,
     "learning_rate": 0.00002,
     "warmup_steps": 4000,
     "weight_decay": 0.001,
     "alpha": 0.5,
     "precision": "bf16",
     "label_smoothing": 0.1,
     "save_interval": 0,
@@ -70,6 +61,7 @@
     "generation_eval_samples_per_language": 128,
     "generation_eval_batch_size": 16,
     "validation_beam": 2,
     "best_metric": "chrF2",
     "early_stopping_patience": 5,
     "early_stopping_min_delta": 0.05,
@@ -87,8 +79,8 @@
     "no_repeat_ngram_size": 0,
     "repetition_penalty": 1.0,
     "length_penalty": 1.0,
-    "metadata_modes": ["default", "oracle"],
-    "bootstrap_samples": 200,
     "bootstrap_seed": 42
   }
 }

 {
   "schema_version": 3,
+  "recipe_id": "nllb200-spm8k-directional-v5",
   "model_family": "nllb",
   "corpus_pipeline_version": "formosan-mt-corpus-v3",
   "mt_standardization": {
     "id": "formosan-mt-standard-v3",
+    "sha256": "3c93c1abecfc68c72e3bb1d547fbe0bdebb14bb74e7a5d3c5c519bd18d34bf6f",
     "namespace": "formosan-mt"
   },
   "base_model": {
     "name": "facebook/nllb-200-distilled-600M",
     "revision": "f8d333a098d19b4fd9a8b18f94170487ad3f821d"
   },
   "splits": {
+    "train_ratio": 0.85,
+    "validate_ratio": 0.05,
+    "test_ratio": 0.10,
+    "min_test_rows": 0,
+    "min_validate_rows": 0,
+    "min_formosan_tokens": 2,
+    "min_target_tokens": 2,
+    "min_combined_tokens": 6,
+    "min_punctuated_combined_tokens": 5,
+    "max_eval_units_per_side": 384,
+    "source_ratio_tolerance": 0.02,
+    "ratio_basis": "all_pairs_by_language_human_rows_by_source",
+    "synthetic_eval_policy": "human_only_source_stratified",
+    "character_ngram_jaccard_threshold": 0.95,
     "tiers": ["in_domain_hard"],
     "headline_tier": "in_domain_hard"
   },
   },
   "training_defaults": {
     "steps": 300000,
+    "batch_size": 8,
+    "grad_accum_steps": 8,
     "effective_batch_size": 64,
     "max_length": 384,
     "learning_rate": 0.00002,
     "warmup_steps": 4000,
     "weight_decay": 0.001,
     "alpha": 0.5,
+    "domain_tag_dropout": 0.25,
+    "dialect_tag_dropout": 0.25,
     "precision": "bf16",
     "label_smoothing": 0.1,
     "save_interval": 0,
     "generation_eval_samples_per_language": 128,
     "generation_eval_batch_size": 16,
     "validation_beam": 2,
+    "validation_metadata_mode": "default",
     "best_metric": "chrF2",
     "early_stopping_patience": 5,
     "early_stopping_min_delta": 0.05,
     "no_repeat_ngram_size": 0,
     "repetition_penalty": 1.0,
     "length_penalty": 1.0,
+    "metadata_modes": ["default"],
+    "bootstrap_samples": 0,
     "bootstrap_seed": 42
   }
 }