KoichiYasuoka
/

roberta-classical-chinese-base-sentence-segmentation

Token Classification

Literary Chinese

classical chinese

literary chinese

ancient chinese

sentence segmentation

Model card Files Files and versions

KoichiYasuoka commited on Aug 17, 2021

Commit

58833e0

·

1 Parent(s): 12cdd2b

bug fix

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -24,7 +24,7 @@ This is a RoBERTa model pre-trained on Classical Chinese texts for sentence segm
 import torch
 from transformers import AutoTokenizer,AutoModelForTokenClassification
 tokenizer=AutoTokenizer.from_pretrained("KoichiYasuoka/roberta-classical-chinese-base-sentence-segmentation")
-model=AutoModelForMaskedLM.from_pretrained("KoichiYasuoka/roberta-classical-chinese-base-sentence-segmentation")
 s="子曰學而時習之不亦說乎有朋自遠方來不亦樂乎人不知而不慍不亦君子乎"
 p=[model.config.id2label[q] for q in torch.argmax(model(tokenizer.encode(s,return_tensors="pt"))[0],dim=2)[0].tolist()[1:-1]]
 print("".join(c+"。" if q=="E" or q=="S" else c for c,q in zip(s,p)))

 import torch
 from transformers import AutoTokenizer,AutoModelForTokenClassification
 tokenizer=AutoTokenizer.from_pretrained("KoichiYasuoka/roberta-classical-chinese-base-sentence-segmentation")
+model=AutoModelForTokenClassification.from_pretrained("KoichiYasuoka/roberta-classical-chinese-base-sentence-segmentation")
 s="子曰學而時習之不亦說乎有朋自遠方來不亦樂乎人不知而不慍不亦君子乎"
 p=[model.config.id2label[q] for q in torch.argmax(model(tokenizer.encode(s,return_tensors="pt"))[0],dim=2)[0].tolist()[1:-1]]
 print("".join(c+"。" if q=="E" or q=="S" else c for c,q in zip(s,p)))