anonymous-stoicheia commited on
Commit
790408f
·
verified ·
1 Parent(s): b7332f4

Fix the usage example: fetch the processor before importing it

Browse files
Files changed (1) hide show
  1. README.md +8 -3
README.md CHANGED
@@ -52,16 +52,21 @@ from processing_char_bert import CharBertProcessor
52
 
53
  proc = CharBertProcessor()
54
 
55
- print(proc.restore_respaced(model, "ἔδοξεν τῇ βου-- καὶ τῷ δήμῳ"))
 
 
56
  # ἔδοξεν τῇ βουλῇ καὶ τῷ δήμῳ
57
 
58
- print(proc.restore_respaced(model, "στεφανῶσαι αὐτὸν χρυσῷ στεφά[3±1] ἀρετς ἕνεκα"))
59
  # στεφανῶσαι αὐτὸν χρυσῷ στεφάνῳ ἀρετῆς ἕνεκα
60
  ```
61
 
62
  `restore_respaced` works in the order an editor would: it fills the letters first, then throws
63
  the spacing away and runs the model again over the resulting *scriptio continua* with word
64
- division and accents unknown everywhere -- the regime it was pretrained on. Deciding the letters
 
 
 
65
  and the segmentation in a single pass, which `decode_restoration` does, leaves the boundary head
66
  hedging against a half-known segmentation, and a correctly restored word can come back cut in
67
  two. The evaluation harness in the code repository goes further still, enumerating the division
 
52
 
53
  proc = CharBertProcessor()
54
 
55
+ # note the input: no accents on τηβου, and no word division either -- the model
56
+ # recovers the missing letters, the accents and the spacing together
57
+ print(proc.restore_respaced(model, "ἔδοξεν τηβου-- καὶ τῷ δήμῳ"))
58
  # ἔδοξεν τῇ βουλῇ καὶ τῷ δήμῳ
59
 
60
+ print(proc.restore_respaced(model, "στεφανῶσαι αὐτὸν χρυσῷ στεφα[3±1]ετης ἕνεκα"))
61
  # στεφανῶσαι αὐτὸν χρυσῷ στεφάνῳ ἀρετῆς ἕνεκα
62
  ```
63
 
64
  `restore_respaced` works in the order an editor would: it fills the letters first, then throws
65
  the spacing away and runs the model again over the resulting *scriptio continua* with word
66
+ division and accents unknown everywhere -- the regime it was pretrained on. Nothing in the
67
+ input has to be normalized first: accents and word division are predictions, not requirements,
68
+ so a bare majuscule transcript is as readable to this model as a modern critical text, and the
69
+ gap is filled in the same pass that decides where the words end. Deciding the letters
70
  and the segmentation in a single pass, which `decode_restoration` does, leaves the boundary head
71
  hedging against a half-known segmentation, and a correctly restored word can come back cut in
72
  two. The evaluation harness in the code repository goes further still, enumerating the division