| 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:425 INFO: Namespace(input_files=['lex/fr/realwords.rw.txt.bz2', 'lex/fr/morph.rw.txt.bz2', 'lex/fr/single_char.rw.txt.bz2', 'lex/fr/frwiki.unigram.freq.tsv.bz2', 'lex/fr/nonwords.nw.txt.bz2'], bloom_path='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-fr.bloom', fp_probability=1e-05, log_level='INFO', log_file='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-fr.bloom.log', config=None, min_frequency=2, single_char_min_frequency=20, diagnose_bloom=True) |
| 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:226 INFO: Starting Bloom Filter creation... |
| 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:178 INFO: Processing nonword file: lex/fr/nonwords.nw.txt.bz2 |
| 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['gazelle'] |
| 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:190 INFO: Excluded 1 words that should never be added |
| 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/realwords.rw.txt.bz2 |
| 2025-02-10 09:54:39,730 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/morph.rw.txt.bz2 |
| 2025-02-10 09:54:40,372 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/single_char.rw.txt.bz2 |
| 2025-02-10 09:54:40,373 ocrqa_create_bloom_filter.py:135 INFO: Processing frequency file: lex/fr/frwiki.unigram.freq.tsv.bz2 |
| 2025-02-10 09:54:45,072 ocrqa_create_bloom_filter.py:240 INFO: low_freq_excluded before removing parts from high-frequency words: 1552240 |
| 2025-02-10 09:54:45,172 ocrqa_create_bloom_filter.py:248 INFO: low_freq_excluded after removing parts from high-frequency words: 1228188 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:252 INFO: Lexical processing complete. |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - nonwords_read: 1 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - nonwords_count: 1 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - realwords_read: 723250 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - realwords_accepted: 736917 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - realwords_nonwords_filtered: 1 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - freq_words_read: 4047539 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - words_accepted: 1992185 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - subwords_accepted: 2383447 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - subwords_filtered: 9 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - low_freq_excluded: 1228188 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - single_char_words_filtered: 6597 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - freq_words_filtered: 2048757 |
| 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:259 INFO: Estimated word count: 1939265 |
| 2025-02-10 09:54:45,657 ocrqa_create_bloom_filter.py:263 INFO: Bloom Filter created and saved to build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-fr.bloom |
| 2025-02-10 09:54:45,996 ocrqa_create_bloom_filter.py:285 INFO: Diagnosis Results: |
| 2025-02-10 09:54:45,996 ocrqa_create_bloom_filter.py:286 INFO: - Excluded words in bloom filter: 0 |
| 2025-02-10 09:54:45,996 ocrqa_create_bloom_filter.py:287 INFO: - Known words not in bloom filter: 0 |
| 2025-02-10 09:54:46,146 ocrqa_create_bloom_filter.py:294 INFO: - Low-frequency words in bloom filter: 8 |
| 2025-02-10 09:54:46,146 ocrqa_create_bloom_filter.py:300 INFO: - Proportion of excluded words in bloom filter: 0.00000000 |
| 2025-02-10 09:54:46,146 ocrqa_create_bloom_filter.py:306 INFO: - Proportion of known words not in bloom filter: 0.00000000 |
| 2025-02-10 09:54:46,146 ocrqa_create_bloom_filter.py:314 INFO: - Proportion of low-frequency words in bloom filter: 0.00000651 |
|
|
