Hugging Face's logo

PantagrueLLM
/
text-base-wiki-mlm

Feature Extraction
Transformers
Safetensors
Fairseq
French
pantagruel_uni
fill-mask
data2vec2
JEPA
text
custom_code
Model card Files
xet
 Community
vpelloin commited on
Commit
dfcd53f
·
verified ·
1 Parent(s): bc92046

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -7
README.md CHANGED
@@ -3,9 +3,14 @@ license: cc-by-nc-sa-2.0
3
  language:
4
  - fr
5
  pipeline_tag: feature-extraction
6
- library_name: fairseq
7
  datasets:
8
  - wikimedia/wikipedia
 
 
 
 
 
 
9
  ---
10
 
11
  # Pantagruel: Unified Self-Supervised Encoders for French Text and Speech
@@ -28,14 +33,14 @@ Pantagruel text encoders are trained on large-scale French text corpora, includi
28
  **Note on model naming convention:** Models that include `camtok` in their name use CamemBERT's tokenizer, which is used for comparison our models to a BERT-based counterpart. If no tokenizer is specified, the model uses our custom tokenizer. All text-based models are trained using the data2vec 2.0 masked feature prediction objective. Models with an `MLM` suffix additionally incorporate the masked language modeling (MLM) objective alongside the main data2vec 2.0 objective.
29
  The table below presents the accuracy of the natural language inference task on the French XNLI dataset.
30
 
31
- | **HuggingFace name**| **Model name (paper)** | **Arch/ Params** | **Pretrained dataset** | **Accuracy on XNLI (FR) (dev / test)** |
32
  |----------|------------------------|-----------------|----------------------|---------------------------------------|
33
- | text-base-camtok-wiki | Pantagruel-B-camtok-Wk | Base / 110M | French Wikipedia 2019 (4GB) | 76.94% / 77.43% |
34
- | text-base-wiki | Pantagruel-B-Wk | Base / 125M | French Wikipedia 2019 (4GB) | 77.40% / 78.41% |
35
  | text-base-wiki-mlm | Pantagruel-B-Wk-MLM | Base / 125M | French Wikipedia 2019 (4GB) | 78.25% / 78.41% |
36
- | text-base-camtok-oscar | Pantagruel-B-camtok-Osc | Base / 110M | OSCAR 2019 (138GB) | 80.40% / 80.53% |
37
- | text-base-oscar-mlm | Pantagruel-B-Osc-MLM | Base / 125M | OSCAR 2019 (138GB) | 81.11% / 81.52% |
38
- | text-base-croissant-mlm | Pantagruel-B-Crs-MLM | Base / 125M | croissantLLM (1.5GB) | 81.05% / 80.69% |
39
 
40
  For more downstream tasks and evaluation datasets, please refer to [our paper](https://arxiv.org/abs/2601.05911).
41
 
 
3
  language:
4
  - fr
5
  pipeline_tag: feature-extraction
 
6
  datasets:
7
  - wikimedia/wikipedia
8
+ library_name: transformers
9
+ tags:
10
+ - data2vec2
11
+ - JEPA
12
+ - text
13
+ - fairseq
14
  ---
15
 
16
  # Pantagruel: Unified Self-Supervised Encoders for French Text and Speech
 
33
  **Note on model naming convention:** Models that include `camtok` in their name use CamemBERT's tokenizer, which is used for comparison our models to a BERT-based counterpart. If no tokenizer is specified, the model uses our custom tokenizer. All text-based models are trained using the data2vec 2.0 masked feature prediction objective. Models with an `MLM` suffix additionally incorporate the masked language modeling (MLM) objective alongside the main data2vec 2.0 objective.
34
  The table below presents the accuracy of the natural language inference task on the French XNLI dataset.
35
 
36
+ | **HuggingFace name**| **Model name (paper)** | **Arch/ Params** | **Pretrained dataset** | **Accuracy on XNLI (FR) (dev / test)** |
37
  |----------|------------------------|-----------------|----------------------|---------------------------------------|
38
+ | [text-base-camtok-wiki](https://huggingface.co/PantagrueLLM/text-base-camtok-wiki) | Pantagruel-B-camtok-Wk | Base / 110M | French Wikipedia 2019 (4GB) | 76.94% / 77.43% |
39
+ | [text-base-wiki](https://huggingface.co/PantagrueLLM/text-base-wiki) | Pantagruel-B-Wk | Base / 125M | French Wikipedia 2019 (4GB) | 77.40% / 78.41% |
40
  | text-base-wiki-mlm | Pantagruel-B-Wk-MLM | Base / 125M | French Wikipedia 2019 (4GB) | 78.25% / 78.41% |
41
+ | [text-base-camtok-oscar](https://huggingface.co/PantagrueLLM/text-base-camtok-oscar) | Pantagruel-B-camtok-Osc | Base / 110M | OSCAR 2019 (138GB) | 80.40% / 80.53% |
42
+ | [text-base-oscar-mlm](https://huggingface.co/PantagrueLLM/text-base-oscar-mlm) | Pantagruel-B-Osc-MLM | Base / 125M | OSCAR 2019 (138GB) | 81.11% / 81.52% |
43
+ | [text-base-croissant-mlm](https://huggingface.co/PantagrueLLM/text-base-croissant-mlm) | Pantagruel-B-Crs-MLM | Base / 125M | croissantLLM (1.5GB) | 81.05% / 80.69% |
44
 
45
  For more downstream tasks and evaluation datasets, please refer to [our paper](https://arxiv.org/abs/2601.05911).
46