--- language: - fr tags: - sentence-transformers - sparse-encoder - sparse - splade - generated_from_trainer - dataset_size:12227 - loss:SpladeLoss - loss:SparseCosineSimilarityLoss - loss:FlopsLoss base_model: EuroBERT/EuroBERT-210m widget: - text: Une femme, un petit garçon et un petit bébé se tiennent devant une statue de vache. - text: En anglais, l'utilisation la plus courante de do est certainement Do-Support. - text: Je ne pense pas que la charge de la preuve repose sur des versions positives ou négatives. - text: Cinq lévriers courent sur une piste de sable. - text: J'envisage de dépenser les 48 dollars par mois pour le système GTD (Getting things done) annoncé par David Allen. datasets: - CATIE-AQ/frenchSTS pipeline_tag: feature-extraction library_name: sentence-transformers metrics: - pearson_cosine - spearman_cosine - active_dims - sparsity_ratio model-index: - name: SPLADE Sparse Encoder results: - task: type: semantic-similarity name: Semantic Similarity dataset: name: sts dev type: sts-dev metrics: - type: pearson_cosine value: 0.36927990368160857 name: Pearson Cosine - type: spearman_cosine value: 0.37469558585276597 name: Spearman Cosine - type: active_dims value: 11.744894981384277 name: Active Dims - type: sparsity_ratio value: 0.9999084261556466 name: Sparsity Ratio - task: type: semantic-similarity name: Semantic Similarity dataset: name: sts test type: sts-test metrics: - type: pearson_cosine value: 0.35396199996568556 name: Pearson Cosine - type: spearman_cosine value: 0.3848714264523539 name: Spearman Cosine - type: active_dims value: 10.178544044494629 name: Active Dims - type: sparsity_ratio value: 0.9999206388469585 name: Sparsity Ratio --- # SPLADE Sparse Encoder This is a [SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [EuroBERT/EuroBERT-210m](https://huggingface.co/EuroBERT/EuroBERT-210m) on the [french_sts](https://huggingface.co/datasets/CATIE-AQ/frenchSTS) dataset using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 128256-dimensional sparse vector space and can be used for semantic search and sparse retrieval. ## Model Details ### Model Description - **Model Type:** SPLADE Sparse Encoder - **Base model:** [EuroBERT/EuroBERT-210m](https://huggingface.co/EuroBERT/EuroBERT-210m) - **Maximum Sequence Length:** 8192 tokens - **Output Dimensionality:** 128256 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - [french_sts](https://huggingface.co/datasets/CATIE-AQ/frenchSTS) - **Language:** fr ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder) ### Full Model Architecture ``` SparseEncoder( (0): MLMTransformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'EuroBertForMaskedLM'}) (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 128256}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SparseEncoder # Download from the 🤗 Hub model = SparseEncoder("bourdoiscatie/SPLADE_EuroBERT-210m_STS") # Run inference sentences = [ "Oui, je peux vous dire d'après mon expérience personnelle qu'ils ont certainement sifflé.", "Il est vrai que les bombes de la Seconde Guerre mondiale faisaient un bruit de sifflet lorsqu'elles tombaient.", "J'envisage de dépenser les 48 dollars par mois pour le système GTD (Getting things done) annoncé par David Allen.", ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 128256] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities) # tensor([[1.0000, 0.6421, 0.5114], # [0.6421, 1.0000, 0.3132], # [0.5114, 0.3132, 1.0000]]) ``` ## Evaluation ### Metrics #### Semantic Similarity * Datasets: `sts-dev` and `sts-test` * Evaluated with [SparseEmbeddingSimilarityEvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseEmbeddingSimilarityEvaluator) | Metric | sts-dev | sts-test | |:--------------------|:-----------|:-----------| | pearson_cosine | 0.3693 | 0.354 | | **spearman_cosine** | **0.3747** | **0.3849** | | active_dims | 11.7449 | 10.1785 | | sparsity_ratio | 0.9999 | 0.9999 | ## Training Details ### Training Dataset #### french_sts * Dataset: [french_sts](https://huggingface.co/datasets/CATIE-AQ/frenchSTS) at [47128cc](https://huggingface.co/datasets/CATIE-AQ/frenchSTS/tree/47128cc18c893e5b93679037cdca303849e05309) * Size: 12,227 training samples * Columns: sentence1, sentence2, and score * Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | score | |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------| | type | string | string | float | | details | | | | * Samples: | sentence1 | sentence2 | score | |:----------------------------------------------------|:----------------------------------------------------|:---------------------------------| | Un avion est en train de décoller. | Un avion est en train de décoller. | 1.0 | | Un homme est en train de fumer. | Un homme fait du patinage. | 0.10000000149011612 | | Une personne jette un chat au plafond. | Une personne jette un chat au plafond. | 1.0 | * Loss: [SpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters: ```json { "loss": "SparseCosineSimilarityLoss(loss_fct='torch.nn.modules.loss.MSELoss')", "document_regularizer_weight": 0.003 } ``` ### Evaluation Dataset #### french_sts * Dataset: [french_sts](https://huggingface.co/datasets/CATIE-AQ/frenchSTS) at [47128cc](https://huggingface.co/datasets/CATIE-AQ/frenchSTS/tree/47128cc18c893e5b93679037cdca303849e05309) * Size: 3,526 evaluation samples * Columns: sentence1, sentence2, and score * Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | score | |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------| | type | string | string | float | | details | | | | * Samples: | sentence1 | sentence2 | score | |:-------------------------------------------------------------------------|:----------------------------------------------------------------------------|:-------------------------------| | Un homme avec un casque de sécurité est en train de danser. | Un homme portant un casque de sécurité est en train de danser. | 1.0 | | Un jeune enfant monte à cheval. | Un enfant monte à cheval. | 0.949999988079071 | | Un homme donne une souris à un serpent. | L'homme donne une souris au serpent. | 1.0 | * Loss: [SpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters: ```json { "loss": "SparseCosineSimilarityLoss(loss_fct='torch.nn.modules.loss.MSELoss')", "document_regularizer_weight": 0.003 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 16 - `per_device_eval_batch_size`: 16 - `bf16`: True #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 16 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 5e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 3 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.0 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `tp_size`: 0 - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | Validation Loss | sts-dev_spearman_cosine | sts-test_spearman_cosine | |:------:|:----:|:-------------:|:---------------:|:-----------------------:|:------------------------:| | -1 | -1 | - | - | 0.3062 | - | | 0.1307 | 100 | 0.1403 | - | - | - | | 0.2614 | 200 | 0.1135 | - | - | - | | 0.3922 | 300 | 0.117 | - | - | - | | 0.5229 | 400 | 0.111 | - | - | - | | 0.6536 | 500 | 0.1003 | - | - | - | | 0.7843 | 600 | 0.1051 | - | - | - | | 0.9150 | 700 | 0.1074 | - | - | - | | 1.0 | 765 | - | 0.1126 | 0.3033 | - | | 1.0458 | 800 | 0.0967 | - | - | - | | 1.1765 | 900 | 0.0788 | - | - | - | | 1.3072 | 1000 | 0.0833 | - | - | - | | 1.4379 | 1100 | 0.076 | - | - | - | | 1.5686 | 1200 | 0.0665 | - | - | - | | 1.6993 | 1300 | 0.072 | - | - | - | | 1.8301 | 1400 | 0.0675 | - | - | - | | 1.9608 | 1500 | 0.073 | - | - | - | | 2.0 | 1530 | - | 0.0919 | 0.3570 | - | | 2.0915 | 1600 | 0.053 | - | - | - | | 2.2222 | 1700 | 0.0467 | - | - | - | | 2.3529 | 1800 | 0.0457 | - | - | - | | 2.4837 | 1900 | 0.0448 | - | - | - | | 2.6144 | 2000 | 0.0457 | - | - | - | | 2.7451 | 2100 | 0.0443 | - | - | - | | 2.8758 | 2200 | 0.042 | - | - | - | | 3.0 | 2295 | - | 0.0783 | 0.3747 | - | | -1 | -1 | - | - | - | 0.3849 | ### Framework Versions - Python: 3.12.3 - Sentence Transformers: 5.0.0 - Transformers: 4.51.3 - PyTorch: 2.6.0+cu124 - Accelerate: 1.6.0 - Datasets: 2.16.0 - Tokenizers: 0.21.0 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### SpladeLoss ```bibtex @misc{formal2022distillationhardnegativesampling, title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective}, author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant}, year={2022}, eprint={2205.04733}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2205.04733}, } ``` #### FlopsLoss ```bibtex @article{paria2020minimizing, title={Minimizing flops to learn efficient sparse representations}, author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s}, journal={arXiv preprint arXiv:2004.05665}, year={2020} } ```