| --- |
| language: |
| - en |
| tags: |
| - sentence-transformers |
| - cross-encoder |
| - reranker |
| - generated_from_trainer |
| - dataset_size:654438 |
| - loss:BinaryCrossEntropyLoss |
| base_model: jhu-clsp/ettin-encoder-68m |
| datasets: |
| - microsoft/ms_marco |
| pipeline_tag: text-ranking |
| library_name: sentence-transformers |
| metrics: |
| - map |
| - mrr@10 |
| - ndcg@10 |
| model-index: |
| - name: CrossEncoder based on jhu-clsp/ettin-encoder-68m |
| results: |
| - task: |
| type: cross-encoder-reranking |
| name: Cross Encoder Reranking |
| dataset: |
| name: NanoMSMARCO R100 |
| type: NanoMSMARCO_R100 |
| metrics: |
| - type: map |
| value: 0.4957 |
| name: Map |
| - type: mrr@10 |
| value: 0.4842 |
| name: Mrr@10 |
| - type: ndcg@10 |
| value: 0.5462 |
| name: Ndcg@10 |
| - task: |
| type: cross-encoder-reranking |
| name: Cross Encoder Reranking |
| dataset: |
| name: NanoNFCorpus R100 |
| type: NanoNFCorpus_R100 |
| metrics: |
| - type: map |
| value: 0.3315 |
| name: Map |
| - type: mrr@10 |
| value: 0.4747 |
| name: Mrr@10 |
| - type: ndcg@10 |
| value: 0.3291 |
| name: Ndcg@10 |
| - task: |
| type: cross-encoder-reranking |
| name: Cross Encoder Reranking |
| dataset: |
| name: NanoNQ R100 |
| type: NanoNQ_R100 |
| metrics: |
| - type: map |
| value: 0.4563 |
| name: Map |
| - type: mrr@10 |
| value: 0.4704 |
| name: Mrr@10 |
| - type: ndcg@10 |
| value: 0.531 |
| name: Ndcg@10 |
| - task: |
| type: cross-encoder-nano-beir |
| name: Cross Encoder Nano BEIR |
| dataset: |
| name: NanoBEIR R100 mean |
| type: NanoBEIR_R100_mean |
| metrics: |
| - type: map |
| value: 0.4279 |
| name: Map |
| - type: mrr@10 |
| value: 0.4764 |
| name: Mrr@10 |
| - type: ndcg@10 |
| value: 0.4688 |
| name: Ndcg@10 |
| --- |
| |
| # CrossEncoder based on jhu-clsp/ettin-encoder-68m |
|
|
| This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [jhu-clsp/ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) on the [ms_marco](https://huggingface.co/datasets/microsoft/ms_marco) dataset using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search. |
|
|
| ## Model Details |
|
|
| ### Model Description |
| - **Model Type:** Cross Encoder |
| - **Base model:** [jhu-clsp/ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) <!-- at revision ac19ae4bc51093b31c475665ac872a936d056cc2 --> |
| - **Maximum Sequence Length:** 7999 tokens |
| - **Number of Output Labels:** 1 label |
| - **Training Dataset:** |
| - [ms_marco](https://huggingface.co/datasets/microsoft/ms_marco) |
| - **Language:** en |
| <!-- - **License:** Unknown --> |
|
|
| ### Model Sources |
|
|
| - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) |
| - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html) |
| - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) |
| - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder) |
|
|
| ## Usage |
|
|
| ### Direct Usage (Sentence Transformers) |
|
|
| First install the Sentence Transformers library: |
|
|
| ```bash |
| pip install -U sentence-transformers |
| ``` |
|
|
| Then you can load this model and run inference. |
| ```python |
| from sentence_transformers import CrossEncoder |
| |
| # Download from the 🤗 Hub |
| model = CrossEncoder("bansalaman18/reranker-msmarco-v1.1-ettin-encoder-68m-bce") |
| # Get scores for pairs of texts |
| pairs = [ |
| ['how to put word count on word', 'To insert a word count into a Word 2013 document, place the cursor where you would like the word count to appear (say in the Header or Footer) and then: 1 click the Insert tab. 2 click the Quick Parts icon (towards the right hand end of the toolbar). 3 on the drop down that appears, select Field...'], |
| ['what is the difference between discipleship and evangelism', 'Discipleship, on the other hand, meant helping someone who was already a believer walk out the life of faith. The word “discipleship” brought to my mind a small group Bible study, a conversation across the table with another woman, or an accountability group. And I knew which one I preferred. As a result, the discipleship I offered others contained a lot of good information but lacked the transforming power that can only come from the gospel. (I was also, simply, a coward.). I am beginning to see that evangelism and discipleship are not all that different.'], |
| ['what metal is a trophy made from', 'The trophy stands 36.5 centimetres (14.4 inches) tall and is made of 5 kg (11 lb) of 18 carat (75%) gold with a base (13 centimetres [5.1 inches] in diameter) containing t … wo layers of malachite. Making the world better, one answer at a time. Trophies can be made out of anything you want. however, aluminum is a very reliable and trustworthy metal and it.......... oh crap.......... i have to do a poo...'], |
| ['how do you define what a cult is?', 'The term cult has been misused. The word cult comes from the French cult which is from the Latin word cultus (care/adoration) and Latin Colere (to cultivate.) So, we can plant seeds of good or bad. You can have political cults such as sit ins during the Vietnam War. A good cult could be a religious one, yet some Christians will consider Jehovah Witness a cult and have labeled them as preying on the weak. When someone labels such a thing it is usually because of the lack of understanding. Good cults are usually a small group of people that can have a cult in most anything.'], |
| ['where is silchar', 'Silchar (/ˈsɪlˌʧə/ or /ˈʃɪlˌʧə/) (Bengali: শিলচর Shilchor) shilchôr is the headquarters Of cachar district in the state Of assam In. India it is 343 (kilometres 213) mi south east Of. Guwahati it is the-second largest city of the state in terms of population and municipal. area 1 The Bhubaneshwar temple is about 50 km from Silchar and is on the top the Bhuvan hill. 2 This is a place of pilgrimage and during the festival of Shivaratri, thousand of Shivayats march towards the hilltop to worship Lord Shiva.'], |
| ] |
| scores = model.predict(pairs) |
| print(scores.shape) |
| # (5,) |
| |
| # Or rank different texts based on similarity to a single text |
| ranks = model.rank( |
| 'how to put word count on word', |
| [ |
| 'To insert a word count into a Word 2013 document, place the cursor where you would like the word count to appear (say in the Header or Footer) and then: 1 click the Insert tab. 2 click the Quick Parts icon (towards the right hand end of the toolbar). 3 on the drop down that appears, select Field...', |
| 'Discipleship, on the other hand, meant helping someone who was already a believer walk out the life of faith. The word “discipleship” brought to my mind a small group Bible study, a conversation across the table with another woman, or an accountability group. And I knew which one I preferred. As a result, the discipleship I offered others contained a lot of good information but lacked the transforming power that can only come from the gospel. (I was also, simply, a coward.). I am beginning to see that evangelism and discipleship are not all that different.', |
| 'The trophy stands 36.5 centimetres (14.4 inches) tall and is made of 5 kg (11 lb) of 18 carat (75%) gold with a base (13 centimetres [5.1 inches] in diameter) containing t … wo layers of malachite. Making the world better, one answer at a time. Trophies can be made out of anything you want. however, aluminum is a very reliable and trustworthy metal and it.......... oh crap.......... i have to do a poo...', |
| 'The term cult has been misused. The word cult comes from the French cult which is from the Latin word cultus (care/adoration) and Latin Colere (to cultivate.) So, we can plant seeds of good or bad. You can have political cults such as sit ins during the Vietnam War. A good cult could be a religious one, yet some Christians will consider Jehovah Witness a cult and have labeled them as preying on the weak. When someone labels such a thing it is usually because of the lack of understanding. Good cults are usually a small group of people that can have a cult in most anything.', |
| 'Silchar (/ˈsɪlˌʧə/ or /ˈʃɪlˌʧə/) (Bengali: শিলচর Shilchor) shilchôr is the headquarters Of cachar district in the state Of assam In. India it is 343 (kilometres 213) mi south east Of. Guwahati it is the-second largest city of the state in terms of population and municipal. area 1 The Bhubaneshwar temple is about 50 km from Silchar and is on the top the Bhuvan hill. 2 This is a place of pilgrimage and during the festival of Shivaratri, thousand of Shivayats march towards the hilltop to worship Lord Shiva.', |
| ] |
| ) |
| # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...] |
| ``` |
|
|
| <!-- |
| ### Direct Usage (Transformers) |
|
|
| <details><summary>Click to see the direct usage in Transformers</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Downstream Usage (Sentence Transformers) |
|
|
| You can finetune this model on your own dataset. |
|
|
| <details><summary>Click to expand</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Out-of-Scope Use |
|
|
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* |
| --> |
|
|
| ## Evaluation |
|
|
| ### Metrics |
|
|
| #### Cross Encoder Reranking |
|
|
| * Datasets: `NanoMSMARCO_R100`, `NanoNFCorpus_R100` and `NanoNQ_R100` |
| * Evaluated with [<code>CrossEncoderRerankingEvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: |
| ```json |
| { |
| "at_k": 10, |
| "always_rerank_positives": true |
| } |
| ``` |
|
|
| | Metric | NanoMSMARCO_R100 | NanoNFCorpus_R100 | NanoNQ_R100 | |
| |:------------|:---------------------|:---------------------|:---------------------| |
| | map | 0.4957 (+0.0061) | 0.3315 (+0.0706) | 0.4563 (+0.0367) | |
| | mrr@10 | 0.4842 (+0.0067) | 0.4747 (-0.0251) | 0.4704 (+0.0437) | |
| | **ndcg@10** | **0.5462 (+0.0058)** | **0.3291 (+0.0040)** | **0.5310 (+0.0304)** | |
| |
| #### Cross Encoder Nano BEIR |
| |
| * Dataset: `NanoBEIR_R100_mean` |
| * Evaluated with [<code>CrossEncoderNanoBEIREvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderNanoBEIREvaluator) with these parameters: |
| ```json |
| { |
| "dataset_names": [ |
| "msmarco", |
| "nfcorpus", |
| "nq" |
| ], |
| "rerank_k": 100, |
| "at_k": 10, |
| "always_rerank_positives": true |
| } |
| ``` |
| |
| | Metric | Value | |
| |:------------|:---------------------| |
| | map | 0.4279 (+0.0378) | |
| | mrr@10 | 0.4764 (+0.0084) | |
| | **ndcg@10** | **0.4688 (+0.0134)** | |
|
|
| <!-- |
| ## Bias, Risks and Limitations |
|
|
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
| --> |
|
|
| <!-- |
| ### Recommendations |
|
|
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
| --> |
|
|
| ## Training Details |
|
|
| ### Training Dataset |
|
|
| #### ms_marco |
| |
| * Dataset: [ms_marco](https://huggingface.co/datasets/microsoft/ms_marco) at [a47ee7a](https://huggingface.co/datasets/microsoft/ms_marco/tree/a47ee7aae8d7d466ba15f9f0bfac3b3681087b3a) |
| * Size: 654,438 training samples |
| * Columns: <code>query</code>, <code>response</code>, and <code>label</code> |
| * Approximate statistics based on the first 1000 samples: |
| | | query | response | label | |
| |:--------|:----------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:---------------------------------------------------------------| |
| | type | string | string | float | |
| | details | <ul><li>min: 11 characters</li><li>mean: 34.1 characters</li><li>max: 95 characters</li></ul> | <ul><li>min: 62 characters</li><li>mean: 417.26 characters</li><li>max: 939 characters</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.13</li><li>max: 1.0</li></ul> | |
| * Samples: |
| | query | response | label | |
| |:-----------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------| |
| | <code>what is a polyhedron definition</code> | <code>A polyhedron is said to be convex if its surface (comprising its faces, edges and vertices) does not intersect itself and the line segment joining any two points of the polyhedron is contained in the interior or surface. A polyhedron is a 3-dimensional example of the more general polytope in any number of dimensions. Polyhedra with congruent regular faces of six or more sides are all non-convex, because the vertex of three regular hexagons defines a plane. The total number of convex polyhedra with equal regular faces is thus ten, comprising the five Platonic solids and the five non-uniform deltahedra.</code> | <code>0.0</code> | |
| | <code>what can you carry in hand luggage on easyjet</code> | <code>Each passenger who pays for a hold bag can take up to 20kg of luggage. This weight allowance applies to the passenger rather than to the bag so purchasing extra bags is possible but will not increase the weight allowance.</code> | <code>0.0</code> | |
| | <code>what is dynamic segmentation in gis</code> | <code>The result of the dynamic segmentation process is a dynamic feature class known as a route event source. A route event source can serve as the data source of a feature layer in ArcMap. For the most part, a dynamic feature layer behaves like any other feature layer. Event locating errors. The dynamic segmentation process creates a shape for each row in the input route event table. In some cases, however, the shape of the event feature might be empty. This happens when there is a reason that the event can't be properly located.</code> | <code>0.0</code> | |
| * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters: |
| ```json |
| { |
| "activation_fn": "torch.nn.modules.linear.Identity", |
| "pos_weight": null |
| } |
| ``` |
| |
| ### Evaluation Dataset |
|
|
| #### ms_marco |
| |
| * Dataset: [ms_marco](https://huggingface.co/datasets/microsoft/ms_marco) at [a47ee7a](https://huggingface.co/datasets/microsoft/ms_marco/tree/a47ee7aae8d7d466ba15f9f0bfac3b3681087b3a) |
| * Size: 1,000 evaluation samples |
| * Columns: <code>query</code>, <code>response</code>, and <code>label</code> |
| * Approximate statistics based on the first 1000 samples: |
| | | query | response | label | |
| |:--------|:------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------|:---------------------------------------------------------------| |
| | type | string | string | float | |
| | details | <ul><li>min: 10 characters</li><li>mean: 33.73 characters</li><li>max: 117 characters</li></ul> | <ul><li>min: 57 characters</li><li>mean: 412.4 characters</li><li>max: 918 characters</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.13</li><li>max: 1.0</li></ul> | |
| * Samples: |
| | query | response | label | |
| |:------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------| |
| | <code>how to put word count on word</code> | <code>To insert a word count into a Word 2013 document, place the cursor where you would like the word count to appear (say in the Header or Footer) and then: 1 click the Insert tab. 2 click the Quick Parts icon (towards the right hand end of the toolbar). 3 on the drop down that appears, select Field...</code> | <code>0.0</code> | |
| | <code>what is the difference between discipleship and evangelism</code> | <code>Discipleship, on the other hand, meant helping someone who was already a believer walk out the life of faith. The word “discipleship” brought to my mind a small group Bible study, a conversation across the table with another woman, or an accountability group. And I knew which one I preferred. As a result, the discipleship I offered others contained a lot of good information but lacked the transforming power that can only come from the gospel. (I was also, simply, a coward.). I am beginning to see that evangelism and discipleship are not all that different.</code> | <code>0.0</code> | |
| | <code>what metal is a trophy made from</code> | <code>The trophy stands 36.5 centimetres (14.4 inches) tall and is made of 5 kg (11 lb) of 18 carat (75%) gold with a base (13 centimetres [5.1 inches] in diameter) containing t … wo layers of malachite. Making the world better, one answer at a time. Trophies can be made out of anything you want. however, aluminum is a very reliable and trustworthy metal and it.......... oh crap.......... i have to do a poo...</code> | <code>1.0</code> | |
| * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters: |
| ```json |
| { |
| "activation_fn": "torch.nn.modules.linear.Identity", |
| "pos_weight": null |
| } |
| ``` |
| |
| ### Training Hyperparameters |
| #### Non-Default Hyperparameters |
|
|
| - `eval_strategy`: steps |
| - `per_device_train_batch_size`: 128 |
| - `per_device_eval_batch_size`: 128 |
| - `learning_rate`: 2e-05 |
| - `num_train_epochs`: 1 |
| - `warmup_ratio`: 0.1 |
| - `seed`: 12 |
| - `bf16`: True |
| - `remove_unused_columns`: False |
| - `load_best_model_at_end`: True |
|
|
| #### All Hyperparameters |
| <details><summary>Click to expand</summary> |
|
|
| - `overwrite_output_dir`: False |
| - `do_predict`: False |
| - `eval_strategy`: steps |
| - `prediction_loss_only`: True |
| - `per_device_train_batch_size`: 128 |
| - `per_device_eval_batch_size`: 128 |
| - `per_gpu_train_batch_size`: None |
| - `per_gpu_eval_batch_size`: None |
| - `gradient_accumulation_steps`: 1 |
| - `eval_accumulation_steps`: None |
| - `torch_empty_cache_steps`: None |
| - `learning_rate`: 2e-05 |
| - `weight_decay`: 0.0 |
| - `adam_beta1`: 0.9 |
| - `adam_beta2`: 0.999 |
| - `adam_epsilon`: 1e-08 |
| - `max_grad_norm`: 1.0 |
| - `num_train_epochs`: 1 |
| - `max_steps`: -1 |
| - `lr_scheduler_type`: linear |
| - `lr_scheduler_kwargs`: {} |
| - `warmup_ratio`: 0.1 |
| - `warmup_steps`: 0 |
| - `log_level`: passive |
| - `log_level_replica`: warning |
| - `log_on_each_node`: True |
| - `logging_nan_inf_filter`: True |
| - `save_safetensors`: True |
| - `save_on_each_node`: False |
| - `save_only_model`: False |
| - `restore_callback_states_from_checkpoint`: False |
| - `no_cuda`: False |
| - `use_cpu`: False |
| - `use_mps_device`: False |
| - `seed`: 12 |
| - `data_seed`: None |
| - `jit_mode_eval`: False |
| - `use_ipex`: False |
| - `bf16`: True |
| - `fp16`: False |
| - `fp16_opt_level`: O1 |
| - `half_precision_backend`: auto |
| - `bf16_full_eval`: False |
| - `fp16_full_eval`: False |
| - `tf32`: None |
| - `local_rank`: 0 |
| - `ddp_backend`: None |
| - `tpu_num_cores`: None |
| - `tpu_metrics_debug`: False |
| - `debug`: [] |
| - `dataloader_drop_last`: False |
| - `dataloader_num_workers`: 0 |
| - `dataloader_prefetch_factor`: None |
| - `past_index`: -1 |
| - `disable_tqdm`: False |
| - `remove_unused_columns`: False |
| - `label_names`: None |
| - `load_best_model_at_end`: True |
| - `ignore_data_skip`: False |
| - `fsdp`: [] |
| - `fsdp_min_num_params`: 0 |
| - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} |
| - `tp_size`: 0 |
| - `fsdp_transformer_layer_cls_to_wrap`: None |
| - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
| - `deepspeed`: None |
| - `label_smoothing_factor`: 0.0 |
| - `optim`: adamw_torch |
| - `optim_args`: None |
| - `adafactor`: False |
| - `group_by_length`: False |
| - `length_column_name`: length |
| - `ddp_find_unused_parameters`: None |
| - `ddp_bucket_cap_mb`: None |
| - `ddp_broadcast_buffers`: False |
| - `dataloader_pin_memory`: True |
| - `dataloader_persistent_workers`: False |
| - `skip_memory_metrics`: True |
| - `use_legacy_prediction_loop`: False |
| - `push_to_hub`: False |
| - `resume_from_checkpoint`: None |
| - `hub_model_id`: None |
| - `hub_strategy`: every_save |
| - `hub_private_repo`: None |
| - `hub_always_push`: False |
| - `gradient_checkpointing`: False |
| - `gradient_checkpointing_kwargs`: None |
| - `include_inputs_for_metrics`: False |
| - `include_for_metrics`: [] |
| - `eval_do_concat_batches`: True |
| - `fp16_backend`: auto |
| - `push_to_hub_model_id`: None |
| - `push_to_hub_organization`: None |
| - `mp_parameters`: |
| - `auto_find_batch_size`: False |
| - `full_determinism`: False |
| - `torchdynamo`: None |
| - `ray_scope`: last |
| - `ddp_timeout`: 1800 |
| - `torch_compile`: False |
| - `torch_compile_backend`: None |
| - `torch_compile_mode`: None |
| - `include_tokens_per_second`: False |
| - `include_num_input_tokens_seen`: False |
| - `neftune_noise_alpha`: None |
| - `optim_target_modules`: None |
| - `batch_eval_metrics`: False |
| - `eval_on_start`: False |
| - `use_liger_kernel`: False |
| - `eval_use_gather_object`: False |
| - `average_tokens_across_devices`: False |
| - `prompts`: None |
| - `batch_sampler`: batch_sampler |
| - `multi_dataset_batch_sampler`: proportional |
| - `router_mapping`: {} |
| - `learning_rate_mapping`: {} |
|
|
| </details> |
|
|
| ### Training Logs |
| | Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_R100_ndcg@10 | NanoNFCorpus_R100_ndcg@10 | NanoNQ_R100_ndcg@10 | NanoBEIR_R100_mean_ndcg@10 | |
| |:----------:|:--------:|:-------------:|:---------------:|:------------------------:|:-------------------------:|:--------------------:|:--------------------------:| |
| | -1 | -1 | - | - | 0.0442 (-0.4962) | 0.2555 (-0.0695) | 0.0464 (-0.4542) | 0.1154 (-0.3400) | |
| | 0.0002 | 1 | 2.1556 | - | - | - | - | - | |
| | 0.0196 | 100 | 0.84 | 0.3976 | 0.0474 (-0.4930) | 0.2826 (-0.0424) | 0.0395 (-0.4612) | 0.1232 (-0.3322) | |
| | 0.0391 | 200 | 0.402 | 0.3895 | 0.0508 (-0.4896) | 0.2367 (-0.0883) | 0.0673 (-0.4334) | 0.1183 (-0.3371) | |
| | 0.0587 | 300 | 0.3988 | 0.3854 | 0.0763 (-0.4642) | 0.2175 (-0.1076) | 0.1093 (-0.3913) | 0.1344 (-0.3210) | |
| | 0.0782 | 400 | 0.3899 | 0.3968 | 0.1461 (-0.3943) | 0.2087 (-0.1163) | 0.1574 (-0.3433) | 0.1707 (-0.2846) | |
| | 0.0978 | 500 | 0.3898 | 0.3708 | 0.2417 (-0.2987) | 0.2259 (-0.0991) | 0.2326 (-0.2680) | 0.2334 (-0.2219) | |
| | 0.1173 | 600 | 0.3783 | 0.3681 | 0.2933 (-0.2471) | 0.2769 (-0.0481) | 0.3659 (-0.1347) | 0.3120 (-0.1433) | |
| | 0.1369 | 700 | 0.3718 | 0.3610 | 0.3654 (-0.1751) | 0.2852 (-0.0398) | 0.4239 (-0.0768) | 0.3582 (-0.0972) | |
| | 0.1565 | 800 | 0.3643 | 0.3657 | 0.4721 (-0.0683) | 0.2658 (-0.0593) | 0.4801 (-0.0206) | 0.4060 (-0.0494) | |
| | 0.1760 | 900 | 0.3639 | 0.3637 | 0.4291 (-0.1113) | 0.2635 (-0.0615) | 0.4308 (-0.0699) | 0.3745 (-0.0809) | |
| | 0.1956 | 1000 | 0.3686 | 0.3518 | 0.4859 (-0.0545) | 0.3071 (-0.0179) | 0.5328 (+0.0322) | 0.4419 (-0.0134) | |
| | 0.2151 | 1100 | 0.3585 | 0.3529 | 0.4581 (-0.0823) | 0.2611 (-0.0640) | 0.5113 (+0.0107) | 0.4102 (-0.0452) | |
| | 0.2347 | 1200 | 0.3624 | 0.3522 | 0.4918 (-0.0486) | 0.3244 (-0.0006) | 0.4936 (-0.0071) | 0.4366 (-0.0188) | |
| | 0.2543 | 1300 | 0.3684 | 0.3607 | 0.3933 (-0.1471) | 0.2893 (-0.0358) | 0.4810 (-0.0196) | 0.3879 (-0.0675) | |
| | 0.2738 | 1400 | 0.3651 | 0.3437 | 0.4999 (-0.0406) | 0.3170 (-0.0081) | 0.5290 (+0.0283) | 0.4486 (-0.0068) | |
| | 0.2934 | 1500 | 0.3568 | 0.3466 | 0.5099 (-0.0306) | 0.3312 (+0.0062) | 0.4966 (-0.0040) | 0.4459 (-0.0095) | |
| | 0.3129 | 1600 | 0.36 | 0.3437 | 0.4762 (-0.0642) | 0.3488 (+0.0237) | 0.4963 (-0.0043) | 0.4404 (-0.0149) | |
| | 0.3325 | 1700 | 0.3557 | 0.3451 | 0.4572 (-0.0833) | 0.3046 (-0.0204) | 0.5173 (+0.0167) | 0.4264 (-0.0290) | |
| | 0.3520 | 1800 | 0.3529 | 0.3406 | 0.4707 (-0.0697) | 0.3180 (-0.0071) | 0.4842 (-0.0165) | 0.4243 (-0.0311) | |
| | 0.3716 | 1900 | 0.3514 | 0.3367 | 0.4901 (-0.0503) | 0.2742 (-0.0509) | 0.5322 (+0.0316) | 0.4321 (-0.0232) | |
| | 0.3912 | 2000 | 0.3499 | 0.3408 | 0.4859 (-0.0545) | 0.2814 (-0.0437) | 0.5011 (+0.0004) | 0.4228 (-0.0326) | |
| | 0.4107 | 2100 | 0.3595 | 0.3393 | 0.4821 (-0.0583) | 0.3004 (-0.0246) | 0.5499 (+0.0493) | 0.4441 (-0.0112) | |
| | 0.4303 | 2200 | 0.356 | 0.3442 | 0.4939 (-0.0465) | 0.3279 (+0.0029) | 0.5454 (+0.0448) | 0.4557 (+0.0004) | |
| | 0.4498 | 2300 | 0.3396 | 0.3351 | 0.5252 (-0.0152) | 0.3024 (-0.0226) | 0.5271 (+0.0264) | 0.4516 (-0.0038) | |
| | 0.4694 | 2400 | 0.3644 | 0.3396 | 0.5307 (-0.0098) | 0.3204 (-0.0046) | 0.5101 (+0.0094) | 0.4537 (-0.0017) | |
| | 0.4889 | 2500 | 0.3508 | 0.3371 | 0.5003 (-0.0402) | 0.3006 (-0.0245) | 0.5404 (+0.0398) | 0.4471 (-0.0083) | |
| | 0.5085 | 2600 | 0.3525 | 0.3396 | 0.5146 (-0.0258) | 0.3001 (-0.0249) | 0.5525 (+0.0518) | 0.4557 (+0.0004) | |
| | 0.5281 | 2700 | 0.3348 | 0.3393 | 0.4800 (-0.0604) | 0.2778 (-0.0472) | 0.5416 (+0.0410) | 0.4332 (-0.0222) | |
| | 0.5476 | 2800 | 0.3448 | 0.3458 | 0.5176 (-0.0229) | 0.2905 (-0.0345) | 0.5243 (+0.0236) | 0.4441 (-0.0113) | |
| | 0.5672 | 2900 | 0.3508 | 0.3379 | 0.4738 (-0.0667) | 0.2924 (-0.0326) | 0.5395 (+0.0389) | 0.4352 (-0.0201) | |
| | 0.5867 | 3000 | 0.3401 | 0.3404 | 0.5246 (-0.0158) | 0.2930 (-0.0321) | 0.5337 (+0.0330) | 0.4504 (-0.0049) | |
| | 0.6063 | 3100 | 0.3508 | 0.3383 | 0.5004 (-0.0400) | 0.2890 (-0.0360) | 0.5321 (+0.0314) | 0.4405 (-0.0149) | |
| | 0.6259 | 3200 | 0.3509 | 0.3364 | 0.5097 (-0.0308) | 0.3321 (+0.0071) | 0.5502 (+0.0496) | 0.4640 (+0.0086) | |
| | 0.6454 | 3300 | 0.3501 | 0.3369 | 0.5172 (-0.0232) | 0.3084 (-0.0167) | 0.5644 (+0.0637) | 0.4633 (+0.0080) | |
| | 0.6650 | 3400 | 0.3417 | 0.3336 | 0.4947 (-0.0457) | 0.3133 (-0.0117) | 0.5404 (+0.0397) | 0.4495 (-0.0059) | |
| | 0.6845 | 3500 | 0.3487 | 0.3335 | 0.4994 (-0.0410) | 0.3328 (+0.0078) | 0.5351 (+0.0344) | 0.4558 (+0.0004) | |
| | 0.7041 | 3600 | 0.3507 | 0.3377 | 0.5103 (-0.0301) | 0.3111 (-0.0139) | 0.5030 (+0.0023) | 0.4415 (-0.0139) | |
| | 0.7236 | 3700 | 0.34 | 0.3382 | 0.5254 (-0.0150) | 0.3320 (+0.0070) | 0.5154 (+0.0148) | 0.4576 (+0.0023) | |
| | 0.7432 | 3800 | 0.3392 | 0.3361 | 0.4892 (-0.0512) | 0.3261 (+0.0011) | 0.5268 (+0.0262) | 0.4474 (-0.0080) | |
| | 0.7628 | 3900 | 0.3511 | 0.3349 | 0.5129 (-0.0276) | 0.3307 (+0.0057) | 0.5180 (+0.0173) | 0.4539 (-0.0015) | |
| | **0.7823** | **4000** | **0.3508** | **0.3368** | **0.5462 (+0.0058)** | **0.3291 (+0.0040)** | **0.5310 (+0.0304)** | **0.4688 (+0.0134)** | |
| | 0.8019 | 4100 | 0.3439 | 0.3348 | 0.5409 (+0.0005) | 0.3307 (+0.0056) | 0.5312 (+0.0306) | 0.4676 (+0.0122) | |
| | 0.8214 | 4200 | 0.3487 | 0.3340 | 0.5324 (-0.0080) | 0.3284 (+0.0034) | 0.5191 (+0.0185) | 0.4600 (+0.0046) | |
| | 0.8410 | 4300 | 0.341 | 0.3351 | 0.5277 (-0.0127) | 0.3276 (+0.0025) | 0.5050 (+0.0044) | 0.4535 (-0.0019) | |
| | 0.8606 | 4400 | 0.3293 | 0.3344 | 0.5150 (-0.0254) | 0.3355 (+0.0105) | 0.5009 (+0.0002) | 0.4505 (-0.0049) | |
| | 0.8801 | 4500 | 0.3525 | 0.3346 | 0.5224 (-0.0180) | 0.3239 (-0.0012) | 0.5127 (+0.0121) | 0.4530 (-0.0024) | |
| | 0.8997 | 4600 | 0.3421 | 0.3331 | 0.5312 (-0.0092) | 0.3376 (+0.0126) | 0.5234 (+0.0228) | 0.4641 (+0.0087) | |
| | 0.9192 | 4700 | 0.3442 | 0.3336 | 0.5227 (-0.0177) | 0.3330 (+0.0080) | 0.5053 (+0.0047) | 0.4537 (-0.0017) | |
| | 0.9388 | 4800 | 0.3361 | 0.3328 | 0.5166 (-0.0238) | 0.3378 (+0.0128) | 0.5194 (+0.0187) | 0.4579 (+0.0026) | |
| | 0.9583 | 4900 | 0.3377 | 0.3335 | 0.5298 (-0.0106) | 0.3300 (+0.0049) | 0.5248 (+0.0241) | 0.4615 (+0.0061) | |
| | 0.9779 | 5000 | 0.3455 | 0.3332 | 0.5298 (-0.0106) | 0.3305 (+0.0055) | 0.5182 (+0.0176) | 0.4595 (+0.0041) | |
| | 0.9975 | 5100 | 0.3422 | 0.3333 | 0.5298 (-0.0106) | 0.3303 (+0.0053) | 0.5240 (+0.0234) | 0.4614 (+0.0060) | |
| | -1 | -1 | - | - | 0.5462 (+0.0058) | 0.3291 (+0.0040) | 0.5310 (+0.0304) | 0.4688 (+0.0134) | |
| |
| * The bold row denotes the saved checkpoint. |
| |
| ### Framework Versions |
| - Python: 3.11.13 |
| - Sentence Transformers: 5.0.0 |
| - Transformers: 4.51.0 |
| - PyTorch: 2.9.1+cu126 |
| - Accelerate: 1.8.1 |
| - Datasets: 3.6.0 |
| - Tokenizers: 0.21.4-dev.0 |
| |
| ## Citation |
| |
| ### BibTeX |
| |
| #### Sentence Transformers |
| ```bibtex |
| @inproceedings{reimers-2019-sentence-bert, |
| title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
| author = "Reimers, Nils and Gurevych, Iryna", |
| booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
| month = "11", |
| year = "2019", |
| publisher = "Association for Computational Linguistics", |
| url = "https://arxiv.org/abs/1908.10084", |
| } |
| ``` |
| |
| <!-- |
| ## Glossary |
| |
| *Clearly define terms in order to be accessible across audiences.* |
| --> |
| |
| <!-- |
| ## Model Card Authors |
| |
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
| --> |
| |
| <!-- |
| ## Model Card Contact |
| |
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
| --> |
