--- language: - en tags: - sentence-transformers - cross-encoder - reranker - generated_from_trainer - dataset_size:9000 - loss:ADRMSELoss base_model: microsoft/MiniLM-L12-H384-uncased datasets: - sentence-transformers/msmarco pipeline_tag: text-ranking library_name: sentence-transformers metrics: - map - mrr@10 - ndcg@10 model-index: - name: CrossEncoder based on microsoft/MiniLM-L12-H384-uncased results: - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoMSMARCO R100 type: NanoMSMARCO_R100 metrics: - type: map value: 0.5412 name: Map - type: mrr@10 value: 0.5341 name: Mrr@10 - type: ndcg@10 value: 0.6073 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoNFCorpus R100 type: NanoNFCorpus_R100 metrics: - type: map value: 0.36 name: Map - type: mrr@10 value: 0.6053 name: Mrr@10 - type: ndcg@10 value: 0.4303 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoNQ R100 type: NanoNQ_R100 metrics: - type: map value: 0.5805 name: Map - type: mrr@10 value: 0.595 name: Mrr@10 - type: ndcg@10 value: 0.65 name: Ndcg@10 - task: type: cross-encoder-nano-beir name: Cross Encoder Nano BEIR dataset: name: NanoBEIR R100 mean type: NanoBEIR_R100_mean metrics: - type: map value: 0.4939 name: Map - type: mrr@10 value: 0.5781 name: Mrr@10 - type: ndcg@10 value: 0.5625 name: Ndcg@10 --- # CrossEncoder based on microsoft/MiniLM-L12-H384-uncased This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [microsoft/MiniLM-L12-H384-uncased](https://huggingface.co/microsoft/MiniLM-L12-H384-uncased) on the [msmarco](https://huggingface.co/datasets/sentence-transformers/msmarco) dataset using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search. ## Model Details ### Model Description - **Model Type:** Cross Encoder - **Base model:** [microsoft/MiniLM-L12-H384-uncased](https://huggingface.co/microsoft/MiniLM-L12-H384-uncased) - **Maximum Sequence Length:** 512 tokens - **Number of Output Labels:** 1 label - **Supported Modality:** Text - **Training Dataset:** - [msmarco](https://huggingface.co/datasets/sentence-transformers/msmarco) - **Language:** en ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder) ### Full Model Architecture ``` CrossEncoder( (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import CrossEncoder # Download from the đŸ¤— Hub model = CrossEncoder("tomaarsen/reranker-msmarco-MiniLM-L12-H384-uncased-adrmse") # Get scores for pairs of inputs pairs = [ ['herbs for lowering blood sugar', 'Ten Herbs To Lower Blood Sugar Naturally. 1. Cinnamon Ă¢\x80\x93 As little as half a teaspoon a day can help improve your insulin sensitivity and control of blood sugar. If you donĂ¢\x80\x99t enjoy sprinkling cinnamon in your daily yogurt or oatmeal, you can easily find capsules of cinnamon to make a cinnamon regimen easier. 2. Sage Ă¢\x80\x93 In one German study, blood sugar levels were reduced when sage infusions were consumed by diabetics on an empty stomach. Sage is one of many powerful herbs.'], ['herbs for lowering blood sugar', 'Ten Herbs To Lower Blood Sugar Naturally. 1. Cinnamon Ă¢\x80\x93 As little as half a teaspoon a day can help improve your insulin sensitivity and control of blood sugar. If you donĂ¢\x80\x99t enjoy sprinkling cinnamon in your daily yogurt or oatmeal, you can easily find capsules of cinnamon to make a cinnamon regimen easier.'], ['herbs for lowering blood sugar', 'Cinnamon, people who have diabetes, is commonly used to reduce blood sugar and cholesterol level in blood. Onions contain falconoid and high sulfur which if consumed two ounces daily by diabetics reduces blood sugar significantly. Garlic is a beneficial herb is another of the foods that lower blood sugar. Garlic helps to elevate production of insulin and its sensitivity. Nuts.'], ['herbs for lowering blood sugar', '#7: Ginseng. Ginseng is a popular herb that has many healing properties, but for diabetes patients, its daily use can help to lower blood sugar levels. This is because of its ability to increase the use of glucose in the cells and to reduce the carbohydrates absorption rates. In turn, this helps to lower the blood sugar levels in the body.'], ['herbs for lowering blood sugar', 'Cinnamon, people who have diabetes, is commonly used to reduce blood sugar and cholesterol level in blood. Onions contain falconoid and high sulfur which if consumed two ounces daily by diabetics reduces blood sugar significantly. Garlic is a beneficial herb is another of the foods that lower blood sugar.'], ] scores = model.predict(pairs) print(scores) # [0.7691 0.7417 0.48 0.7066 0.4656] # Or rank different texts based on similarity to a single text ranks = model.rank( 'herbs for lowering blood sugar', [ 'Ten Herbs To Lower Blood Sugar Naturally. 1. Cinnamon Ă¢\x80\x93 As little as half a teaspoon a day can help improve your insulin sensitivity and control of blood sugar. If you donĂ¢\x80\x99t enjoy sprinkling cinnamon in your daily yogurt or oatmeal, you can easily find capsules of cinnamon to make a cinnamon regimen easier. 2. Sage Ă¢\x80\x93 In one German study, blood sugar levels were reduced when sage infusions were consumed by diabetics on an empty stomach. Sage is one of many powerful herbs.', 'Ten Herbs To Lower Blood Sugar Naturally. 1. Cinnamon Ă¢\x80\x93 As little as half a teaspoon a day can help improve your insulin sensitivity and control of blood sugar. If you donĂ¢\x80\x99t enjoy sprinkling cinnamon in your daily yogurt or oatmeal, you can easily find capsules of cinnamon to make a cinnamon regimen easier.', 'Cinnamon, people who have diabetes, is commonly used to reduce blood sugar and cholesterol level in blood. Onions contain falconoid and high sulfur which if consumed two ounces daily by diabetics reduces blood sugar significantly. Garlic is a beneficial herb is another of the foods that lower blood sugar. Garlic helps to elevate production of insulin and its sensitivity. Nuts.', '#7: Ginseng. Ginseng is a popular herb that has many healing properties, but for diabetes patients, its daily use can help to lower blood sugar levels. This is because of its ability to increase the use of glucose in the cells and to reduce the carbohydrates absorption rates. In turn, this helps to lower the blood sugar levels in the body.', 'Cinnamon, people who have diabetes, is commonly used to reduce blood sugar and cholesterol level in blood. Onions contain falconoid and high sulfur which if consumed two ounces daily by diabetics reduces blood sugar significantly. Garlic is a beneficial herb is another of the foods that lower blood sugar.', ] ) # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...] ``` ## Evaluation ### Metrics #### Cross Encoder Reranking * Datasets: `NanoMSMARCO_R100`, `NanoNFCorpus_R100` and `NanoNQ_R100` * Evaluated with [CrossEncoderRerankingEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: ```json { "at_k": 10, "always_rerank_positives": true } ``` | Metric | NanoMSMARCO_R100 | NanoNFCorpus_R100 | NanoNQ_R100 | |:------------|:---------------------|:---------------------|:---------------------| | map | 0.5412 (+0.0516) | 0.3600 (+0.0990) | 0.5805 (+0.1609) | | mrr@10 | 0.5341 (+0.0566) | 0.6053 (+0.1054) | 0.5950 (+0.1683) | | **ndcg@10** | **0.6073 (+0.0669)** | **0.4303 (+0.1053)** | **0.6500 (+0.1493)** | #### Cross Encoder Nano BEIR * Dataset: `NanoBEIR_R100_mean` * Evaluated with [CrossEncoderNanoBEIREvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderNanoBEIREvaluator) with these parameters: ```json { "dataset_names": [ "msmarco", "nfcorpus", "nq" ], "dataset_id": "sentence-transformers/NanoBEIR-en", "rerank_k": 100, "at_k": 10, "always_rerank_positives": true } ``` | Metric | Value | |:------------|:---------------------| | map | 0.4939 (+0.1038) | | mrr@10 | 0.5781 (+0.1101) | | **ndcg@10** | **0.5625 (+0.1071)** | ## Training Details ### Training Dataset #### msmarco * Dataset: [msmarco](https://huggingface.co/datasets/sentence-transformers/msmarco) at [2a16a97](https://huggingface.co/datasets/sentence-transformers/msmarco/tree/2a16a97b87802876f28636ca673e872248336951) * Size: 9,000 training samples * Columns: query, docs, and scores * Approximate statistics based on the first 1000 samples: | | query | docs | scores | |:--------|:---------------------------------------------------------------------------------|:------------------------------------|:------------------------------------| | type | string | list | list | | details | | | | * Samples: | query | docs | scores | |:--------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------| | what is medical d. o. | ['Medical Definition of D.O. D.O.: Abbreviation for Doctor of Osteopathy, an osteopathic physician. Osteopathy is a system of therapy founded in the 19th century based on the concept that the body can formulate its own remedies against diseases when the body is in a normal structural relationship, has a normal environment and enjoys good nutrition.', 'Doctor of Osteopathic Medicine (D.O.) is a professional doctoral degree for physicians and surgeons offered by medical schools in the United States. A D.O. degree graduate may become licensed as an osteopathic physician, having equivalent rights, privileges, and responsibilities as a physician who has earned the Doctor of Medicine (M.D.) degree. D.O. physicians are licensed to practice the full scope of medicine and surgery in sixty five countries, and all fifty states in the US.', 'Doctor of Osteopathic Medicine (D.O.) is a professional doctoral degree for physicians and surgeons offered by medical schools in the United States. A D.O. degree graduate may become licensed as an osteopathic physician, having equivalent rights, privileges, and responsibilities as a physician who has earned the Doctor of Medicine (M.D.) degree. D.O. physicians are licensed to practice the full scope of medicine and surgery in sixty-five countries, and in all fifty states.', 'Doctor of Osteopathic Medicine (D.O.) is a professional doctoral degree for physicians and surgeons offered by medical schools in the United States. A D.O. degree graduate may become licensed as an osteopathic physician, having equivalent rights, privileges, and responsibilities as a physician who has earned the Doctor of Medicine (M.D.) degree.', 'A doctor of osteopathic medicine (D.O.) is a fully licensed medical doctor offering all the techniques and treatments of modern medicine with the added benefits of hands-on diagnostics and a holistic philosophy.', ...] | [20, 19, 18, 17, 16, ...] | | pending sale definition | ['Sale pending means that there is a transaction that is going through Escrow but that has not closed yet. Translation it is too late to put in an offer now. If it falls out of escrow and the seller decides if he wants to put it back on the market then you can put in your offer.', 'Answers. 1 Sale pending means that there is a transaction that is going through Escrow but that has not closed yet. Translation it is too late to put in an offer now. If it falls out of escrow and the seller decides if he wants to put it back on the market then you can put in your offer.', '· just now. 1 Sale pending means that there is a transaction that is going through Escrow but that has not closed yet. 2 Sale pending can mean anything from an offer being accepted or contracts signed. 3 It means the buyer and seller have agreed on a contract, but closing has not happened yet.', "ThereĂ¢\x80\x99s just one problem: It's sale pending. But what does sale pending mean exactly? Are you too late or do you still have a shot? The short answer: If a home you love is pending sale, donĂ¢\x80\x99t give up hope. What does sale pending mean? A pending sale means the seller has an offer but hasn't closed yet. (This is different from a contingent sale.) A property is placed in pending status the minute a contract is executed.", "What does pending mean in real estate? Definition of Pending A home's status is changed to Pending or Pending Offer once it's set to close and all contingencies have been satisfied or waived. This is when the lender and the escrow agent or attorney process the loan and title documents to make sure the deal closes on time.", ...] | [20, 19, 18, 17, 16, ...] | | diseases and symptoms list a-z | ['Diseases A to Z. Find disease information including general information, medical symptoms and treatments. Disease information also contains links to the primary and secondary symptoms of the disease. A : B : C : D : E : F : G : H : I : J : K : L : M : N : O : P : Q : R : S : T : U : V : W : X : Y : Z.', 'The A to Z index of medical diseases comprises links to topics with information about that particular health condition. The information included in these topics comprises symptoms, causes, diagnosis, prevention, and treatment measures for the respective condition.', 'A to Z List of Medical Diseases, Disorders and Medical Conditions. Below is an alphabetical list of diseases with information on a wide variety of common medical conditions, diseases, disorders, syndromes, illnesses, and injuries.', "Diseases & Conditions A-Z List. Search by Letter: Click Here ». Find relevant and reliable medical information on diseases and conditions. Find your medical topic by using the comprehensive A-Z list above, click on the browse health centers below or one of our categorized listings of health and medical conditions. 1 Allergies. Alzheimer's.", 'Find relevant and reliable medical information on diseases and conditions. Find your medical topic by using the comprehensive A-Z list above, click on the browse health centers below or one of our categorized listings of health and medical conditions.', ...] | [20, 19, 18, 17, 16, ...] | * Loss: [ADRMSELoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#adrmseloss) with these parameters: ```json { "alpha": 1.0, "activation_fn": "torch.nn.modules.linear.Identity", "mini_batch_size": 16 } ``` ### Evaluation Dataset #### msmarco * Dataset: [msmarco](https://huggingface.co/datasets/sentence-transformers/msmarco) at [2a16a97](https://huggingface.co/datasets/sentence-transformers/msmarco/tree/2a16a97b87802876f28636ca673e872248336951) * Size: 1,000 evaluation samples * Columns: query, docs, and scores * Approximate statistics based on the first 1000 samples: | | query | docs | scores | |:--------|:---------------------------------------------------------------------------------|:------------------------------------|:------------------------------------| | type | string | list | list | | details | | | | * Samples: | query | docs | scores | |:---------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------| | herbs for lowering blood sugar | ['Ten Herbs To Lower Blood Sugar Naturally. 1. Cinnamon Ă¢\x80\x93 As little as half a teaspoon a day can help improve your insulin sensitivity and control of blood sugar. If you donĂ¢\x80\x99t enjoy sprinkling cinnamon in your daily yogurt or oatmeal, you can easily find capsules of cinnamon to make a cinnamon regimen easier. 2. Sage Ă¢\x80\x93 In one German study, blood sugar levels were reduced when sage infusions were consumed by diabetics on an empty stomach. Sage is one of many powerful herbs.', 'Ten Herbs To Lower Blood Sugar Naturally. 1. Cinnamon Ă¢\x80\x93 As little as half a teaspoon a day can help improve your insulin sensitivity and control of blood sugar. If you donĂ¢\x80\x99t enjoy sprinkling cinnamon in your daily yogurt or oatmeal, you can easily find capsules of cinnamon to make a cinnamon regimen easier.', 'Cinnamon, people who have diabetes, is commonly used to reduce blood sugar and cholesterol level in blood. Onions contain falconoid and high sulfur which if consumed two ounces daily by diabetics reduces blood sugar significantly. Garlic is a beneficial herb is another of the foods that lower blood sugar. Garlic helps to elevate production of insulin and its sensitivity. Nuts.', '#7: Ginseng. Ginseng is a popular herb that has many healing properties, but for diabetes patients, its daily use can help to lower blood sugar levels. This is because of its ability to increase the use of glucose in the cells and to reduce the carbohydrates absorption rates. In turn, this helps to lower the blood sugar levels in the body.', 'Cinnamon, people who have diabetes, is commonly used to reduce blood sugar and cholesterol level in blood. Onions contain falconoid and high sulfur which if consumed two ounces daily by diabetics reduces blood sugar significantly. Garlic is a beneficial herb is another of the foods that lower blood sugar.', ...] | [20, 19, 18, 17, 16, ...] | | define oxygen saturation | ['Definition: Oxygen Saturation. Oxygen saturation refers to the extent to which hemoglobin is saturated with oxygen. Hemoglobin is an element in the blood that binds with oxygen to carry it through the bloodstream to the organs, tissues and cells of the body. Normal oxygen saturation is usually between 96% and 98%.', 'Oxygen saturation is a term referring to the fraction of oxygen-saturated hemoglobin relative to total hemoglobin (unsaturated + saturated) in the blood. The human body requires and regulates a very precise and specific balance of oxygen in the blood. Normal blood oxygen levels in humans are considered 95-100 percent.', 'Oxygen saturation (medicine) Oxygen saturation is a term referring to the fraction of oxygen-saturated hemoglobin relative to total hemoglobin (unsaturated + saturated) in the blood. The human body requires and regulates a very precise and specific balance of oxygen in the blood. Normal blood oxygen levels in humans are considered 95-100 percent.', 'Oxygen saturation (medicine) Oxygen saturation is a term referring to the fraction of oxygen-saturated hemoglobin relative to total hemoglobin (unsaturated + saturated) in the blood. The human body requires and regulates a very precise and specific balance of oxygen in the blood. Normal blood oxygen levels in humans are considered 95-100 percent. If the level is below 90 percent, it is considered low resulting in hypoxemia. Blood oxygen levels below 80 percent may compromise organ function, such as the brain and heart, and should be promptly addressed. Continued low oxygen levels may lead to respiratory or cardiac arrest.', 'Oxygen saturation is a term referring to the fraction of oxygen-saturated hemoglobin relative to total hemoglobin (unsaturated + saturated) in the blood. The human body requires and regulates a very precise and specific balance of oxygen in the blood. Normal blood oxygen levels in humans are considered 95-100 percent.n medicine, oxygen saturation (SO 2), commonly referred to as sats, measures the percentage of hemoglobin binding sites in the bloodstream occupied by oxygen. At low partial pressures of oxygen, most hemoglobin is deoxygenated.', ...] | [20, 19, 18, 17, 16, ...] | | what type of chemical is copper | ['Copper is a chemical element with symbol Cu (from Latin: cuprum) and atomic number 29. It is a ductile metal with very high thermal and electrical conductivity. Pure copper is soft and malleable; a freshly exposed surface has a reddish-orange color.It is used as a conductor of heat and electricity, a building material, and a constituent of various metal alloys.ts compounds are commonly encountered as copper(II) salts, which often impart blue or green colors to minerals such as azurite and turquoise and have been widely used historically as pigments. Architectural structures built with copper corrode to give green verdigris (or patina).', 'Copper is a chemical element with symbol Cu (from Latin: cuprum) and atomic number 29. It is a ductile metal with very high thermal and electrical conductivity. Pure copper is soft and malleable; a freshly exposed surface has a reddish-orange color.It is used as a conductor of heat and electricity, a building material, and a constituent of various metal alloys.ttrium barium copper oxide (YBa 2 Cu 3 O 7) consists of both Cu(II) and Cu(III) centres. Like oxide, fluoride is a highly basic anion and is known to stabilize metal ions in high oxidation states. Indeed, both copper(III) and even copper(IV) fluorides are known, K3CuF6 and Cs2CuF6, respectively.', 'When it joins with other atoms, copper behaves chemically in two quite different ways to form compounds that are either described as copper (I), also known as cuprous, or copper (II), also known as cupric.The cupric compounds are more stable; cuprous ones generally turn into cupric ones.The two most important copper compounds are copper (II) sulphate, which is bright blue and used in agriculture and medicine, and copper (II) chloride, which is used as a wood preservative and in the printing and dyeing industries.he cupric compounds are more stable; cuprous ones generally turn into cupric ones. The two most important copper compounds are copper (II) sulphate, which is bright blue and used in agriculture and medicine, and copper (II) chloride, which is used as a wood preservative and in the printing and dyeing industries.', 'Copper is a chemical element. It is the 29th element in the periodic table and has 29 protons. Its mass number is 63.55. It is a transition metal in the middle of the periodic table. The symbol for copper is Cu, which comes from the Latin word cuprum, which, in turn, came from the Latin word for the island of Cyprus, where copper was found.', 'Copper is a chemical element. It is the 29th element in the periodic table and has 29 protons. Its mass number is 63.55. It is a transition metal in the middle of the periodic table.', ...] | [20, 19, 18, 17, 16, ...] | * Loss: [ADRMSELoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#adrmseloss) with these parameters: ```json { "alpha": 1.0, "activation_fn": "torch.nn.modules.linear.Identity", "mini_batch_size": 16 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `per_device_train_batch_size`: 16 - `num_train_epochs`: 1 - `learning_rate`: 2e-05 - `warmup_steps`: 0.1 - `bf16`: True - `per_device_eval_batch_size`: 16 - `load_best_model_at_end`: True - `seed`: 12 #### All Hyperparameters
Click to expand - `per_device_train_batch_size`: 16 - `num_train_epochs`: 1 - `max_steps`: -1 - `learning_rate`: 2e-05 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: None - `warmup_steps`: 0.1 - `optim`: adamw_torch_fused - `optim_args`: None - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `optim_target_modules`: None - `gradient_accumulation_steps`: 1 - `average_tokens_across_devices`: True - `max_grad_norm`: 1.0 - `label_smoothing_factor`: 0.0 - `bf16`: True - `fp16`: False - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `use_liger_kernel`: False - `liger_kernel_config`: None - `use_cache`: False - `neftune_noise_alpha`: None - `torch_empty_cache_steps`: None - `auto_find_batch_size`: False - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `include_num_input_tokens_seen`: no - `log_level`: passive - `log_level_replica`: warning - `disable_tqdm`: False - `project`: huggingface - `trackio_space_id`: trackio - `per_device_eval_batch_size`: 16 - `prediction_loss_only`: True - `eval_on_start`: False - `eval_do_concat_batches`: True - `eval_use_gather_object`: False - `eval_accumulation_steps`: None - `include_for_metrics`: [] - `batch_eval_metrics`: False - `save_only_model`: False - `save_on_each_node`: False - `enable_jit_checkpoint`: False - `push_to_hub`: False - `hub_private_repo`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_always_push`: False - `hub_revision`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `restore_callback_states_from_checkpoint`: False - `full_determinism`: False - `seed`: 12 - `data_seed`: None - `use_cpu`: False - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `dataloader_prefetch_factor`: None - `remove_unused_columns`: True - `label_names`: None - `train_sampling_strategy`: random - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `ddp_backend`: None - `ddp_timeout`: 1800 - `fsdp`: [] - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `deepspeed`: None - `debug`: [] - `skip_memory_metrics`: True - `do_predict`: False - `resume_from_checkpoint`: None - `warmup_ratio`: None - `local_rank`: -1 - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_R100_ndcg@10 | NanoNFCorpus_R100_ndcg@10 | NanoNQ_R100_ndcg@10 | NanoBEIR_R100_mean_ndcg@10 | |:-------:|:-------:|:-------------:|:---------------:|:------------------------:|:-------------------------:|:--------------------:|:--------------------------:| | -1 | -1 | - | - | 0.0300 (-0.5104) | 0.2528 (-0.0723) | 0.0168 (-0.4839) | 0.0999 (-0.3555) | | 0.0018 | 1 | 14.3181 | - | - | - | - | - | | 0.0515 | 29 | 14.3139 | - | - | - | - | - | | 0.1030 | 58 | 14.3144 | - | - | - | - | - | | 0.1545 | 87 | 14.2601 | - | - | - | - | - | | 0.2007 | 113 | - | 12.6358 | 0.4474 (-0.0930) | 0.3621 (+0.0371) | 0.5767 (+0.0761) | 0.4621 (+0.0067) | | 0.2060 | 116 | 13.2220 | - | - | - | - | - | | 0.2575 | 145 | 12.7249 | - | - | - | - | - | | 0.3091 | 174 | 12.5317 | - | - | - | - | - | | 0.3606 | 203 | 11.9536 | - | - | - | - | - | | 0.4014 | 226 | - | 11.6810 | 0.6019 (+0.0615) | 0.4339 (+0.1088) | 0.6231 (+0.1225) | 0.5530 (+0.0976) | | 0.4121 | 232 | 11.7751 | - | - | - | - | - | | 0.4636 | 261 | 11.9774 | - | - | - | - | - | | 0.5151 | 290 | 11.4589 | - | - | - | - | - | | 0.5666 | 319 | 11.7365 | - | - | - | - | - | | 0.6021 | 339 | - | 11.0220 | 0.6192 (+0.0788) | 0.4210 (+0.0960) | 0.6315 (+0.1308) | 0.5572 (+0.1019) | | 0.6181 | 348 | 11.4237 | - | - | - | - | - | | 0.6696 | 377 | 11.4196 | - | - | - | - | - | | 0.7211 | 406 | 11.2971 | - | - | - | - | - | | 0.7726 | 435 | 11.2319 | - | - | - | - | - | | 0.8028 | 452 | - | 10.8140 | 0.6179 (+0.0774) | 0.4177 (+0.0927) | 0.6458 (+0.1452) | 0.5605 (+0.1051) | | 0.8242 | 464 | 11.4555 | - | - | - | - | - | | 0.8757 | 493 | 11.2641 | - | - | - | - | - | | 0.9272 | 522 | 11.1994 | - | - | - | - | - | | 0.9787 | 551 | 11.1511 | - | - | - | - | - | | **1.0** | **563** | **-** | **10.7337** | **0.6073 (+0.0669)** | **0.4303 (+0.1053)** | **0.6500 (+0.1493)** | **0.5625 (+0.1071)** | | -1 | -1 | - | - | 0.6073 (+0.0669) | 0.4303 (+0.1053) | 0.6500 (+0.1493) | 0.5625 (+0.1071) | * The bold row denotes the saved checkpoint. ### Training Time - **Training**: 11.3 minutes - **Evaluation**: 5.0 minutes - **Total**: 16.4 minutes ### Framework Versions - Python: 3.11.6 - Sentence Transformers: 5.5.0.dev0 - Transformers: 5.5.0 - PyTorch: 2.10.0+cu128 - Accelerate: 1.13.0.dev0 - Datasets: 4.8.4 - Tokenizers: 0.22.2 ## Additional Resources - [Training and Finetuning Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-reranker): the end-to-end guide for training or finetuning Cross Encoder (reranker) models. - [Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/multimodal-sentence-transformers): use text, image, audio, and video reranker models through the same API. - [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers): training multimodal Cross Encoders, including Any-to-Any and Feature Extraction architectures. ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### ADRMSELoss ```bibtex @inproceedings{schlatt2025rankdistillm, title={Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-ranking}, author={Schlatt, Ferdinand and Fröbe, Maik and Scells, Harrisen and Zhuang, Shengyao and Koopman, Bevan and Zuccon, Guido and Stein, Benno and Potthast, Martin and Hagen, Matthias}, booktitle={Advances in Information Retrieval (ECIR 2025)}, year={2025}, doi={10.1007/978-3-031-88714-7_31}, } ```