tomaarsen/reranker-msmarco-v1.1-MiniLM-L12-H384-uncased-plistmle-pr3827-baseline

CrossEncoder based on microsoft/MiniLM-L12-H384-uncased

This is a Cross Encoder model finetuned from microsoft/MiniLM-L12-H384-uncased on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: microsoft/MiniLM-L12-H384-uncased
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Supported Modality: Text
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of inputs
pairs = [
    ['are mentos halal', "Making the world better, one answer at a time. well you see i have checked to see if mentos are haram but its says it has no gelatin (used to make things thicker made from pig skin). Another thing that is haram is  eating anything that is not 'halal'. For many people, this just  means checking if there is a 'halal' label on their packets of  chicken and meat. But this is more than that. Lets just look at  poultry for a while."],
    ['are mentos halal', 'Mentos Chewy Mints are offered in several refreshing flavor varieties. Mentos Fruit has a mix of strawberry, orange and lemon. This is a must-have for any afternoon snack or a snack anytime of the day!'],
    ['are mentos halal', 'According to the following website below are Halal Mentos Mint Spearmint Fruit (Strawberry/Lemon/Orange) Rainbow (Green Apple/Raspberry/Pineapple/Watermelon/Orange/Strawberry/Grape) Duo (Blackcurrant &Lemon/Strawberry &Lime) http://www.stuartalexander.com.au/nutritional_dietary_questions_answers_fact_sheet.php#halal. '],
    ['are mentos halal', '(By a family member). The ingredients: Sugar, glucose syrup, hydrogenated vegetable oil, citric acid, fruit juices from concentrate, starch, sucrose esters of fatty acids, natural, colors (carmine, paprika oleoresin, beta-carotene, beetrood red), carnauba wax, beeswax E414, E418. I do not want to go into detail as to why it is Haram, Mufti Saab has said E473 is Haram, so for me it is Haram. If you want to go into further detail, you are welcome to study this detailed document of what (SUCROSE esters of fatty acids) is www.fao.org/ag/agn/jecfa-additives/specs/monograph10/addi...'],
    ['are mentos halal', "Haram things are haram because they don't go together with Islamic  principles. Islamic principles are in turn, a way of life and good  for us both morally and physically. T … his is best explained by  examples. For example, Muslims arent allowed to drink. Another thing that is haram is  eating anything that is not 'halal'. For many people, this just  means checking if there is a 'halal' label on their packets of  chicken and meat. But this is more than that. Lets just look at  poultry for a while."],
]
scores = model.predict(pairs)
print(scores)
# [0.9887 0.9752 0.9879 0.9748 0.9716]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'are mentos halal',
    [
        "Making the world better, one answer at a time. well you see i have checked to see if mentos are haram but its says it has no gelatin (used to make things thicker made from pig skin). Another thing that is haram is  eating anything that is not 'halal'. For many people, this just  means checking if there is a 'halal' label on their packets of  chicken and meat. But this is more than that. Lets just look at  poultry for a while.",
        'Mentos Chewy Mints are offered in several refreshing flavor varieties. Mentos Fruit has a mix of strawberry, orange and lemon. This is a must-have for any afternoon snack or a snack anytime of the day!',
        'According to the following website below are Halal Mentos Mint Spearmint Fruit (Strawberry/Lemon/Orange) Rainbow (Green Apple/Raspberry/Pineapple/Watermelon/Orange/Strawberry/Grape) Duo (Blackcurrant &Lemon/Strawberry &Lime) http://www.stuartalexander.com.au/nutritional_dietary_questions_answers_fact_sheet.php#halal. ',
        '(By a family member). The ingredients: Sugar, glucose syrup, hydrogenated vegetable oil, citric acid, fruit juices from concentrate, starch, sucrose esters of fatty acids, natural, colors (carmine, paprika oleoresin, beta-carotene, beetrood red), carnauba wax, beeswax E414, E418. I do not want to go into detail as to why it is Haram, Mufti Saab has said E473 is Haram, so for me it is Haram. If you want to go into further detail, you are welcome to study this detailed document of what (SUCROSE esters of fatty acids) is www.fao.org/ag/agn/jecfa-additives/specs/monograph10/addi...',
        "Haram things are haram because they don't go together with Islamic  principles. Islamic principles are in turn, a way of life and good  for us both morally and physically. T … his is best explained by  examples. For example, Muslims arent allowed to drink. Another thing that is haram is  eating anything that is not 'halal'. For many people, this just  means checking if there is a 'halal' label on their packets of  chicken and meat. But this is more than that. Lets just look at  poultry for a while.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	NanoMSMARCO_R100	NanoNFCorpus_R100	NanoNQ_R100
map	0.4785 (-0.0110)	0.3247 (+0.0637)	0.5619 (+0.1423)
mrr@10	0.4720 (-0.0055)	0.5696 (+0.0698)	0.5931 (+0.1664)
ndcg@10	0.5392 (-0.0012)	0.3614 (+0.0364)	0.6400 (+0.1393)

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en",
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	Value
map	0.4550 (+0.0650)
mrr@10	0.5449 (+0.0769)
ndcg@10	0.5135 (+0.0582)

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 78,704 training samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 100 samples:

  	query	docs	labels
  type	string	list	list
  modality	text
  details	min: 5 tokens mean: 9.12 tokens max: 14 tokens	min: 4 elements mean: 7.95 elements max: 10 elements	min: 4 elements mean: 7.95 elements max: 10 elements

• Samples:

  query	docs	labels
  how many breaths do we deliver to the casualty	["1 Start CPR with 30 chest compressions before checking the airway and giving rescue breaths. 2 Trained but rusty. 3 If you've previously received CPR training but you're not confident in your abilities, then just do chest compressions at a rate of about 100 a minute. 1 Push hard at a rate of about 100 compressions a minute. 2 If you haven't been trained in CPR, continue chest compressions until there are signs of movement or until emergency medical personnel take over. 3 If you have been trained in CPR, go on to checking the airway and rescue breathing.", "You need to know what to do if he or she has stopped breathing. The best thing to do is to start CPR, including rescue breathing, until help arrives. Seconds can count. If you're unsure how to perform rescue breathing, this article will focus on what to do for adults, children, and infants. Though you definitely want to call for help as quickly as possible, with a child or infant, it's important to give a 2 minute round of CPR before dialing 911, as damage can set in more quickly. 3. Increase the rescue breaths to five.", 'When talking to people about first aid and CPR in particular, I find a lot of confusion centred around the ratio of compressions to rescue breaths. That is, how many times should you press down on the chest and how many times should you give mouth to mouth breaths. This is really not too surprising, as it has been subject to change over the years. I started to learn first aid in the winter of 1971/72, at the tender age of 8. If, however, you have help, then you should give 5 compressions to 1 breath. In my 1979 book, I am told that the correct ratio is 5 to 1, with or without help, but by the second half of the eighties we are back to 15 to 2 on our own or 5 to 1 with help.', 'Pinch the nose but deliver breaths as if blowing out a candle and use just one hand to perform CPR. Do not pinch the nose but cover the mouth and nose with your mouth and deliver breaths as if blowing out a candle and use just one hand to perform CPR.', "Chest compressions and rescue breaths keep blood and oxygen circulating in the body. If someone is not breathing normally and is not moving or responding to you after an accident, call 999 or 112 for an ambulance. Then, if you can, start CPR straight away. 1 Place the heel of your hand on the centre of the person's chest, then place the other hand on top and press down by 5–6cm at a steady rate, at approximately 100 compressions per minute. 2 After every 30 chest compressions, give two breaths.", ...]	[1, 0, 0, 0, 0, ...]
  best lubricant for pole elastic	["Pole Elastic Lubricant 300ml. Product Ref: G-LUBE. Easy to use pump action silicone based elastic lubricant that is kind to your elastic. Simply pull out your bung from your top kit, spray the lube up the chamber of the pole tip, reposition the bung and it's ready to use. To keep elastic perfectly lubricated it is advised to repeat this process regularly throughout the session.", 'Elastics…. To fish the pole, you need to thread elastic through the top two or three sections to act as a shock absorber. You can be easily baffled with the wide range of elastics that are available nowadays. Elastics range from a really fine number one ‘knicker elastic’ to really heavy versions of 25+ ‘bull rope’. Like solid elastic, it’s available from the lightest number 1 right up to a 20. Many anglers favour latex as it offers a much smoother and more forgiving action than solid elastic, but doesn’t stretch as far as hollow elastic, meaning fish can be landed on the top two or three sections of pole.', 'The line is attached to the end of elastic, placed inside the pole tip and runs inside of the top few section of pole. The elastic is used to assist in fighting fish; it relieves tension from the line when the fish is on the hook. Using elastic in the pole tip helps to keep a tight line between the fish and pole tip. Usually you fit elastics 2 - 6 through the top two sections, and bigger elastics through the top three. 1. Choose your elastic and check you have a bush it easily slides through. 2. Cut the pole back by rolling a Stanley knife over it until the bush fits snuggly over the tip of the pole.', "Product Description. Easy to use pump action silicone based elastic lubricant that is kind to your elastic. Simply pull out your bung from your top kit, spray the Lube up the chamber of the pole tip, reposition the bung and it's ready to use! To keep elastics perfectly lubricated it is advised to repeat this process regularly throughout the session. 300ml bottle. Shopping Basket. 0 Products Product (empty).", 'Slick pole elastic lubricant from Drennan is a must for smooth, trouble free, fish playing. It provides a micro film which literally makes the elastic feel polished and slick. Regular use will prolong the life of your elastic and it will not build up a deposit inside the pole or leave a sticky residue. ', ...]	[1, 0, 0, 0, 0, ...]
  what do lymph glands do	['Lymph nodes and what they do. Lymph vessels route lymph fluid through nodes throughout the body. Lymph nodes are small structures that work as filters for harmful substances. They contain immune cells that can help fight infection by attacking and destroying germs that are carried in through the lymph fluid.', 'Lymph in the lymph vessels eventually reaches a lymph node -- there are about 100 nodes scattered throughout the body. Lymph nodes filter the lymph and also contain large numbers of white blood cells (a big part of the immune system), which remove foreign cells and debris from the lymph.', 'The lymph system is probably most familiar to people because doctors often check for swollen lymph nodes in the neck. The lymph nodes, however, are just one part of a system that reaches points throughout the body in much the same way blood vessels do.', 'This fluid may filter through lymph nodes at the elbow, or those under the arm. Fluid from the head, scalp, and face flows down through lymph nodes in the neck. Some lymph nodes are deep inside the body, such as between the lungs or around the bowel, to filter fluid in those areas.', "Lymph nodes are small, bean-shaped glands throughout the body. They are part of the lymph system, which carries fluid (lymph fluid), nutrients, and waste material between the body tissues and the bloodstream. The lymph system is an important part of the immune system, the body's defense system against disease. The lymph nodes filter lymph fluid as it flows through them, trapping bacteria, viruses, and other foreign substances, which are then destroyed by special white blood cells called lymphocytes.", ...]	[1, 0, 0, 0, 0, ...]

• Loss: PListMLELoss with these parameters:

  {
      "lambda_weight": "sentence_transformers.cross_encoder.losses.plist_mle.PListMLELambdaWeight",
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16,
      "respect_input_order": true
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 1,000 evaluation samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 100 samples:

  	query	docs	labels
  type	string	list	list
  modality	text
  details	min: 5 tokens mean: 9.64 tokens max: 21 tokens	min: 4 elements mean: 8.36 elements max: 10 elements	min: 4 elements mean: 8.36 elements max: 10 elements

• Samples:

  query	docs	labels
  are mentos halal	["Making the world better, one answer at a time. well you see i have checked to see if mentos are haram but its says it has no gelatin (used to make things thicker made from pig skin). Another thing that is haram is eating anything that is not 'halal'. For many people, this just means checking if there is a 'halal' label on their packets of chicken and meat. But this is more than that. Lets just look at poultry for a while.", 'Mentos Chewy Mints are offered in several refreshing flavor varieties. Mentos Fruit has a mix of strawberry, orange and lemon. This is a must-have for any afternoon snack or a snack anytime of the day!', 'According to the following website below are Halal Mentos Mint Spearmint Fruit (Strawberry/Lemon/Orange) Rainbow (Green Apple/Raspberry/Pineapple/Watermelon/Orange/Strawberry/Grape) Duo (Blackcurrant &Lemon/Strawberry &Lime) http://www.stuartalexander.com.au/nutritional_dietary_questions_answers_fact_sheet.php#halal. ', '(By a family member). The ingredients: Sugar, glucose syrup, hydrogenated vegetable oil, citric acid, fruit juices from concentrate, starch, sucrose esters of fatty acids, natural, colors (carmine, paprika oleoresin, beta-carotene, beetrood red), carnauba wax, beeswax E414, E418. I do not want to go into detail as to why it is Haram, Mufti Saab has said E473 is Haram, so for me it is Haram. If you want to go into further detail, you are welcome to study this detailed document of what (SUCROSE esters of fatty acids) is www.fao.org/ag/agn/jecfa-additives/specs/monograph10/addi...', "Haram things are haram because they don't go together with Islamic principles. Islamic principles are in turn, a way of life and good for us both morally and physically. T … his is best explained by examples. For example, Muslims arent allowed to drink. Another thing that is haram is eating anything that is not 'halal'. For many people, this just means checking if there is a 'halal' label on their packets of chicken and meat. But this is more than that. Lets just look at poultry for a while.", ...]	[1, 0, 0, 0, 0, ...]
  how long does it take to get a masters degree in divinity	['How Long it Takes. In some schools it is possible to finish the Master of Divinity degree in three years if you live on campus or if you are a student pastor. For most full-time students in most seminaries, it takes four years; and it can take part-time students as many as eight years.', 'It is the most common academic degree in seminaries and divinity schools (e.g. in 2014 nearly 44% of all U.S. students in schools accredited by the Association of Theological Schools were enrolled in an MDiv program). Many programs also contain courses in church growth, ecclesiology, evangelism, systematic theology, Christian education, liturgical studies, Latin, Hebrew, canon law, and patristics. The degree may or may not include a thesis.', 'The Master of Divinity (M.Div.) degree involves the study of Christian theology. This degree is the first professional degree of the pastoral profession in the United States. Holders of this degree are typically prepared for pastorship, priesthood, or professional ministry. This type of program is much more comprehensive than master’s degree programs in most other fields, so if you’re new to considering theology as a field of study, it can be a bit confusing to understand your options.', "In general the master's of divinity program lasts two to three years. The length of time for completion of a doctoral degree is about the same as for a master's degree, although the time frame could be longer depending on the school. Salary. According to Pay Scale, the average salary in 2009 for a pastor with a divinity degree is about $50,000, and for a chaplain with this degree the salary is about $46,000. Your individual salary depends on your location, type of job and years of experience", "1. Earn a master's degree. While specific requirements vary at each college, university and seminary, most theological doctorate programs require students to have either a master's degree in divinity or a master's degree in religion, philosophy or theology from an accredited university to be considered for enrollment. Ad. A doctorate in theology, often referred to as a Ph.D. in theology, is the most advanced degree available in theology. Most students who earn a doctorate in theology pursue teaching positions at colleges and universities.", ...]	[1, 0, 0, 0, 0, ...]
  cost of full-time daycare	['1 Assuming full-time day care for a 2 year old child on weekdays, according to Runzheimer International, the U.S. national average cost for full-time day care is $611 a month. 2 Large cities such as Boston and New York are most expensive, with average daycare costs of a little more than $1,000 a month.', 'The report finds the average daycare cost for full-time infant care in a daycare center in Massachusetts in 2011 was $14,980 a€“ â more than any other. (State the next highest Is New york, state with an average annual cost of14,009.). 14009 only in the federal district Of washington do parents pay more€“ a â a20,178. $ 20178', 'The Cost of Daycare. – Tuition: The full time daycares in our area (8:00am – 6:00pm) range from $1500 – $2200/month or $16,500 – $24,200/year. – Summer vacation: Some daycares close for summer vacation. Ours closes for an entire month (during which you do not pay tuition) so you have to find backup childcare.', 'Average Cost. The U.S. average seems to suggest that for one toddler, in full-time day care, on weekdays, the cost is approximately $600.00 per month. But, that figure can rise to well over $1000.00 per month in major metro areas, and fall to less than $350 in rural areas.', 'The average cost of center-based daycare in the United States is $11,666 per year ($972 a month), but prices range from $3,582 to $18,773 a year ($300 to $1,564 monthly), according to the National Association of Child Care Resource & Referral Agencies (NACCRRA).', ...]	[1, 0, 0, 0, 0, ...]

• Loss: PListMLELoss with these parameters:

  {
      "lambda_weight": "sentence_transformers.cross_encoder.losses.plist_mle.PListMLELambdaWeight",
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16,
      "respect_input_order": true
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.1
• bf16: True
• per_device_eval_batch_size: 16
• load_best_model_at_end: True
• seed: 12

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 12
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_R100_ndcg@10	NanoNFCorpus_R100_ndcg@10	NanoNQ_R100_ndcg@10	NanoBEIR_R100_mean_ndcg@10
-1	-1	-	-	0.0300 (-0.5104)	0.2528 (-0.0723)	0.0168 (-0.4839)	0.0999 (-0.3555)
0.0002	1	2.1957	-	-	-	-	-
0.0508	250	2.1013	-	-	-	-	-
0.1016	500	1.9558	1.8973	0.2356 (-0.3048)	0.2227 (-0.1023)	0.3599 (-0.1407)	0.2727 (-0.1826)
0.1525	750	1.8802	-	-	-	-	-
0.2033	1000	1.8724	1.8491	0.5454 (+0.0049)	0.3366 (+0.0115)	0.6214 (+0.1208)	0.5011 (+0.0457)
0.2541	1250	1.8551	-	-	-	-	-
0.3049	1500	1.8414	1.8376	0.5071 (-0.0333)	0.3601 (+0.0351)	0.6261 (+0.1255)	0.4978 (+0.0424)
0.3558	1750	1.8354	-	-	-	-	-
0.4066	2000	1.8338	1.8308	0.5349 (-0.0055)	0.3427 (+0.0176)	0.6330 (+0.1324)	0.5035 (+0.0482)
0.4574	2250	1.8317	-	-	-	-	-
0.5082	2500	1.8384	1.8244	0.5272 (-0.0132)	0.3627 (+0.0376)	0.6456 (+0.1449)	0.5118 (+0.0565)
0.5591	2750	1.8273	-	-	-	-	-
0.6099	3000	1.8152	1.8158	0.5367 (-0.0038)	0.3556 (+0.0306)	0.6325 (+0.1319)	0.5083 (+0.0529)
0.6607	3250	1.8279	-	-	-	-	-
0.7115	3500	1.8267	1.8114	0.5355 (-0.0049)	0.3518 (+0.0268)	0.6310 (+0.1304)	0.5061 (+0.0507)
0.7624	3750	1.8171	-	-	-	-	-
0.8132	4000	1.8137	1.8099	0.5236 (-0.0168)	0.3634 (+0.0383)	0.6420 (+0.1414)	0.5097 (+0.0543)
0.8640	4250	1.8071	-	-	-	-	-
0.9148	4500	1.8132	1.8117	0.5344 (-0.0060)	0.3602 (+0.0351)	0.6438 (+0.1431)	0.5128 (+0.0574)
0.9656	4750	1.8149	-	-	-	-	-
1.0	4919	-	1.8081	0.5392 (-0.0012)	0.3614 (+0.0364)	0.6400 (+0.1393)	0.5135 (+0.0582)
-1	-1	-	-	0.5392 (-0.0012)	0.3614 (+0.0364)	0.6400 (+0.1393)	0.5135 (+0.0582)

• The bold row denotes the saved checkpoint.

Training Time

• Training: 48.4 minutes
• Evaluation: 7.8 minutes
• Total: 56.2 minutes

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.7.0.dev0
• Transformers: 5.10.2
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Reranker Models with Sentence Transformers: the end-to-end guide for training or finetuning Cross Encoder (reranker) models.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video reranker models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: training multimodal Cross Encoders.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

PListMLELoss

@inproceedings{lan2014position,
  title={Position-Aware ListMLE: A Sequential Learning Process for Ranking},
  author={Lan, Yanyan and Zhu, Yadong and Guo, Jiafeng and Niu, Shuzi and Cheng, Xueqi},
  booktitle={UAI},
  volume={14},
  pages={449--458},
  year={2014}
}