yjoonjang/reranker-msmarco-v1.1-MiniLM-L12-H384-uncased-plistmle-customweight

CrossEncoder based on microsoft/MiniLM-L12-H384-uncased

This is a Cross Encoder model finetuned from microsoft/MiniLM-L12-H384-uncased on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: microsoft/MiniLM-L12-H384-uncased
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("yjoonjang/reranker-msmarco-v1.1-MiniLM-L12-H384-uncased-plistmle-customweight")
# Get scores for pairs of texts
pairs = [
    ['How many calories in an egg', 'There are on average between 55 and 80 calories in an egg depending on its size.'],
    ['How many calories in an egg', 'Egg whites are very low in calories, have no fat, no cholesterol, and are loaded with protein.'],
    ['How many calories in an egg', 'Most of the calories in an egg come from the yellow yolk in the center.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (3,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'How many calories in an egg',
    [
        'There are on average between 55 and 80 calories in an egg depending on its size.',
        'Egg whites are very low in calories, have no fat, no cholesterol, and are loaded with protein.',
        'Most of the calories in an egg come from the yellow yolk in the center.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	NanoMSMARCO_R100	NanoNFCorpus_R100	NanoNQ_R100
map	0.4776 (-0.0120)	0.3041 (+0.0431)	0.4790 (+0.0594)
mrr@10	0.4665 (-0.0110)	0.4803 (-0.0195)	0.4831 (+0.0564)
ndcg@10	0.5330 (-0.0074)	0.3170 (-0.0081)	0.5407 (+0.0401)

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	Value
map	0.4202 (+0.0302)
mrr@10	0.4766 (+0.0086)
ndcg@10	0.4636 (+0.0082)

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 78,704 training samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 10 characters mean: 34.15 characters max: 100 characters	min: 3 elements mean: 6.50 elements max: 10 elements	min: 3 elements mean: 6.50 elements max: 10 elements

• Samples:

  query	docs	labels
  is dna acidic	['Great question. Deoxyribonucleic Acid is made up of a sugar (the deoxyribose), a phosphate ion, and a nitrogenous base. the phosphate ion is where the acidity is, its also called phosphoric acid while in your cells.', 'Confidence votes 134. Great question. Deoxyribonucleic Acid is made up of a sugar (the deoxyribose), a phosphate ion, and a nitrogenous base. the phosphate ion is where the acidity is, its also called phosphoric acid while in your cells.', 'Yes, DNA is an acid. In the structure of DNA, there is sugar (deoxyribose), a phosphate ion, and a nitrogenous base. the phosphate ion gives the acidic property. Fri Jan 14, 2005 5:07 pm. Which is kind of funny because the nitrogen base really IS a base.', 'First of all, DNA is not made up of nucleotide bases but of nucleotides. These consist of a sugar bound to one of the 4 nucleobases Adenine, Cytosine, Guanine or Thymine (Uracil in the case of RNA) and a phosphate group.', 'As you already know, the letters in DNA stand for Deoxyr...	[1, 1, 0, 0, 0, ...]
  what is a nerve conduction study	['A nerve conduction study (NCS), also called a nerve conduction velocity (NCV) test--is a measurement of the speed of conduction of an electrical impulse through a nerve. NCS can determine nerve damage and destruction. During the test, the nerve is stimulated, usually with surface electrode patches attached to the skin. The nerve conduction velocity (speed) is then calculated by measuring the distance between electrodes and the time it takes for electrical impulses to travel between electrodes. A related procedure that may be performed is electromyography (EMG).', 'Nerve conduction studies and needle EMG are commonly performed by physical medicine and rehabilitation or neurology specialists to assess the ability of the nervous system to conduct electrical impulses and to evaluate nerve/muscle function to determine if neuromuscular disease is present. Motor nerve conduction studies. In motor nerve conduction studies, motor nerves are stimulated and the compound muscle action potential ...	[1, 0, 0, 0, 0, ...]
  average nhl salary	['Most NHL teams, 17 of them, paid an average salary per player ranging from $1.8 million to $1 million. Only five teams--Philadelphia Flyers, Colorado Avalanche, New York Islanders, Anaheim Ducks and the Tampa Bay Lightning paid average salaries of under a million dollars per player. The minimum NHL player salary, per the collective bargaining agreement, is $500,000 in 2009-10 and 2010-11. The maximum seasonal salary for players new to the NHL is $900,000 for 2009 and 2010; and $925,000 for 2011 draftees.', 'Sports NHL NHL Salaries 2015. National Hockey League salary cap is what every team has to spend a limited amount of money on purchasing players every year. So that makes it more compatible to earn good players in all the teams and make it more balanced in strength. This led some teams to trade away well paid star players to fit the cap. But the teams were allowed to spend $70.4 Million in a year for the prorated shorted season length. The highest NHL career salary earns as of now ...	[1, 0, 0, 0, 0, ...]

• Loss: ListMLELoss with these parameters:

  {
      "lambda_weight": "sentence_transformers.cross_encoder.losses.ListMLELoss.ListMLELambdaWeight",
      "activation_fct": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16,
      "respect_input_order": true
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 1,000 evaluation samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 12 characters mean: 33.4 characters max: 93 characters	min: 3 elements mean: 6.50 elements max: 10 elements	min: 3 elements mean: 6.50 elements max: 10 elements

• Samples:

  query	docs	labels
  what is contamination	['Contamination is the presence of an unwanted constituent, contaminant or impurity in a material, physical body, natural environment, workplace, etc. Contamination may include residual radioactive material remaining at a site after the completion of decommissioning of a site where there was a nuclear reactor, such as a power plant, experimental reactor, isotope reactor or a nuclear powered ship or submarine.', 'contamination. 1. the soiling or making inferior by contact or mixture, as by introduction of infectious organisms into a wound, into water, milk, food or onto the external surface of the body or on bandages and other dressings. 2. the deposition of radioactive material in any place where it is not desired. See Cross contamination Public health The presence of any foreign or undesired material in a system–eg, toxic contamination of the ground water in an ecosystem or untreated sewage into a stream Radiation physics The deposition of radioactive material in any place where it is...	[1, 0, 0, 0, 0, ...]
  what does crisis representation in social research mean	['The crisis of representation that now seems so apparent after the writing of Baudrillard was also the result of a convergence of historical conditions both inside and outside of art. As a result, art’s capacity to depict the world was effected. Digging through all of my archived materials continues to be a good distraction from being productive. I stumbled upon this diatribe… but can’t figure out if I wrote this for a class or for my own geekful bliss…. Issues surrounding representation have played a key role in the development of postmodern art.', 'CRISIS OF REPRESENTATION. This phrase was coined by George Marcus and Michael Fischer to refer specifically to the uncertainty within the human sciences about adequate means of describing social reality. This crisis arises from the (noncontroversial) claim that no interpretive account can ever directly or completely capture lived experience. Broadly conceived, the crisis is part of a more general set of ideas across the human sciences tha...	[1, 1, 0, 0, 0, ...]
  what was moche	['The Moche civilization (alternatively, the Mochica culture, Early Chimu, Pre-Chimu, Proto-Chimu, etc.) flourished in northern Peru with its capital near present-day Moche and Trujillo, from about 100 AD to 800 AD, during the Regional Development Epoch.', 'Moche Politics and Economy. The Moche were a stratified society with a powerful elite and an elaborate, well-codified ritual process. The political economy was based on the presence of large civic-ceremonial centers that produced a wide range of goods which were marketed to rural agrarian villages.', 'The Moche civilization (also known as the Mochica) flourished along the northern coast and valleys of ancient Peru, in particular, in the Chicama and Trujillo Valleys, between 1 CE and 800 CE.', 'While Mochica has been used in place of Moche by people describing this culture, the word Mochica actually refers to a particular dialect. This dialect, however, was not proven to be the dialect of the Moche Civilization and culture.', 'Moche ...	[1, 0, 0, 0, 0, ...]

• Loss: ListMLELoss with these parameters:

  {
      "lambda_weight": "sentence_transformers.cross_encoder.losses.ListMLELoss.ListMLELambdaWeight",
      "activation_fct": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16,
      "respect_input_order": true
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_R100_ndcg@10	NanoNFCorpus_R100_ndcg@10	NanoNQ_R100_ndcg@10	NanoBEIR_R100_mean_ndcg@10
-1	-1	-	-	0.0284 (-0.5121)	0.2663 (-0.0587)	0.0359 (-0.4647)	0.1102 (-0.3452)
0.0002	1	11.2052	-	-	-	-	-
0.0508	250	10.1581	-	-	-	-	-
0.1016	500	9.1585	9.0436	0.0470 (-0.4934)	0.3321 (+0.0071)	0.0242 (-0.4764)	0.1344 (-0.3209)
0.1525	750	9.0556	-	-	-	-	-
0.2033	1000	8.9995	8.9401	0.2576 (-0.2828)	0.2456 (-0.0795)	0.3359 (-0.1648)	0.2797 (-0.1757)
0.2541	1250	8.9878	-	-	-	-	-
0.3049	1500	8.9811	8.8985	0.4518 (-0.0886)	0.2943 (-0.0307)	0.3790 (-0.1217)	0.3750 (-0.0803)
0.3558	1750	8.9185	-	-	-	-	-
0.4066	2000	8.863	8.9124	0.4213 (-0.1191)	0.2972 (-0.0278)	0.4477 (-0.0530)	0.3887 (-0.0667)
0.4574	2250	8.8962	-	-	-	-	-
0.5082	2500	8.9063	8.8869	0.5117 (-0.0287)	0.3135 (-0.0116)	0.5208 (+0.0202)	0.4487 (-0.0067)
0.5591	2750	8.9379	-	-	-	-	-
0.6099	3000	8.869	8.8610	0.5208 (-0.0196)	0.3203 (-0.0048)	0.4566 (-0.0440)	0.4326 (-0.0228)
0.6607	3250	8.8965	-	-	-	-	-
0.7115	3500	8.8487	8.8466	0.5024 (-0.0380)	0.3007 (-0.0243)	0.4827 (-0.0179)	0.4286 (-0.0267)
0.7624	3750	8.8695	-	-	-	-	-
0.8132	4000	8.8732	8.8497	0.5207 (-0.0197)	0.3247 (-0.0003)	0.5292 (+0.0285)	0.4582 (+0.0028)
0.8640	4250	8.9325	-	-	-	-	-
0.9148	4500	8.8244	8.8205	0.5330 (-0.0074)	0.3170 (-0.0081)	0.5407 (+0.0401)	0.4636 (+0.0082)
0.9656	4750	8.858	-	-	-	-	-
-1	-1	-	-	0.5330 (-0.0074)	0.3170 (-0.0081)	0.5407 (+0.0401)	0.4636 (+0.0082)

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.49.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.4.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ListMLELoss

@inproceedings{lan2013position,
    title={Position-aware ListMLE: a sequential learning process for ranking},
    author={Lan, Yanyan and Guo, Jiafeng and Cheng, Xueqi and Liu, Tie-Yan},
    booktitle={Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence},
    pages={333--342},
    year={2013}
}