Amdestya/bm25cat-minilm-l12

CrossEncoder based on microsoft/MiniLM-L12-H384-uncased

This is a Cross Encoder model finetuned from microsoft/MiniLM-L12-H384-uncased using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: microsoft/MiniLM-L12-H384-uncased
• Maximum Sequence Length: 256 tokens
• Number of Output Labels: 1 label

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of texts
pairs = [
    ['11 [SEP] what is a bumblebee', 'Bumblebee is one of the smallest and physically weakest Autobots. While his stature allows him to do his job better than most Autobots could manage, he is self-conscious about his size. Maybe this is why he makes fast friends among the humans.'],
    ['14 [SEP] what kingdom do viruses belong to', "A: Earthworms belong to phylum annelida. This phylum is part of the animal kingdom and includes marine, freshwater and terrestrial worms in addition to leeches. The phylum annelida is characterized by its members' body segments."],
    ['19 [SEP] when was george lucas born', 'Star Wars is an American epic space opera franchise centered on a film series created by George Lucas.'],
    ['28 [SEP] weather in upper marlboro', 'Upper Marlboro 5 Day Weather. Saturday:The Upper Marlboro forecast for Apr 15 is 59 degrees and Overcast. There is 65 percentage chance of rain and 11 mph winds from the South. Sunday:The Upper Marlboro forecast for Apr 16 is 68 degrees and Partly cloudy. There is 53 percentage chance of rain and 14 mph winds from the West-Southwest. Monday:The Upper Marlboro forecast for Apr 17 is 69 degrees and Cloudy. There is 38 percentage chance of rain and 11 mph winds from the North-Northwest.'],
    ['10 [SEP] what year was a zipper created', "Ziploc is a brand of disposable, re-sealable zipper storage bags and containers originally developed by Dow Chemical Company, and now produced by S. C. Johnson & Son. Accordinâ\x80¦g to Dow's website, the bags were originally test marketed in 1968."],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    '11 [SEP] what is a bumblebee',
    [
        'Bumblebee is one of the smallest and physically weakest Autobots. While his stature allows him to do his job better than most Autobots could manage, he is self-conscious about his size. Maybe this is why he makes fast friends among the humans.',
        "A: Earthworms belong to phylum annelida. This phylum is part of the animal kingdom and includes marine, freshwater and terrestrial worms in addition to leeches. The phylum annelida is characterized by its members' body segments.",
        'Star Wars is an American epic space opera franchise centered on a film series created by George Lucas.',
        'Upper Marlboro 5 Day Weather. Saturday:The Upper Marlboro forecast for Apr 15 is 59 degrees and Overcast. There is 65 percentage chance of rain and 11 mph winds from the South. Sunday:The Upper Marlboro forecast for Apr 16 is 68 degrees and Partly cloudy. There is 53 percentage chance of rain and 14 mph winds from the West-Southwest. Monday:The Upper Marlboro forecast for Apr 17 is 69 degrees and Cloudy. There is 38 percentage chance of rain and 11 mph winds from the North-Northwest.',
        "Ziploc is a brand of disposable, re-sealable zipper storage bags and containers originally developed by Dow Chemical Company, and now produced by S. C. Johnson & Son. Accordinâ\x80¦g to Dow's website, the bags were originally test marketed in 1968.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Training Details

Training Dataset

Unnamed Dataset

• Size: 5,000,000 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 16 characters mean: 43.38 characters max: 182 characters	min: 61 characters mean: 346.12 characters max: 1042 characters	min: -11.86 mean: 0.58 max: 10.99

• Samples:

  sentence_0	sentence_1	label
  11 [SEP] what is a bumblebee	Bumblebee is one of the smallest and physically weakest Autobots. While his stature allows him to do his job better than most Autobots could manage, he is self-conscious about his size. Maybe this is why he makes fast friends among the humans.	9.059940656026205
  14 [SEP] what kingdom do viruses belong to	A: Earthworms belong to phylum annelida. This phylum is part of the animal kingdom and includes marine, freshwater and terrestrial worms in addition to leeches. The phylum annelida is characterized by its members' body segments.	-3.277885595957438
  19 [SEP] when was george lucas born	Star Wars is an American epic space opera franchise centered on a film series created by George Lucas.	-6.936024824778239

• Loss: FitMixinLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 1
• fp16: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0032	500	59.9783
0.0064	1000	59.2136
0.0096	1500	45.5225
0.0128	2000	35.2272
0.016	2500	27.4072
0.0192	3000	20.5064
0.0224	3500	14.9079
0.0256	4000	10.5768
0.0288	4500	8.0322
0.032	5000	6.6162
0.0352	5500	5.5398
0.0384	6000	4.8188
0.0416	6500	4.3552
0.0448	7000	4.2357
0.048	7500	3.8302
0.0512	8000	3.6362
0.0544	8500	3.6564
0.0576	9000	3.4237
0.0608	9500	3.3077
0.064	10000	3.3007
0.0672	10500	3.1063
0.0704	11000	3.1284
0.0736	11500	3.0224
0.0768	12000	2.9805
0.08	12500	2.9563
0.0832	13000	2.8132
0.0864	13500	2.7918
0.0896	14000	2.7815
0.0928	14500	2.7124
0.096	15000	2.6761
0.0992	15500	2.6197
0.1024	16000	2.6526
0.1056	16500	2.5311
0.1088	17000	2.5568
0.112	17500	2.6828
0.1152	18000	2.4638
0.1184	18500	2.4284
0.1216	19000	2.4414
0.1248	19500	2.3855
0.128	20000	2.333
0.1312	20500	2.4313
0.1344	21000	2.3931
0.1376	21500	2.2852
0.1408	22000	2.305
0.144	22500	2.2756
0.1472	23000	2.3067
0.1504	23500	2.1949
0.1536	24000	2.2292
0.1568	24500	2.1303
0.16	25000	2.263
0.1632	25500	2.1788
0.1664	26000	2.2207
0.1696	26500	2.1475
0.1728	27000	2.164
0.176	27500	2.153
0.1792	28000	2.1501
0.1824	28500	2.053
0.1856	29000	2.0759
0.1888	29500	2.0559
0.192	30000	2.0965
0.1952	30500	2.0919
0.1984	31000	2.0474
0.2016	31500	2.0588
0.2048	32000	2.0081
0.208	32500	1.9454
0.2112	33000	2.0132
0.2144	33500	1.9604
0.2176	34000	1.9983
0.2208	34500	1.96
0.224	35000	2.0316
0.2272	35500	1.9134
0.2304	36000	1.8939
0.2336	36500	1.9711
0.2368	37000	1.8776
0.24	37500	1.9105
0.2432	38000	1.9143
0.2464	38500	1.9823
0.2496	39000	1.8753
0.2528	39500	1.8919
0.256	40000	1.9114
0.2592	40500	1.7991
0.2624	41000	1.9024
0.2656	41500	1.8768
0.2688	42000	1.8603
0.272	42500	1.8989
0.2752	43000	1.8351
0.2784	43500	1.9194
0.2816	44000	1.8442
0.2848	44500	1.7796
0.288	45000	1.8053
0.2912	45500	1.8206
0.2944	46000	1.7968
0.2976	46500	1.7822
0.3008	47000	1.8164
0.304	47500	1.8004
0.3072	48000	1.7753
0.3104	48500	1.7208
0.3136	49000	1.7466
0.3168	49500	1.7683
0.32	50000	1.74
0.3232	50500	1.7443
0.3264	51000	1.7038
0.3296	51500	1.7652
0.3328	52000	1.7273
0.336	52500	1.6958
0.3392	53000	1.7244
0.3424	53500	1.7494
0.3456	54000	1.7154
0.3488	54500	1.639
0.352	55000	1.7243
0.3552	55500	1.7498
0.3584	56000	1.6738
0.3616	56500	1.7433
0.3648	57000	1.6463
0.368	57500	1.6569
0.3712	58000	1.6156
0.3744	58500	1.6724
0.3776	59000	1.6717
0.3808	59500	1.6632
0.384	60000	1.6583
0.3872	60500	1.6025
0.3904	61000	1.673
0.3936	61500	1.6196
0.3968	62000	1.5965
0.4	62500	1.6495
0.4032	63000	1.6045
0.4064	63500	1.6149
0.4096	64000	1.6235
0.4128	64500	1.6124
0.416	65000	1.6573
0.4192	65500	1.564
0.4224	66000	1.6334
0.4256	66500	1.6125
0.4288	67000	1.5991
0.432	67500	1.6553
0.4352	68000	1.5881
0.4384	68500	1.5837
0.4416	69000	1.6174
0.4448	69500	1.5853
0.448	70000	1.5943
0.4512	70500	1.5788
0.4544	71000	1.5897
0.4576	71500	1.5761
0.4608	72000	1.5861
0.464	72500	1.5995
0.4672	73000	1.5139
0.4704	73500	1.5077
0.4736	74000	1.5454
0.4768	74500	1.5739
0.48	75000	1.5388
0.4832	75500	1.5605
0.4864	76000	1.5008
0.4896	76500	1.5394
0.4928	77000	1.5495
0.496	77500	1.5117
0.4992	78000	1.5299
0.5024	78500	1.5119
0.5056	79000	1.5316
0.5088	79500	1.495
0.512	80000	1.5183
0.5152	80500	1.5114
0.5184	81000	1.588
0.5216	81500	1.529
0.5248	82000	1.4675
0.528	82500	1.461
0.5312	83000	1.5184
0.5344	83500	1.5096
0.5376	84000	1.4947
0.5408	84500	1.5226
0.544	85000	1.4527
0.5472	85500	1.5005
0.5504	86000	1.4343
0.5536	86500	1.5295
0.5568	87000	1.5124
0.56	87500	1.4846
0.5632	88000	1.4529
0.5664	88500	1.5142
0.5696	89000	1.4657
0.5728	89500	1.4037
0.576	90000	1.5109
0.5792	90500	1.4811
0.5824	91000	1.4747
0.5856	91500	1.4703
0.5888	92000	1.4983
0.592	92500	1.5026
0.5952	93000	1.4535
0.5984	93500	1.4487
0.6016	94000	1.4738
0.6048	94500	1.4541
0.608	95000	1.4961
0.6112	95500	1.4642
0.6144	96000	1.4701
0.6176	96500	1.4009
0.6208	97000	1.4525
0.624	97500	1.4293
0.6272	98000	1.4813
0.6304	98500	1.4325
0.6336	99000	1.442
0.6368	99500	1.3995
0.64	100000	1.4422
0.6432	100500	1.4562
0.6464	101000	1.3974
0.6496	101500	1.41
0.6528	102000	1.4163
0.656	102500	1.3739
0.6592	103000	1.4092
0.6624	103500	1.4165
0.6656	104000	1.3969
0.6688	104500	1.3604
0.672	105000	1.4236
0.6752	105500	1.3966
0.6784	106000	1.3986
0.6816	106500	1.4416
0.6848	107000	1.4469
0.688	107500	1.3491
0.6912	108000	1.3696
0.6944	108500	1.4174
0.6976	109000	1.4076
0.7008	109500	1.3907
0.704	110000	1.4002
0.7072	110500	1.4038
0.7104	111000	1.3615
0.7136	111500	1.3949
0.7168	112000	1.3701
0.72	112500	1.4129
0.7232	113000	1.363
0.7264	113500	1.4329
0.7296	114000	1.3815
0.7328	114500	1.4081
0.736	115000	1.3349
0.7392	115500	1.3973
0.7424	116000	1.4061
0.7456	116500	1.3453
0.7488	117000	1.4026
0.752	117500	1.4208
0.7552	118000	1.3831
0.7584	118500	1.3848
0.7616	119000	1.3612
0.7648	119500	1.3605
0.768	120000	1.3819
0.7712	120500	1.3848
0.7744	121000	1.3684
0.7776	121500	1.3351
0.7808	122000	1.3105
0.784	122500	1.3632
0.7872	123000	1.3292
0.7904	123500	1.3687
0.7936	124000	1.3921
0.7968	124500	1.365
0.8	125000	1.4125
0.8032	125500	1.3744
0.8064	126000	1.3761
0.8096	126500	1.3648
0.8128	127000	1.3296
0.816	127500	1.3268
0.8192	128000	1.3452
0.8224	128500	1.2776
0.8256	129000	1.3698
0.8288	129500	1.3082
0.832	130000	1.3633
0.8352	130500	1.3342
0.8384	131000	1.353
0.8416	131500	1.3235
0.8448	132000	1.3248
0.848	132500	1.3217
0.8512	133000	1.2987
0.8544	133500	1.3315
0.8576	134000	1.3236
0.8608	134500	1.3408
0.864	135000	1.3374
0.8672	135500	1.3414
0.8704	136000	1.3265
0.8736	136500	1.3494
0.8768	137000	1.3242
0.88	137500	1.3467
0.8832	138000	1.3015
0.8864	138500	1.3263
0.8896	139000	1.3156
0.8928	139500	1.3236
0.896	140000	1.3199
0.8992	140500	1.2884
0.9024	141000	1.3108
0.9056	141500	1.3395
0.9088	142000	1.3369
0.912	142500	1.3775
0.9152	143000	1.3051
0.9184	143500	1.304
0.9216	144000	1.2586
0.9248	144500	1.3062
0.928	145000	1.3766
0.9312	145500	1.3018
0.9344	146000	1.3378
0.9376	146500	1.3242
0.9408	147000	1.3094
0.944	147500	1.2792
0.9472	148000	1.3306
0.9504	148500	1.3269
0.9536	149000	1.3134
0.9568	149500	1.263
0.96	150000	1.3085
0.9632	150500	1.2959
0.9664	151000	1.323
0.9696	151500	1.3102
0.9728	152000	1.2929
0.976	152500	1.2887
0.9792	153000	1.307
0.9824	153500	1.3269
0.9856	154000	1.3066
0.9888	154500	1.3549
0.992	155000	1.3293
0.9952	155500	1.2797
0.9984	156000	1.2933

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 5.1.1
• Transformers: 4.57.0
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.1
• Datasets: 4.1.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}