---
language:
- en
license: apache-2.0
tags:
- sentence-transformers
- multi-vector
- colbert
- late-interaction
- generated_from_trainer
- dataset_size:50000
- loss:MultiVectorMultipleNegativesRankingLoss
base_model: answerdotai/ModernBERT-base
widget:
- text: The job of a physical education teacher is to promote students' physical fitness
through exercise and sport activities. A bachelor's degree in physical education
is required along with hands-on student teaching experience. Those who teach in
public schools must become licensed. Show me 10 popular schools.
- text: The Handheld Braille Labeler, model 2891, is a braille labeler designed for
use by individuals who are blind or deaf blind or have low vision. This lightweight
plastic braille label gun has both braille and print characters on its character
selection dial.
- text: surname meaning, fontana
- text: garlic did originate near Siberia, Russia so powder was made also at russia
at american factory. Garlic is native to central Asia, but its use spread across
the world more than 5000 years ago, before recorded history. It was worshipped
by the Egyptians and fed to workers building the Gread Pyramid at Giza, about
2600 BC. Greek athletes ate it to build their strength.
- text: "With a 50-50 ratio of marijuana to tobacco, the cost of producing a pack\
\ of 20 pre-rolled joints could be brought down to just a little more than $20â\x80\
\x94so a $40 pack at the store. It isnâ\x80\x99t as easy as it seems, though.\
\ The government has a vested interest in producing income from the selling of\
\ marijuana.nother solution: mix the marijuana with tobacco. If marijuana cigarettes\
\ were to be mixed with tobacco, at a 50-50 ratio, it would bring the cost down\
\ significantly. Many tobacco farmers will wholesale a pound of their product\
\ for less than $2."
datasets:
- sentence-transformers/msmarco-bm25
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
- maxsim_accuracy@1
- maxsim_accuracy@3
- maxsim_accuracy@5
- maxsim_accuracy@10
- maxsim_precision@1
- maxsim_precision@3
- maxsim_precision@5
- maxsim_precision@10
- maxsim_recall@1
- maxsim_recall@3
- maxsim_recall@5
- maxsim_recall@10
- maxsim_ndcg@10
- maxsim_mrr@10
- maxsim_map@100
model-index:
- name: ColBERT ModernBERT-base adapter trained on MS MARCO triplets
results:
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoMSMARCO
type: NanoMSMARCO
metrics:
- type: maxsim_accuracy@1
value: 0.28
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.36
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.44
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.58
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.28
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.11999999999999998
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.08800000000000001
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.05800000000000001
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.28
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.36
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.44
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.58
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.40729511421184744
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.35540476190476183
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.37407265817151775
name: Maxsim Map@100
- type: maxsim_accuracy@1
value: 0.36
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.48
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.54
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.66
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.36
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.15999999999999998
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.10800000000000003
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.066
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.36
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.48
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.54
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.66
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.49442274317178536
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.4429365079365079
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.45963344261609657
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoNQ
type: NanoNQ
metrics:
- type: maxsim_accuracy@1
value: 0.24
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.42
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.5
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.68
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.24
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.14
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.1
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.07
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.21
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.39
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.46
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.62
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.40266896873385
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.3570555555555556
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.3367445324988335
name: Maxsim Map@100
- type: maxsim_accuracy@1
value: 0.24
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.48
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.52
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.68
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.24
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.15999999999999998
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.10800000000000001
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.07
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.22
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.44
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.49
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.63
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.42335725741016633
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.3764126984126985
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.3641117600913058
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoFiQA2018
type: NanoFiQA2018
metrics:
- type: maxsim_accuracy@1
value: 0.28
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.36
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.38
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.46
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.28
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.14666666666666667
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.10400000000000001
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.07
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.18085714285714286
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.24421428571428572
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.2613809523809524
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.3177936507936508
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.2829365248597584
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.3277460317460318
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.25526883700917796
name: Maxsim Map@100
- type: maxsim_accuracy@1
value: 0.28
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.36
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.4
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.54
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.28
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.15333333333333332
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.11200000000000002
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.07600000000000001
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.16752380952380952
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.2548809523809524
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.2767142857142857
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.3509365079365079
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.29588260673176586
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.34221428571428575
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.2562432169252871
name: Maxsim Map@100
- task:
type: multi-vector-nano-beir
name: Multi Vector Nano BEIR
dataset:
name: NanoBEIR mean
type: NanoBEIR_mean
metrics:
- type: maxsim_accuracy@1
value: 0.26666666666666666
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.38000000000000006
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.43999999999999995
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.5733333333333334
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.26666666666666666
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.13555555555555557
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.09733333333333334
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.066
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.2236190476190476
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.3314047619047619
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.38712698412698415
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.5059312169312169
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.36430020260181867
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.34673544973544973
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.32202867589317646
name: Maxsim Map@100
- type: maxsim_accuracy@1
value: 0.4177080062794349
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.5826059654631083
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.6612244897959183
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.7476295133437991
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.4177080062794349
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.26118262689691263
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.20367974882260598
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.14069073783359495
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.2458300801965985
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.37450152074921533
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.43942477624630827
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.5091784001743137
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.4601047185655218
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.52210818320002
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.39426164391037877
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoClimateFEVER
type: NanoClimateFEVER
metrics:
- type: maxsim_accuracy@1
value: 0.24
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.36
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.42
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.58
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.24
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.12666666666666665
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.096
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.07
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.115
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.1716666666666667
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.21
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.2956666666666667
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.24041745872675477
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.32949206349206345
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.19119463961186825
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoDBPedia
type: NanoDBPedia
metrics:
- type: maxsim_accuracy@1
value: 0.62
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.78
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.84
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.84
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.62
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.5066666666666666
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.4600000000000001
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.37199999999999994
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.07597496495197292
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.13453155589956123
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.18091247197440335
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.24417018558997275
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.47528049469754174
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.7096666666666667
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.33319642109507475
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoFEVER
type: NanoFEVER
metrics:
- type: maxsim_accuracy@1
value: 0.66
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.86
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.94
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.96
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.66
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.29333333333333333
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.19199999999999995
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.09799999999999998
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.6166666666666667
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.8266666666666667
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.8866666666666667
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.9066666666666667
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.7785598343056287
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.7646666666666667
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.7262135076252723
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoHotpotQA
type: NanoHotpotQA
metrics:
- type: maxsim_accuracy@1
value: 0.6
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.78
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.84
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.88
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.6
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.3066666666666667
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.21199999999999997
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.11
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.3
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.46
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.53
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.55
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.5333513308213242
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.7101904761904763
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.46056571092666815
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoNFCorpus
type: NanoNFCorpus
metrics:
- type: maxsim_accuracy@1
value: 0.34
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.44
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.5
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.56
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.34
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.3066666666666666
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.26
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.20200000000000004
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.04575906542583815
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.07529172371810992
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.08624348944602006
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.10951166766943608
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.26495764448071357
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.4045238095238095
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.12386011576706756
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoQuoraRetrieval
type: NanoQuoraRetrieval
metrics:
- type: maxsim_accuracy@1
value: 0.8
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.88
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.92
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.96
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.8
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.3666666666666666
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.23999999999999994
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.12999999999999998
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.6906666666666667
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.8386666666666668
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.8953333333333333
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.9493333333333334
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.8671373304417355
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.8551904761904763
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.8334354497354497
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoSCIDOCS
type: NanoSCIDOCS
metrics:
- type: maxsim_accuracy@1
value: 0.24
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.52
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.62
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.72
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.24
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.24666666666666667
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.204
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.13799999999999998
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.05066666666666667
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.15266666666666667
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.20866666666666664
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.2826666666666667
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.262749080389903
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.4024126984126984
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.20512257052463553
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoArguAna
type: NanoArguAna
metrics:
- type: maxsim_accuracy@1
value: 0.08
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.34
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.6
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.7
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.08
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.11333333333333333
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.12000000000000002
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.07
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.08
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.34
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.6
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.7
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.3683627635846936
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.26357142857142846
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.27589617371875436
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoSciFact
type: NanoSciFact
metrics:
- type: maxsim_accuracy@1
value: 0.46
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.6
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.66
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.68
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.46
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.22
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.14400000000000002
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.07800000000000001
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.435
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.59
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.65
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.68
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.5754141216372103
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.5419999999999999
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.5522978989964283
name: Maxsim Map@100
- task:
type: multi-vector-information-retrieval
name: Multi Vector Information Retrieval
dataset:
name: NanoTouche2020
type: NanoTouche2020
metrics:
- type: maxsim_accuracy@1
value: 0.5102040816326531
name: Maxsim Accuracy@1
- type: maxsim_accuracy@3
value: 0.6938775510204082
name: Maxsim Accuracy@3
- type: maxsim_accuracy@5
value: 0.7959183673469388
name: Maxsim Accuracy@5
- type: maxsim_accuracy@10
value: 0.9591836734693877
name: Maxsim Accuracy@10
- type: maxsim_precision@1
value: 0.5102040816326531
name: Maxsim Precision@1
- type: maxsim_precision@3
value: 0.43537414965986393
name: Maxsim Precision@3
- type: maxsim_precision@5
value: 0.39183673469387753
name: Maxsim Precision@5
- type: maxsim_precision@10
value: 0.3489795918367347
name: Maxsim Precision@10
- type: maxsim_recall@1
value: 0.038533202654159965
name: Maxsim Recall@1
- type: maxsim_recall@3
value: 0.1041488710745086
name: Maxsim Recall@3
- type: maxsim_recall@5
value: 0.15798517740063198
name: Maxsim Recall@5
- type: maxsim_recall@10
value: 0.26036750773682693
name: Maxsim Recall@10
- type: maxsim_ndcg@10
value: 0.4014686749525606
name: Maxsim Ndcg@10
- type: maxsim_mrr@10
value: 0.6441286038224814
name: Maxsim Mrr@10
- type: maxsim_map@100
value: 0.3436304632010159
name: Maxsim Map@100
---
# ColBERT ModernBERT-base adapter trained on MS MARCO triplets
This is a [Multi-Vector Encoder](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the [msmarco-bm25](https://huggingface.co/datasets/sentence-transformers/msmarco-bm25) dataset using the [sentence-transformers](https://www.SBERT.net) library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.
## Model Details
### Model Description
- **Model Type:** Multi-Vector Encoder
- **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base)
- **Maximum Sequence Length:** 8192 tokens
- **Output Dimensionality:** 128 dimensions
- **Similarity Function:** maxsim
- **Supported Modality:** Text
- **Training Dataset:**
- [msmarco-bm25](https://huggingface.co/datasets/sentence-transformers/msmarco-bm25)
- **Language:** en
- **License:** apache-2.0
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Documentation:** [Multi-Vector Encoder Documentation](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)
- **Hugging Face:** [Multi-Vector Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=multi-vector)
### Full Model Architecture
```
MultiVectorEncoder(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
(1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
(2): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
(3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import MultiVectorEncoder
# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-ModernBERT-base-msmarco-peft")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
'what does marijuana cost per joint',
]
documents = [
'With a 50-50 ratio of marijuana to tobacco, the cost of producing a pack of 20 pre-rolled joints could be brought down to just a little more than $20â\x80\x94so a $40 pack at the store. It isnâ\x80\x99t as easy as it seems, though. The government has a vested interest in producing income from the selling of marijuana.nother solution: mix the marijuana with tobacco. If marijuana cigarettes were to be mixed with tobacco, at a 50-50 ratio, it would bring the cost down significantly. Many tobacco farmers will wholesale a pound of their product for less than $2.',
'What does a dime,dub,eigth,quarter,and a zip of marijuana look like and cost?',
'In January of 1980, residents decided to incorporate by an overwhelming margin. The Town of Farragut was incorporated on January 16, 1980, with the first board of Mayor and Alderman elected on April 1, 1980.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (8, 128) (129, 128)
# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[7.3787, 4.1125, 1.2479]])
```
## Evaluation
### Metrics
#### Multi Vector Information Retrieval
* Datasets: `NanoMSMARCO`, `NanoNQ`, `NanoFiQA2018`, `NanoClimateFEVER`, `NanoDBPedia`, `NanoFEVER`, `NanoFiQA2018`, `NanoHotpotQA`, `NanoMSMARCO`, `NanoNFCorpus`, `NanoNQ`, `NanoQuoraRetrieval`, `NanoSCIDOCS`, `NanoArguAna`, `NanoSciFact` and `NanoTouche2020`
* Evaluated with [MultiVectorInformationRetrievalEvaluator](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#sentence_transformers.multi_vector_encoder.evaluation.MultiVectorInformationRetrievalEvaluator)
| Metric | NanoMSMARCO | NanoNQ | NanoFiQA2018 | NanoClimateFEVER | NanoDBPedia | NanoFEVER | NanoHotpotQA | NanoNFCorpus | NanoQuoraRetrieval | NanoSCIDOCS | NanoArguAna | NanoSciFact | NanoTouche2020 |
|:--------------------|:------------|:-----------|:-------------|:-----------------|:------------|:-----------|:-------------|:-------------|:-------------------|:------------|:------------|:------------|:---------------|
| maxsim_accuracy@1 | 0.36 | 0.24 | 0.28 | 0.24 | 0.62 | 0.66 | 0.6 | 0.34 | 0.8 | 0.24 | 0.08 | 0.46 | 0.5102 |
| maxsim_accuracy@3 | 0.48 | 0.48 | 0.36 | 0.36 | 0.78 | 0.86 | 0.78 | 0.44 | 0.88 | 0.52 | 0.34 | 0.6 | 0.6939 |
| maxsim_accuracy@5 | 0.54 | 0.52 | 0.4 | 0.42 | 0.84 | 0.94 | 0.84 | 0.5 | 0.92 | 0.62 | 0.6 | 0.66 | 0.7959 |
| maxsim_accuracy@10 | 0.66 | 0.68 | 0.54 | 0.58 | 0.84 | 0.96 | 0.88 | 0.56 | 0.96 | 0.72 | 0.7 | 0.68 | 0.9592 |
| maxsim_precision@1 | 0.36 | 0.24 | 0.28 | 0.24 | 0.62 | 0.66 | 0.6 | 0.34 | 0.8 | 0.24 | 0.08 | 0.46 | 0.5102 |
| maxsim_precision@3 | 0.16 | 0.16 | 0.1533 | 0.1267 | 0.5067 | 0.2933 | 0.3067 | 0.3067 | 0.3667 | 0.2467 | 0.1133 | 0.22 | 0.4354 |
| maxsim_precision@5 | 0.108 | 0.108 | 0.112 | 0.096 | 0.46 | 0.192 | 0.212 | 0.26 | 0.24 | 0.204 | 0.12 | 0.144 | 0.3918 |
| maxsim_precision@10 | 0.066 | 0.07 | 0.076 | 0.07 | 0.372 | 0.098 | 0.11 | 0.202 | 0.13 | 0.138 | 0.07 | 0.078 | 0.349 |
| maxsim_recall@1 | 0.36 | 0.22 | 0.1675 | 0.115 | 0.076 | 0.6167 | 0.3 | 0.0458 | 0.6907 | 0.0507 | 0.08 | 0.435 | 0.0385 |
| maxsim_recall@3 | 0.48 | 0.44 | 0.2549 | 0.1717 | 0.1345 | 0.8267 | 0.46 | 0.0753 | 0.8387 | 0.1527 | 0.34 | 0.59 | 0.1041 |
| maxsim_recall@5 | 0.54 | 0.49 | 0.2767 | 0.21 | 0.1809 | 0.8867 | 0.53 | 0.0862 | 0.8953 | 0.2087 | 0.6 | 0.65 | 0.158 |
| maxsim_recall@10 | 0.66 | 0.63 | 0.3509 | 0.2957 | 0.2442 | 0.9067 | 0.55 | 0.1095 | 0.9493 | 0.2827 | 0.7 | 0.68 | 0.2604 |
| **maxsim_ndcg@10** | **0.4944** | **0.4234** | **0.2959** | **0.2404** | **0.4753** | **0.7786** | **0.5334** | **0.265** | **0.8671** | **0.2627** | **0.3684** | **0.5754** | **0.4015** |
| maxsim_mrr@10 | 0.4429 | 0.3764 | 0.3422 | 0.3295 | 0.7097 | 0.7647 | 0.7102 | 0.4045 | 0.8552 | 0.4024 | 0.2636 | 0.542 | 0.6441 |
| maxsim_map@100 | 0.4596 | 0.3641 | 0.2562 | 0.1912 | 0.3332 | 0.7262 | 0.4606 | 0.1239 | 0.8334 | 0.2051 | 0.2759 | 0.5523 | 0.3436 |
#### Multi Vector Nano BEIR
* Dataset: `NanoBEIR_mean`
* Evaluated with [MultiVectorNanoBEIREvaluator](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#sentence_transformers.multi_vector_encoder.evaluation.MultiVectorNanoBEIREvaluator) with these parameters:
```json
{
"dataset_names": [
"msmarco",
"nq",
"fiqa2018"
],
"dataset_id": "sentence-transformers/NanoBEIR-en"
}
```
| Metric | Value |
|:--------------------|:-----------|
| maxsim_accuracy@1 | 0.2667 |
| maxsim_accuracy@3 | 0.38 |
| maxsim_accuracy@5 | 0.44 |
| maxsim_accuracy@10 | 0.5733 |
| maxsim_precision@1 | 0.2667 |
| maxsim_precision@3 | 0.1356 |
| maxsim_precision@5 | 0.0973 |
| maxsim_precision@10 | 0.066 |
| maxsim_recall@1 | 0.2236 |
| maxsim_recall@3 | 0.3314 |
| maxsim_recall@5 | 0.3871 |
| maxsim_recall@10 | 0.5059 |
| **maxsim_ndcg@10** | **0.3643** |
| maxsim_mrr@10 | 0.3467 |
| maxsim_map@100 | 0.322 |
#### Multi Vector Nano BEIR
* Dataset: `NanoBEIR_mean`
* Evaluated with [MultiVectorNanoBEIREvaluator](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#sentence_transformers.multi_vector_encoder.evaluation.MultiVectorNanoBEIREvaluator) with these parameters:
```json
{
"dataset_names": [
"climatefever",
"dbpedia",
"fever",
"fiqa2018",
"hotpotqa",
"msmarco",
"nfcorpus",
"nq",
"quoraretrieval",
"scidocs",
"arguana",
"scifact",
"touche2020"
],
"dataset_id": "sentence-transformers/NanoBEIR-en"
}
```
| Metric | Value |
|:--------------------|:-----------|
| maxsim_accuracy@1 | 0.4177 |
| maxsim_accuracy@3 | 0.5826 |
| maxsim_accuracy@5 | 0.6612 |
| maxsim_accuracy@10 | 0.7476 |
| maxsim_precision@1 | 0.4177 |
| maxsim_precision@3 | 0.2612 |
| maxsim_precision@5 | 0.2037 |
| maxsim_precision@10 | 0.1407 |
| maxsim_recall@1 | 0.2458 |
| maxsim_recall@3 | 0.3745 |
| maxsim_recall@5 | 0.4394 |
| maxsim_recall@10 | 0.5092 |
| **maxsim_ndcg@10** | **0.4601** |
| maxsim_mrr@10 | 0.5221 |
| maxsim_map@100 | 0.3943 |
## Training Details
### Training Dataset
#### msmarco-bm25
* Dataset: [msmarco-bm25](https://huggingface.co/datasets/sentence-transformers/msmarco-bm25) at [ce8a493](https://huggingface.co/datasets/sentence-transformers/msmarco-bm25/tree/ce8a493a65af5e872c3c92f72a89e2e99e175f02)
* Size: 50,000 training samples
* Columns: query, positive, and negative
* Approximate statistics based on the first 100 samples:
| | query | positive | negative |
|:---------|:--------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| type | string | string | string |
| modality | text | text | text |
| details | - min: 5 tokens
- mean: 9.2 tokens
- max: 23 tokens
| - min: 24 tokens
- mean: 85.21 tokens
- max: 234 tokens
| - min: 24 tokens
- mean: 78.79 tokens
- max: 189 tokens
|
* Samples:
| query | positive | negative |
|:---------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| how many days to renew philippine passport to usa | The United States requires non-citizens to keep a foreign passport that is valid for six months beyond their date of departure. If you are in the United States legally, then you can renew your Philippine passport at the consulate general's office in Los Angeles. | How much does it cost to renew a Philippines passport? Philippine Passport Fees for Renewal is P 950 for 15 working days and 1,200 for 7 working days. This is according to the Department of Freign Affairs website. |
| which sexually transmitted diseases can lead to infections inside joint spaces? | Gonorrhea is a sexually transmitted disease (STD) that can infect both men and women. It can cause infections in the genitals, rectum, and throat.It is a very common infection, especially among young people ages 15-24 years.omen with gonorrhea are at risk of developing serious complications from the infection, even if they donât have any symptoms. Symptoms in women can include: 1 Painful or burning sensation when urinating; 2 Increased vaginal discharge; 3 Vaginal bleeding between periods. | STDs and Infertility. Sexually transmitted diseases, STDs, also called sexually transmitted infections or STIs, can cause immediate, annoying symptoms with long-lasting, serious repercussions. Few people realize that these sexually transmitted diseases can cause damage that may eventually lead to infertility. |
| when was eviva amore constructed | Nasher Sculpture Center Press Images Back of the garden, Nasher Sculpture Center; photo by Tim Hursley. Mark di Suvero, Eviva Amore, 2001 in gardens of Nasher Sculpture Center; photo by Tim Hursley. Jaume Plensa, The Long Night (From Ausias March to Vincent Andres Andrés) , estelles, estellés 2007 At Nasher; sculpture center Photo By. tim hursley | Richard Serra, My Curves Are Not Mad, 1987 and Augustus Rodin, Eve, 1881 (cast before 1932) at Nasher Sculpture Center; photo by Tim Hursley. Mark di Suvero, Eviva Amore, 2001 at dusk in gardens of Nasher Sculpture Center; photo by Tim Hursley. Jeremy Strick, Director of the Nasher Sculpture. |
* Loss: [MultiVectorMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/multi_vector_encoder/losses.html#multivectormultiplenegativesrankingloss) with these parameters:
```json
{
"score_metric": "colbert_scores",
"scale": 1.0,
"score_mini_batch_size": null,
"size_average": true,
"gather_across_devices": false
}
```
### Evaluation Dataset
#### msmarco-bm25
* Dataset: [msmarco-bm25](https://huggingface.co/datasets/sentence-transformers/msmarco-bm25) at [ce8a493](https://huggingface.co/datasets/sentence-transformers/msmarco-bm25/tree/ce8a493a65af5e872c3c92f72a89e2e99e175f02)
* Size: 1,000 evaluation samples
* Columns: query, positive, and negative
* Approximate statistics based on the first 100 samples:
| | query | positive | negative |
|:---------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| type | string | string | string |
| modality | text | text | text |
| details | - min: 4 tokens
- mean: 9.15 tokens
- max: 19 tokens
| - min: 39 tokens
- mean: 82.03 tokens
- max: 199 tokens
| - min: 27 tokens
- mean: 81.62 tokens
- max: 157 tokens
|
* Samples:
| query | positive | negative |
|:------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| what is chor means | ⢠CHORE (noun) The noun CHORE has 1 sense: 1. a specific piece of work required to be done as a duty or for a specific fee. Familiarity information: CHORE used as a noun is very rare. | Any two different languages and not just English and other language. Example 1. Chore (pronounced as cHor) means 'a routine task' in English language. Whereas Chor {à¤à¥à¤°} (also pronounced as CHor) means a thief or a burglar in both Hindi and Marathi language. |
| how is gravity measured | The gravity of Earth, which is denoted by g, refers to the acceleration that the Earth imparts to objects on or near its surface due to gravity. In SI units this acceleration is measured in metres per second squared (in symbols, m/s2 or m·sâ2) or equivalently in newtons per kilogram (N/kg or N·kgâ1). | When the wort is first added to the yeast, the specific gravity of the mixture is measured. Later, the specific gravity may be measured again to determine how much alcohol is in the beer, and to know when to stop the fermentation.hen the wort is first added to the yeast, the specific gravity of the mixture is measured. Later, the specific gravity may be measured again to determine how much alcohol is in the beer, and to know when to stop the fermentation. |
| salary of doctor during fellowship | Average fellowship salary and wage. The median expected salary for a Fellowship physician in the United States averages to about $150,353 per annum and an average hourly wage is around $20 per hour. fellowship physician in USA receives an average yearly salary ranging from between $34,225 â $59,542. In addition, a yearly bonus of around $4,888 will be included as part of the annual salary package. | Medical Fellowship Salary. Medical Fellowship average salary is $55,008, median salary is $- with a salary range from $- to $-.Medical Fellowship salaries are collected from government agencies and companies. Each salary is associated with a real job position.Medical Fellowship salary statistics is not exclusive and is for reference only.They are presented as is and updated regularly.edical Fellowship salaries are collected from government agencies and companies. Each salary is associated with a real job position. Medical Fellowship salary statistics is not exclusive and is for reference only. They are presented as is and updated regularly. |
* Loss: [MultiVectorMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/multi_vector_encoder/losses.html#multivectormultiplenegativesrankingloss) with these parameters:
```json
{
"score_metric": "colbert_scores",
"scale": 1.0,
"score_mini_batch_size": null,
"size_average": true,
"gather_across_devices": false
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `per_device_train_batch_size`: 16
- `num_train_epochs`: 1
- `learning_rate`: 3e-05
- `warmup_steps`: 0.05
- `bf16`: True
- `per_device_eval_batch_size`: 16
- `load_best_model_at_end`: True
- `batch_sampler`: no_duplicates
#### All Hyperparameters
Click to expand
- `per_device_train_batch_size`: 16
- `num_train_epochs`: 1
- `max_steps`: -1
- `learning_rate`: 3e-05
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: None
- `warmup_steps`: 0.05
- `optim`: adamw_torch_fused
- `optim_args`: None
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `optim_target_modules`: None
- `gradient_accumulation_steps`: 1
- `average_tokens_across_devices`: True
- `max_grad_norm`: 1.0
- `label_smoothing_factor`: 0.0
- `bf16`: True
- `fp16`: False
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `use_liger_kernel`: False
- `liger_kernel_config`: None
- `use_cache`: False
- `neftune_noise_alpha`: None
- `torch_empty_cache_steps`: None
- `auto_find_batch_size`: False
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `include_num_input_tokens_seen`: no
- `log_level`: passive
- `log_level_replica`: warning
- `disable_tqdm`: False
- `project`: huggingface
- `trackio_space_id`: None
- `trackio_bucket_id`: None
- `trackio_static_space_id`: None
- `per_device_eval_batch_size`: 16
- `prediction_loss_only`: True
- `eval_on_start`: False
- `eval_do_concat_batches`: True
- `eval_use_gather_object`: False
- `eval_accumulation_steps`: None
- `include_for_metrics`: []
- `batch_eval_metrics`: False
- `save_only_model`: False
- `save_on_each_node`: False
- `enable_jit_checkpoint`: False
- `push_to_hub`: False
- `hub_private_repo`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_always_push`: False
- `hub_revision`: None
- `load_best_model_at_end`: True
- `ignore_data_skip`: False
- `restore_callback_states_from_checkpoint`: False
- `full_determinism`: False
- `seed`: 42
- `data_seed`: None
- `use_cpu`: False
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `parallelism_config`: None
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `dataloader_prefetch_factor`: None
- `remove_unused_columns`: True
- `label_names`: None
- `train_sampling_strategy`: random
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `ddp_static_graph`: None
- `ddp_backend`: None
- `ddp_timeout`: 1800
- `fsdp`: None
- `fsdp_config`: None
- `deepspeed`: None
- `debug`: []
- `skip_memory_metrics`: True
- `do_predict`: False
- `resume_from_checkpoint`: None
- `warmup_ratio`: None
- `local_rank`: -1
- `prompts`: None
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional
- `router_mapping`: {}
- `learning_rate_mapping`: {}
Click to expand
| Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_maxsim_ndcg@10 | NanoNQ_maxsim_ndcg@10 | NanoFiQA2018_maxsim_ndcg@10 | NanoBEIR_mean_maxsim_ndcg@10 | NanoClimateFEVER_maxsim_ndcg@10 | NanoDBPedia_maxsim_ndcg@10 | NanoFEVER_maxsim_ndcg@10 | NanoHotpotQA_maxsim_ndcg@10 | NanoNFCorpus_maxsim_ndcg@10 | NanoQuoraRetrieval_maxsim_ndcg@10 | NanoSCIDOCS_maxsim_ndcg@10 | NanoArguAna_maxsim_ndcg@10 | NanoSciFact_maxsim_ndcg@10 | NanoTouche2020_maxsim_ndcg@10 |
|:----------:|:--------:|:-------------:|:---------------:|:--------------------------:|:---------------------:|:---------------------------:|:----------------------------:|:-------------------------------:|:--------------------------:|:------------------------:|:---------------------------:|:---------------------------:|:---------------------------------:|:--------------------------:|:--------------------------:|:--------------------------:|:-----------------------------:|
| -1 | -1 | - | - | 0.1622 | 0.1676 | 0.1806 | 0.1701 | - | - | - | - | - | - | - | - | - | - |
| 0.0102 | 32 | 3.1388 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.0205 | 64 | 3.0199 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.0307 | 96 | 2.6161 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.0410 | 128 | 1.9225 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.0512 | 160 | 1.4234 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.0614 | 192 | 1.3101 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.0717 | 224 | 1.2589 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.0819 | 256 | 1.1922 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.0922 | 288 | 1.1277 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1002 | 313 | - | 1.0385 | 0.4334 | 0.3820 | 0.3084 | 0.3746 | - | - | - | - | - | - | - | - | - | - |
| 0.1024 | 320 | 1.0804 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1126 | 352 | 1.0462 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1229 | 384 | 0.9987 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1331 | 416 | 0.9750 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1434 | 448 | 0.9402 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1536 | 480 | 0.9562 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1638 | 512 | 0.9837 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1741 | 544 | 0.8821 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1843 | 576 | 0.8749 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.1946 | 608 | 0.8840 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2003 | 626 | - | 0.8499 | 0.3980 | 0.3747 | 0.3384 | 0.3703 | - | - | - | - | - | - | - | - | - | - |
| 0.2048 | 640 | 0.8558 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2150 | 672 | 0.8557 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2253 | 704 | 0.8472 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2355 | 736 | 0.8419 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2458 | 768 | 0.7444 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.256 | 800 | 0.7030 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2662 | 832 | 0.6964 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2765 | 864 | 0.7182 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2867 | 896 | 0.7058 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.2970 | 928 | 0.6766 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3005 | 939 | - | 0.6826 | 0.3959 | 0.3808 | 0.2961 | 0.3576 | - | - | - | - | - | - | - | - | - | - |
| 0.3072 | 960 | 0.7021 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3174 | 992 | 0.6570 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3277 | 1024 | 0.6808 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3379 | 1056 | 0.6907 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3482 | 1088 | 0.6609 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3584 | 1120 | 0.6528 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3686 | 1152 | 0.6631 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3789 | 1184 | 0.6654 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3891 | 1216 | 0.6284 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.3994 | 1248 | 0.6762 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4006 | 1252 | - | 0.6361 | 0.3756 | 0.3687 | 0.3029 | 0.3491 | - | - | - | - | - | - | - | - | - | - |
| 0.4096 | 1280 | 0.6682 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4198 | 1312 | 0.6325 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4301 | 1344 | 0.6488 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4403 | 1376 | 0.6196 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4506 | 1408 | 0.6216 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4608 | 1440 | 0.6220 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4710 | 1472 | 0.6139 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4813 | 1504 | 0.6488 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.4915 | 1536 | 0.6492 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5008 | 1565 | - | 0.6224 | 0.4053 | 0.3853 | 0.2963 | 0.3623 | - | - | - | - | - | - | - | - | - | - |
| 0.5018 | 1568 | 0.6268 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.512 | 1600 | 0.6198 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5222 | 1632 | 0.6044 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5325 | 1664 | 0.6817 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5427 | 1696 | 0.6618 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5530 | 1728 | 0.6119 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5632 | 1760 | 0.6654 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5734 | 1792 | 0.5773 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5837 | 1824 | 0.5980 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.5939 | 1856 | 0.6382 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6010 | 1878 | - | 0.6125 | 0.3983 | 0.3761 | 0.3120 | 0.3621 | - | - | - | - | - | - | - | - | - | - |
| 0.6042 | 1888 | 0.5787 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6144 | 1920 | 0.5919 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6246 | 1952 | 0.5871 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6349 | 1984 | 0.6255 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6451 | 2016 | 0.5961 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6554 | 2048 | 0.5441 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6656 | 2080 | 0.6188 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6758 | 2112 | 0.5860 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6861 | 2144 | 0.5706 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.6963 | 2176 | 0.5720 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| **0.7011** | **2191** | **-** | **0.5954** | **0.4944** | **0.4234** | **0.2959** | **0.4046** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** |
| 0.7066 | 2208 | 0.6503 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.7168 | 2240 | 0.5401 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.7270 | 2272 | 0.6104 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.7373 | 2304 | 0.5852 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.7475 | 2336 | 0.5631 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.7578 | 2368 | 0.5845 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.768 | 2400 | 0.5561 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.7782 | 2432 | 0.6066 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.7885 | 2464 | 0.6072 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.7987 | 2496 | 0.5782 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8013 | 2504 | - | 0.5808 | 0.3923 | 0.4085 | 0.3019 | 0.3676 | - | - | - | - | - | - | - | - | - | - |
| 0.8090 | 2528 | 0.6032 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8192 | 2560 | 0.5813 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8294 | 2592 | 0.5351 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8397 | 2624 | 0.5818 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8499 | 2656 | 0.5505 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8602 | 2688 | 0.5446 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8704 | 2720 | 0.5842 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8806 | 2752 | 0.5545 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.8909 | 2784 | 0.5430 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9011 | 2816 | 0.5418 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9014 | 2817 | - | 0.5728 | 0.3911 | 0.3852 | 0.2770 | 0.3511 | - | - | - | - | - | - | - | - | - | - |
| 0.9114 | 2848 | 0.5551 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9216 | 2880 | 0.5842 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9318 | 2912 | 0.5949 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9421 | 2944 | 0.5717 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9523 | 2976 | 0.5669 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9626 | 3008 | 0.5380 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9728 | 3040 | 0.5454 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9830 | 3072 | 0.5609 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0.9933 | 3104 | 0.5555 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 1.0 | 3125 | - | 0.5623 | 0.4073 | 0.4027 | 0.2829 | 0.3643 | - | - | - | - | - | - | - | - | - | - |
| -1 | -1 | - | - | 0.4944 | 0.4234 | 0.2959 | 0.4601 | 0.2404 | 0.4753 | 0.7786 | 0.5334 | 0.2650 | 0.8671 | 0.2627 | 0.3684 | 0.5754 | 0.4015 |
* The bold row denotes the saved checkpoint.