--- language: - en license: mit tags: - sentence-transformers - sparse-encoder - sparse - splade - generated_from_trainer - dataset_size:99000 - loss:SpladeLoss - loss:SparseMultipleNegativesRankingLoss - loss:FlopsLoss base_model: jhu-clsp/ettin-encoder-17m widget: - text: In addition to proof of identity, such as passport, birth certificate or adoption certificate, and proof of your address, or a letter from your university or college confirming your place, and bring it along with your other identity and address verification documents to your local branch. - text: An abiotic factor is a non-living part of an ecosystem that shapes its environment. In a terrestrial ecosystem, examples might include temperature, light, and water. In a marine ecosystem, abiotic factors would include salinity and ocean currents. Abiotic and biotic factors work together to create a unique ecosystem. - text: how many 16 oz bottles of water is a gallon? - text: A vet assistant and vet tech both provide general animal care and assist with the treatment of sick and injured animals. Depending on the setting, a vet assistant may have more administrative responsibilities, while a vet tech may have more clinical responsibilities. - text: The two main stars are Alpha Centauri A and Alpha Centauri B, which form a binary pair. They are an average of 4.3 light-years from Earth. The third star is Proxima Centauri. It is about 4.22 light-years from Earth and is the closest star other than the sun. datasets: - sentence-transformers/gooaq pipeline_tag: feature-extraction library_name: sentence-transformers metrics: - dot_accuracy@1 - dot_accuracy@3 - dot_accuracy@5 - dot_accuracy@10 - dot_precision@1 - dot_precision@3 - dot_precision@5 - dot_precision@10 - dot_recall@1 - dot_recall@3 - dot_recall@5 - dot_recall@10 - dot_ndcg@10 - dot_mrr@10 - dot_map@100 - query_active_dims - query_sparsity_ratio - corpus_active_dims - corpus_sparsity_ratio - avg_flops model-index: - name: splade-ettin-encoder-17m trained on GooAQ results: - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoMSMARCO 256 type: NanoMSMARCO_256 metrics: - type: dot_accuracy@1 value: 0.2 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.34 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.44 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.58 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.2 name: Dot Precision@1 - type: dot_precision@3 value: 0.11333333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.08800000000000001 name: Dot Precision@5 - type: dot_precision@10 value: 0.057999999999999996 name: Dot Precision@10 - type: dot_recall@1 value: 0.2 name: Dot Recall@1 - type: dot_recall@3 value: 0.34 name: Dot Recall@3 - type: dot_recall@5 value: 0.44 name: Dot Recall@5 - type: dot_recall@10 value: 0.58 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.36748190085250804 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.30210317460317454 name: Dot Mrr@10 - type: dot_map@100 value: 0.3224995402831516 name: Dot Map@100 - type: query_active_dims value: 252.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949968233799238 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780179 name: Corpus Sparsity Ratio - type: avg_flops value: 104.76126861572266 name: Avg Flops - type: dot_accuracy@1 value: 0.2 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.34 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.42 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.6 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.2 name: Dot Precision@1 - type: dot_precision@3 value: 0.11333333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.084 name: Dot Precision@5 - type: dot_precision@10 value: 0.06000000000000001 name: Dot Precision@10 - type: dot_recall@1 value: 0.2 name: Dot Recall@1 - type: dot_recall@3 value: 0.34 name: Dot Recall@3 - type: dot_recall@5 value: 0.42 name: Dot Recall@5 - type: dot_recall@10 value: 0.6 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.37143757052695625 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.302047619047619 name: Dot Mrr@10 - type: dot_map@100 value: 0.3209443084622113 name: Dot Map@100 - type: query_active_dims value: 252.10000610351562 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949948378711977 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780179 name: Corpus Sparsity Ratio - type: avg_flops value: 104.66670989990234 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoNFCorpus 256 type: NanoNFCorpus_256 metrics: - type: dot_accuracy@1 value: 0.24 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.32 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.4 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.52 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.24 name: Dot Precision@1 - type: dot_precision@3 value: 0.18666666666666668 name: Dot Precision@3 - type: dot_precision@5 value: 0.18 name: Dot Precision@5 - type: dot_precision@10 value: 0.14400000000000002 name: Dot Precision@10 - type: dot_recall@1 value: 0.007416693841083107 name: Dot Recall@1 - type: dot_recall@3 value: 0.01728407183582485 name: Dot Recall@3 - type: dot_recall@5 value: 0.03449216239510752 name: Dot Recall@5 - type: dot_recall@10 value: 0.04713697363333782 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.16244101089890484 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.3040714285714286 name: Dot Mrr@10 - type: dot_map@100 value: 0.04634431882731895 name: Dot Map@100 - type: query_active_dims value: 255.6199951171875 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949249524476416 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780177 name: Corpus Sparsity Ratio - type: avg_flops value: 101.3379135131836 name: Avg Flops - type: dot_accuracy@1 value: 0.22 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.34 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.38 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.52 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.22 name: Dot Precision@1 - type: dot_precision@3 value: 0.19333333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.18 name: Dot Precision@5 - type: dot_precision@10 value: 0.14 name: Dot Precision@10 - type: dot_recall@1 value: 0.006928888963034326 name: Dot Recall@1 - type: dot_recall@3 value: 0.018744633333151052 name: Dot Recall@3 - type: dot_recall@5 value: 0.030603858996218224 name: Dot Recall@5 - type: dot_recall@10 value: 0.04577448609594998 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.15706743654657598 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.29643650793650794 name: Dot Mrr@10 - type: dot_map@100 value: 0.046568619637846746 name: Dot Map@100 - type: query_active_dims value: 255.6199951171875 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949249524476416 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780177 name: Corpus Sparsity Ratio - type: avg_flops value: 101.26585388183594 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoNQ 256 type: NanoNQ_256 metrics: - type: dot_accuracy@1 value: 0.1 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.32 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.42 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.52 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.1 name: Dot Precision@1 - type: dot_precision@3 value: 0.10666666666666666 name: Dot Precision@3 - type: dot_precision@5 value: 0.084 name: Dot Precision@5 - type: dot_precision@10 value: 0.05800000000000001 name: Dot Precision@10 - type: dot_recall@1 value: 0.09 name: Dot Recall@1 - type: dot_recall@3 value: 0.29 name: Dot Recall@3 - type: dot_recall@5 value: 0.38 name: Dot Recall@5 - type: dot_recall@10 value: 0.51 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.2892021613044471 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.2285793650793651 name: Dot Mrr@10 - type: dot_map@100 value: 0.22850363025154985 name: Dot Map@100 - type: query_active_dims value: 256.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949174078780177 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780176 name: Corpus Sparsity Ratio - type: avg_flops value: 114.13877868652344 name: Avg Flops - type: dot_accuracy@1 value: 0.1 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.28 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.4 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.54 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.1 name: Dot Precision@1 - type: dot_precision@3 value: 0.09333333333333334 name: Dot Precision@3 - type: dot_precision@5 value: 0.08000000000000002 name: Dot Precision@5 - type: dot_precision@10 value: 0.05800000000000001 name: Dot Precision@10 - type: dot_recall@1 value: 0.09 name: Dot Recall@1 - type: dot_recall@3 value: 0.25 name: Dot Recall@3 - type: dot_recall@5 value: 0.36 name: Dot Recall@5 - type: dot_recall@10 value: 0.52 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.29254729188799355 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.2322698412698413 name: Dot Mrr@10 - type: dot_map@100 value: 0.23046052440382125 name: Dot Map@100 - type: query_active_dims value: 256.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949174078780177 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780176 name: Corpus Sparsity Ratio - type: avg_flops value: 114.126953125 name: Avg Flops - task: type: sparse-nano-beir name: Sparse Nano BEIR dataset: name: NanoBEIR mean 256 type: NanoBEIR_mean_256 metrics: - type: dot_accuracy@1 value: 0.18000000000000002 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.32666666666666666 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.42 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.54 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.18000000000000002 name: Dot Precision@1 - type: dot_precision@3 value: 0.13555555555555554 name: Dot Precision@3 - type: dot_precision@5 value: 0.11733333333333335 name: Dot Precision@5 - type: dot_precision@10 value: 0.08666666666666667 name: Dot Precision@10 - type: dot_recall@1 value: 0.0991388979470277 name: Dot Recall@1 - type: dot_recall@3 value: 0.21576135727860826 name: Dot Recall@3 - type: dot_recall@5 value: 0.2848307207983692 name: Dot Recall@5 - type: dot_recall@10 value: 0.3790456578777793 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.27304169101861997 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.27825132275132275 name: Dot Mrr@10 - type: dot_map@100 value: 0.1991158297873401 name: Dot Map@100 - type: query_active_dims value: 254.53999837239584 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949463945685276 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780177 name: Corpus Sparsity Ratio - type: avg_flops value: 105.0486068725586 name: Avg Flops - type: dot_accuracy@1 value: 0.2730612244897959 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.47042386185243334 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.556734693877551 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.6553532182103611 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.2730612244897959 name: Dot Precision@1 - type: dot_precision@3 value: 0.20167451596023023 name: Dot Precision@3 - type: dot_precision@5 value: 0.16646153846153847 name: Dot Precision@5 - type: dot_precision@10 value: 0.12314913657770801 name: Dot Precision@10 - type: dot_recall@1 value: 0.12970938218659442 name: Dot Recall@1 - type: dot_recall@3 value: 0.26896230737703214 name: Dot Recall@3 - type: dot_recall@5 value: 0.3370186758223455 name: Dot Recall@5 - type: dot_recall@10 value: 0.4261361667823346 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.345879757383274 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.3924966360170442 name: Dot Mrr@10 - type: dot_map@100 value: 0.2731986853504099 name: Dot Map@100 - type: query_active_dims value: 249.2619424828763 name: Query Active Dims - type: query_sparsity_ratio value: 0.9950511844329163 name: Query Sparsity Ratio - type: corpus_active_dims value: 254.11144146927373 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949549030839169 name: Corpus Sparsity Ratio - type: avg_flops value: 109.02598571777344 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoClimateFEVER 256 type: NanoClimateFEVER_256 metrics: - type: dot_accuracy@1 value: 0.2 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.38 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.42 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.48 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.2 name: Dot Precision@1 - type: dot_precision@3 value: 0.12666666666666665 name: Dot Precision@3 - type: dot_precision@5 value: 0.092 name: Dot Precision@5 - type: dot_precision@10 value: 0.066 name: Dot Precision@10 - type: dot_recall@1 value: 0.085 name: Dot Recall@1 - type: dot_recall@3 value: 0.17833333333333332 name: Dot Recall@3 - type: dot_recall@5 value: 0.2 name: Dot Recall@5 - type: dot_recall@10 value: 0.26 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.215122111282969 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.29888888888888887 name: Dot Mrr@10 - type: dot_map@100 value: 0.1741526011558945 name: Dot Map@100 - type: query_active_dims value: 256.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949174078780177 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780177 name: Corpus Sparsity Ratio - type: avg_flops value: 129.39976501464844 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoDBPedia 256 type: NanoDBPedia_256 metrics: - type: dot_accuracy@1 value: 0.4 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.66 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.74 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.84 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.4 name: Dot Precision@1 - type: dot_precision@3 value: 0.3733333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.34800000000000003 name: Dot Precision@5 - type: dot_precision@10 value: 0.324 name: Dot Precision@10 - type: dot_recall@1 value: 0.043815756303188326 name: Dot Recall@1 - type: dot_recall@3 value: 0.08533510688544765 name: Dot Recall@3 - type: dot_recall@5 value: 0.13559343564169452 name: Dot Recall@5 - type: dot_recall@10 value: 0.20974054299022687 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.3792423494760472 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.5518571428571429 name: Dot Mrr@10 - type: dot_map@100 value: 0.2586818991199033 name: Dot Map@100 - type: query_active_dims value: 253.9600067138672 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949579096506935 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780179 name: Corpus Sparsity Ratio - type: avg_flops value: 108.96640014648438 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoFEVER 256 type: NanoFEVER_256 metrics: - type: dot_accuracy@1 value: 0.34 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.7 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.78 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.92 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.34 name: Dot Precision@1 - type: dot_precision@3 value: 0.2333333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.15600000000000003 name: Dot Precision@5 - type: dot_precision@10 value: 0.09399999999999999 name: Dot Precision@10 - type: dot_recall@1 value: 0.30666666666666664 name: Dot Recall@1 - type: dot_recall@3 value: 0.6566666666666666 name: Dot Recall@3 - type: dot_recall@5 value: 0.7366666666666666 name: Dot Recall@5 - type: dot_recall@10 value: 0.8766666666666667 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.5965342963229342 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.5312936507936507 name: Dot Mrr@10 - type: dot_map@100 value: 0.4997229758545548 name: Dot Map@100 - type: query_active_dims value: 256.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949174078780177 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780179 name: Corpus Sparsity Ratio - type: avg_flops value: 120.7999038696289 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoFiQA2018 256 type: NanoFiQA2018_256 metrics: - type: dot_accuracy@1 value: 0.18 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.3 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.4 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.44 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.18 name: Dot Precision@1 - type: dot_precision@3 value: 0.14 name: Dot Precision@3 - type: dot_precision@5 value: 0.124 name: Dot Precision@5 - type: dot_precision@10 value: 0.07400000000000001 name: Dot Precision@10 - type: dot_recall@1 value: 0.0821904761904762 name: Dot Recall@1 - type: dot_recall@3 value: 0.17335714285714285 name: Dot Recall@3 - type: dot_recall@5 value: 0.24057936507936506 name: Dot Recall@5 - type: dot_recall@10 value: 0.2906349206349207 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.2236717246585837 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.25316666666666665 name: Dot Mrr@10 - type: dot_map@100 value: 0.18633896011707374 name: Dot Map@100 - type: query_active_dims value: 240.22000122070312 name: Query Active Dims - type: query_sparsity_ratio value: 0.9952307020088011 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780176 name: Corpus Sparsity Ratio - type: avg_flops value: 116.01726531982422 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoHotpotQA 256 type: NanoHotpotQA_256 metrics: - type: dot_accuracy@1 value: 0.58 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.7 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.78 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.84 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.58 name: Dot Precision@1 - type: dot_precision@3 value: 0.32 name: Dot Precision@3 - type: dot_precision@5 value: 0.21599999999999994 name: Dot Precision@5 - type: dot_precision@10 value: 0.12599999999999997 name: Dot Precision@10 - type: dot_recall@1 value: 0.29 name: Dot Recall@1 - type: dot_recall@3 value: 0.48 name: Dot Recall@3 - type: dot_recall@5 value: 0.54 name: Dot Recall@5 - type: dot_recall@10 value: 0.63 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.5600780523888909 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.6586666666666666 name: Dot Mrr@10 - type: dot_map@100 value: 0.49395504821298336 name: Dot Map@100 - type: query_active_dims value: 256.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949174078780177 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780176 name: Corpus Sparsity Ratio - type: avg_flops value: 114.6017074584961 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoQuoraRetrieval 256 type: NanoQuoraRetrieval_256 metrics: - type: dot_accuracy@1 value: 0.16 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.38 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.4 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.46 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.16 name: Dot Precision@1 - type: dot_precision@3 value: 0.12666666666666665 name: Dot Precision@3 - type: dot_precision@5 value: 0.08 name: Dot Precision@5 - type: dot_precision@10 value: 0.04800000000000001 name: Dot Precision@10 - type: dot_recall@1 value: 0.16 name: Dot Recall@1 - type: dot_recall@3 value: 0.354 name: Dot Recall@3 - type: dot_recall@5 value: 0.374 name: Dot Recall@5 - type: dot_recall@10 value: 0.444 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.31465958594721516 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.2770238095238095 name: Dot Mrr@10 - type: dot_map@100 value: 0.2826009380599248 name: Dot Map@100 - type: query_active_dims value: 229.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9954534625158831 name: Query Sparsity Ratio - type: corpus_active_dims value: 234.99563598632812 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9953344259056082 name: Corpus Sparsity Ratio - type: avg_flops value: 119.71977996826172 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoSCIDOCS 256 type: NanoSCIDOCS_256 metrics: - type: dot_accuracy@1 value: 0.34 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.5 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.58 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.72 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.34 name: Dot Precision@1 - type: dot_precision@3 value: 0.22 name: Dot Precision@3 - type: dot_precision@5 value: 0.18 name: Dot Precision@5 - type: dot_precision@10 value: 0.13799999999999998 name: Dot Precision@10 - type: dot_recall@1 value: 0.07266666666666667 name: Dot Recall@1 - type: dot_recall@3 value: 0.13766666666666666 name: Dot Recall@3 - type: dot_recall@5 value: 0.18566666666666667 name: Dot Recall@5 - type: dot_recall@10 value: 0.2836666666666667 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.2672568147859396 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.4424126984126984 name: Dot Mrr@10 - type: dot_map@100 value: 0.19041553956973106 name: Dot Map@100 - type: query_active_dims value: 256.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949174078780177 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780177 name: Corpus Sparsity Ratio - type: avg_flops value: 123.3700942993164 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoArguAna 256 type: NanoArguAna_256 metrics: - type: dot_accuracy@1 value: 0.04 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.32 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.42 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.48 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.04 name: Dot Precision@1 - type: dot_precision@3 value: 0.10666666666666666 name: Dot Precision@3 - type: dot_precision@5 value: 0.084 name: Dot Precision@5 - type: dot_precision@10 value: 0.04800000000000001 name: Dot Precision@10 - type: dot_recall@1 value: 0.04 name: Dot Recall@1 - type: dot_recall@3 value: 0.32 name: Dot Recall@3 - type: dot_recall@5 value: 0.42 name: Dot Recall@5 - type: dot_recall@10 value: 0.48 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.2567073862056186 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.18450000000000003 name: Dot Mrr@10 - type: dot_map@100 value: 0.18984106283244218 name: Dot Map@100 - type: query_active_dims value: 256.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949174078780177 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780177 name: Corpus Sparsity Ratio - type: avg_flops value: 150.7188262939453 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoSciFact 256 type: NanoSciFact_256 metrics: - type: dot_accuracy@1 value: 0.3 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.44 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.64 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.7 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.3 name: Dot Precision@1 - type: dot_precision@3 value: 0.1533333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.14 name: Dot Precision@5 - type: dot_precision@10 value: 0.078 name: Dot Precision@10 - type: dot_recall@1 value: 0.275 name: Dot Recall@1 - type: dot_recall@3 value: 0.42 name: Dot Recall@3 - type: dot_recall@5 value: 0.61 name: Dot Recall@5 - type: dot_recall@10 value: 0.68 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.4730952763241869 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.41774603174603164 name: Dot Mrr@10 - type: dot_map@100 value: 0.40792257335430643 name: Dot Map@100 - type: query_active_dims value: 256.0 name: Query Active Dims - type: query_sparsity_ratio value: 0.9949174078780177 name: Query Sparsity Ratio - type: corpus_active_dims value: 256.0 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949174078780179 name: Corpus Sparsity Ratio - type: avg_flops value: 129.66539001464844 name: Avg Flops - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoTouche2020 256 type: NanoTouche2020_256 metrics: - type: dot_accuracy@1 value: 0.4897959183673469 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.7755102040816326 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.8775510204081632 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.9795918367346939 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.4897959183673469 name: Dot Precision@1 - type: dot_precision@3 value: 0.4217687074829932 name: Dot Precision@3 - type: dot_precision@5 value: 0.4 name: Dot Precision@5 - type: dot_precision@10 value: 0.34693877551020413 name: Dot Precision@10 - type: dot_recall@1 value: 0.03395351363569543 name: Dot Recall@1 - type: dot_recall@3 value: 0.08240644615901016 name: Dot Recall@3 - type: dot_recall@5 value: 0.12813279263988042 name: Dot Recall@5 - type: dot_recall@10 value: 0.21928688511591962 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.38901694962865085 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.6561467444120505 name: Dot Mrr@10 - type: dot_map@100 value: 0.2699778587746352 name: Dot Map@100 - type: query_active_dims value: 216.85714721679688 name: Query Active Dims - type: query_sparsity_ratio value: 0.9956945452029702 name: Query Sparsity Ratio - type: corpus_active_dims value: 255.92410278320312 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949189147319091 name: Corpus Sparsity Ratio - type: avg_flops value: 86.41167449951172 name: Avg Flops --- # splade-ettin-encoder-17m trained on GooAQ This is a [SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [jhu-clsp/ettin-encoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-17m) on the [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) dataset using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 50368-dimensional sparse vector space and can be used for semantic search and sparse retrieval. ## Model Details ### Model Description - **Model Type:** SPLADE Sparse Encoder - **Base model:** [jhu-clsp/ettin-encoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-17m) - **Maximum Sequence Length:** 256 tokens - **Output Dimensionality:** 50368 dimensions - **Similarity Function:** Dot Product - **Supported Modality:** Text - **Training Dataset:** - [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) - **Language:** en - **License:** mit ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder) ### Full Model Architecture ``` SparseEncoder( (0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertForMaskedLM'}) (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 50368}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SparseEncoder # Download from the 🤗 Hub model = SparseEncoder("capemox/splade-ettin-encoder-17m-gooaq") # Run inference sentences = [ 'how close is the closest star?', 'The two main stars are Alpha Centauri A and Alpha Centauri B, which form a binary pair. They are an average of 4.3 light-years from Earth. The third star is Proxima Centauri. It is about 4.22 light-years from Earth and is the closest star other than the sun.', 'One gallon can of paint will cover up to 400 square feet, which is enough to cover a small room like a bathroom. Two gallon cans of paint cover up to 800 square feet, which is enough to cover an average size room. This is the most common amount needed, especially when considering second coat coverage.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 50368] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities) # tensor([[144.9772, 158.3885, 140.6785], # [158.3885, 496.0056, 344.2902], # [140.6785, 344.2902, 528.3322]]) ``` ## Evaluation ### Metrics #### Sparse Information Retrieval * Datasets: `NanoMSMARCO_256`, `NanoNFCorpus_256`, `NanoNQ_256`, `NanoClimateFEVER_256`, `NanoDBPedia_256`, `NanoFEVER_256`, `NanoFiQA2018_256`, `NanoHotpotQA_256`, `NanoMSMARCO_256`, `NanoNFCorpus_256`, `NanoNQ_256`, `NanoQuoraRetrieval_256`, `NanoSCIDOCS_256`, `NanoArguAna_256`, `NanoSciFact_256` and `NanoTouche2020_256` * Evaluated with [SparseInformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseInformationRetrievalEvaluator) with these parameters: ```json { "max_active_dims": 256 } ``` | Metric | NanoMSMARCO_256 | NanoNFCorpus_256 | NanoNQ_256 | NanoClimateFEVER_256 | NanoDBPedia_256 | NanoFEVER_256 | NanoFiQA2018_256 | NanoHotpotQA_256 | NanoQuoraRetrieval_256 | NanoSCIDOCS_256 | NanoArguAna_256 | NanoSciFact_256 | NanoTouche2020_256 | |:----------------------|:----------------|:-----------------|:-----------|:---------------------|:----------------|:--------------|:-----------------|:-----------------|:-----------------------|:----------------|:----------------|:----------------|:-------------------| | dot_accuracy@1 | 0.2 | 0.22 | 0.1 | 0.2 | 0.4 | 0.34 | 0.18 | 0.58 | 0.16 | 0.34 | 0.04 | 0.3 | 0.4898 | | dot_accuracy@3 | 0.34 | 0.34 | 0.28 | 0.38 | 0.66 | 0.7 | 0.3 | 0.7 | 0.38 | 0.5 | 0.32 | 0.44 | 0.7755 | | dot_accuracy@5 | 0.42 | 0.38 | 0.4 | 0.42 | 0.74 | 0.78 | 0.4 | 0.78 | 0.4 | 0.58 | 0.42 | 0.64 | 0.8776 | | dot_accuracy@10 | 0.6 | 0.52 | 0.54 | 0.48 | 0.84 | 0.92 | 0.44 | 0.84 | 0.46 | 0.72 | 0.48 | 0.7 | 0.9796 | | dot_precision@1 | 0.2 | 0.22 | 0.1 | 0.2 | 0.4 | 0.34 | 0.18 | 0.58 | 0.16 | 0.34 | 0.04 | 0.3 | 0.4898 | | dot_precision@3 | 0.1133 | 0.1933 | 0.0933 | 0.1267 | 0.3733 | 0.2333 | 0.14 | 0.32 | 0.1267 | 0.22 | 0.1067 | 0.1533 | 0.4218 | | dot_precision@5 | 0.084 | 0.18 | 0.08 | 0.092 | 0.348 | 0.156 | 0.124 | 0.216 | 0.08 | 0.18 | 0.084 | 0.14 | 0.4 | | dot_precision@10 | 0.06 | 0.14 | 0.058 | 0.066 | 0.324 | 0.094 | 0.074 | 0.126 | 0.048 | 0.138 | 0.048 | 0.078 | 0.3469 | | dot_recall@1 | 0.2 | 0.0069 | 0.09 | 0.085 | 0.0438 | 0.3067 | 0.0822 | 0.29 | 0.16 | 0.0727 | 0.04 | 0.275 | 0.034 | | dot_recall@3 | 0.34 | 0.0187 | 0.25 | 0.1783 | 0.0853 | 0.6567 | 0.1734 | 0.48 | 0.354 | 0.1377 | 0.32 | 0.42 | 0.0824 | | dot_recall@5 | 0.42 | 0.0306 | 0.36 | 0.2 | 0.1356 | 0.7367 | 0.2406 | 0.54 | 0.374 | 0.1857 | 0.42 | 0.61 | 0.1281 | | dot_recall@10 | 0.6 | 0.0458 | 0.52 | 0.26 | 0.2097 | 0.8767 | 0.2906 | 0.63 | 0.444 | 0.2837 | 0.48 | 0.68 | 0.2193 | | **dot_ndcg@10** | **0.3714** | **0.1571** | **0.2925** | **0.2151** | **0.3792** | **0.5965** | **0.2237** | **0.5601** | **0.3147** | **0.2673** | **0.2567** | **0.4731** | **0.389** | | dot_mrr@10 | 0.302 | 0.2964 | 0.2323 | 0.2989 | 0.5519 | 0.5313 | 0.2532 | 0.6587 | 0.277 | 0.4424 | 0.1845 | 0.4177 | 0.6561 | | dot_map@100 | 0.3209 | 0.0466 | 0.2305 | 0.1742 | 0.2587 | 0.4997 | 0.1863 | 0.494 | 0.2826 | 0.1904 | 0.1898 | 0.4079 | 0.27 | | query_active_dims | 252.1 | 255.62 | 256.0 | 256.0 | 253.96 | 256.0 | 240.22 | 256.0 | 229.0 | 256.0 | 256.0 | 256.0 | 216.8571 | | query_sparsity_ratio | 0.995 | 0.9949 | 0.9949 | 0.9949 | 0.995 | 0.9949 | 0.9952 | 0.9949 | 0.9955 | 0.9949 | 0.9949 | 0.9949 | 0.9957 | | corpus_active_dims | 256.0 | 256.0 | 256.0 | 256.0 | 256.0 | 256.0 | 256.0 | 256.0 | 234.9956 | 256.0 | 256.0 | 256.0 | 255.9241 | | corpus_sparsity_ratio | 0.9949 | 0.9949 | 0.9949 | 0.9949 | 0.9949 | 0.9949 | 0.9949 | 0.9949 | 0.9953 | 0.9949 | 0.9949 | 0.9949 | 0.9949 | | avg_flops | 104.6667 | 101.2659 | 114.127 | 129.3998 | 108.9664 | 120.7999 | 116.0173 | 114.6017 | 119.7198 | 123.3701 | 150.7188 | 129.6654 | 86.4117 | #### Sparse Nano BEIR * Dataset: `NanoBEIR_mean_256` * Evaluated with [SparseNanoBEIREvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseNanoBEIREvaluator) with these parameters: ```json { "dataset_names": [ "msmarco", "nfcorpus", "nq" ], "dataset_id": "sentence-transformers/NanoBEIR-en", "max_active_dims": 256 } ``` | Metric | Value | |:----------------------|:----------| | dot_accuracy@1 | 0.18 | | dot_accuracy@3 | 0.3267 | | dot_accuracy@5 | 0.42 | | dot_accuracy@10 | 0.54 | | dot_precision@1 | 0.18 | | dot_precision@3 | 0.1356 | | dot_precision@5 | 0.1173 | | dot_precision@10 | 0.0867 | | dot_recall@1 | 0.0991 | | dot_recall@3 | 0.2158 | | dot_recall@5 | 0.2848 | | dot_recall@10 | 0.379 | | **dot_ndcg@10** | **0.273** | | dot_mrr@10 | 0.2783 | | dot_map@100 | 0.1991 | | query_active_dims | 254.54 | | query_sparsity_ratio | 0.9949 | | corpus_active_dims | 256.0 | | corpus_sparsity_ratio | 0.9949 | | avg_flops | 105.0486 | #### Sparse Nano BEIR * Dataset: `NanoBEIR_mean_256` * Evaluated with [SparseNanoBEIREvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseNanoBEIREvaluator) with these parameters: ```json { "dataset_names": [ "climatefever", "dbpedia", "fever", "fiqa2018", "hotpotqa", "msmarco", "nfcorpus", "nq", "quoraretrieval", "scidocs", "arguana", "scifact", "touche2020" ], "dataset_id": "sentence-transformers/NanoBEIR-en", "max_active_dims": 256 } ``` | Metric | Value | |:----------------------|:-----------| | dot_accuracy@1 | 0.2731 | | dot_accuracy@3 | 0.4704 | | dot_accuracy@5 | 0.5567 | | dot_accuracy@10 | 0.6554 | | dot_precision@1 | 0.2731 | | dot_precision@3 | 0.2017 | | dot_precision@5 | 0.1665 | | dot_precision@10 | 0.1231 | | dot_recall@1 | 0.1297 | | dot_recall@3 | 0.269 | | dot_recall@5 | 0.337 | | dot_recall@10 | 0.4261 | | **dot_ndcg@10** | **0.3459** | | dot_mrr@10 | 0.3925 | | dot_map@100 | 0.2732 | | query_active_dims | 249.2619 | | query_sparsity_ratio | 0.9951 | | corpus_active_dims | 254.1114 | | corpus_sparsity_ratio | 0.995 | | avg_flops | 109.026 | ## Training Details ### Training Dataset #### gooaq * Dataset: [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) at [b089f72](https://huggingface.co/datasets/sentence-transformers/gooaq/tree/b089f728748a068b7bc5234e5bcf5b25e3c8279c) * Size: 99,000 training samples * Columns: question and answer * Approximate statistics based on the first 1000 samples: | | question | answer | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | question | answer | |:-----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | what are the 5 characteristics of a star? | Key Concept: Characteristics used to classify stars include color, temperature, size, composition, and brightness. | | are copic markers alcohol ink? | Copic Ink is alcohol-based and flammable. Keep away from direct sunlight and extreme temperatures. | | what is the difference between appellate term and appellate division? | Appellate terms An appellate term is an intermediate appellate court that hears appeals from the inferior courts within their designated counties or judicial districts, and are intended to ease the workload on the Appellate Division and provide a less expensive forum closer to the people. | * Loss: [SpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters: ```json { "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)", "document_regularizer_weight": 3e-05, "query_regularizer_weight": 5e-05 } ``` ### Evaluation Dataset #### gooaq * Dataset: [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) at [b089f72](https://huggingface.co/datasets/sentence-transformers/gooaq/tree/b089f728748a068b7bc5234e5bcf5b25e3c8279c) * Size: 1,000 evaluation samples * Columns: question and answer * Approximate statistics based on the first 1000 samples: | | question | answer | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | question | answer | |:-----------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | should you take ibuprofen with high blood pressure? | In general, people with high blood pressure should use acetaminophen or possibly aspirin for over-the-counter pain relief. Unless your health care provider has said it's OK, you should not use ibuprofen, ketoprofen, or naproxen sodium. If aspirin or acetaminophen doesn't help with your pain, call your doctor. | | how old do you have to be to work in sc? | The general minimum age of employment for South Carolina youth is 14, although the state allows younger children who are performers to work in show business. If their families are agricultural workers, children younger than age 14 may also participate in farm labor. | | how to write a topic proposal for a research paper? | ['Write down the main topic of your paper. ... ', 'Write two or three short sentences under the main topic that explain why you chose that topic. ... ', 'Write a thesis sentence that states the angle and purpose of your research paper. ... ', 'List the items you will cover in the body of the paper that support your thesis statement.'] | * Loss: [SpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters: ```json { "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)", "document_regularizer_weight": 3e-05, "query_regularizer_weight": 5e-05 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `per_device_train_batch_size`: 32 - `num_train_epochs`: 1 - `learning_rate`: 2e-05 - `bf16`: True - `per_device_eval_batch_size`: 32 - `load_best_model_at_end`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `per_device_train_batch_size`: 32 - `num_train_epochs`: 1 - `max_steps`: -1 - `learning_rate`: 2e-05 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: None - `warmup_steps`: 0 - `optim`: adamw_torch_fused - `optim_args`: None - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `optim_target_modules`: None - `gradient_accumulation_steps`: 1 - `average_tokens_across_devices`: True - `max_grad_norm`: 1.0 - `label_smoothing_factor`: 0.0 - `bf16`: True - `fp16`: False - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `use_liger_kernel`: False - `liger_kernel_config`: None - `use_cache`: False - `neftune_noise_alpha`: None - `torch_empty_cache_steps`: None - `auto_find_batch_size`: False - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `include_num_input_tokens_seen`: no - `log_level`: passive - `log_level_replica`: warning - `disable_tqdm`: False - `project`: huggingface - `trackio_space_id`: trackio - `per_device_eval_batch_size`: 32 - `prediction_loss_only`: True - `eval_on_start`: False - `eval_do_concat_batches`: True - `eval_use_gather_object`: False - `eval_accumulation_steps`: None - `include_for_metrics`: [] - `batch_eval_metrics`: False - `save_only_model`: False - `save_on_each_node`: False - `enable_jit_checkpoint`: False - `push_to_hub`: False - `hub_private_repo`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_always_push`: False - `hub_revision`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `restore_callback_states_from_checkpoint`: False - `full_determinism`: False - `seed`: 42 - `data_seed`: None - `use_cpu`: False - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `dataloader_prefetch_factor`: None - `remove_unused_columns`: True - `label_names`: None - `train_sampling_strategy`: random - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `ddp_backend`: None - `ddp_timeout`: 1800 - `fsdp`: [] - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `deepspeed`: None - `debug`: [] - `skip_memory_metrics`: True - `do_predict`: False - `resume_from_checkpoint`: None - `warmup_ratio`: None - `local_rank`: -1 - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_256_dot_ndcg@10 | NanoNFCorpus_256_dot_ndcg@10 | NanoNQ_256_dot_ndcg@10 | NanoBEIR_mean_256_dot_ndcg@10 | NanoClimateFEVER_256_dot_ndcg@10 | NanoDBPedia_256_dot_ndcg@10 | NanoFEVER_256_dot_ndcg@10 | NanoFiQA2018_256_dot_ndcg@10 | NanoHotpotQA_256_dot_ndcg@10 | NanoQuoraRetrieval_256_dot_ndcg@10 | NanoSCIDOCS_256_dot_ndcg@10 | NanoArguAna_256_dot_ndcg@10 | NanoSciFact_256_dot_ndcg@10 | NanoTouche2020_256_dot_ndcg@10 | |:----------:|:--------:|:-------------:|:---------------:|:---------------------------:|:----------------------------:|:----------------------:|:-----------------------------:|:--------------------------------:|:---------------------------:|:-------------------------:|:----------------------------:|:----------------------------:|:----------------------------------:|:---------------------------:|:---------------------------:|:---------------------------:|:------------------------------:| | 0.0323 | 100 | 954.7106 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.0646 | 200 | 17.8241 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.0970 | 300 | 4.5136 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.1293 | 400 | 2.5427 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.1616 | 500 | 1.4733 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.1939 | 600 | 1.0940 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.1972 | 610 | - | 0.8522 | 0.1440 | 0.0453 | 0.0771 | 0.0888 | - | - | - | - | - | - | - | - | - | - | | 0.2262 | 700 | 0.7541 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.2586 | 800 | 0.7425 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.2909 | 900 | 0.5966 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.3232 | 1000 | 0.5606 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.3555 | 1100 | 0.5440 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.3878 | 1200 | 0.4032 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.3943 | 1220 | - | 0.4165 | 0.2494 | 0.0613 | 0.2065 | 0.1724 | - | - | - | - | - | - | - | - | - | - | | 0.4202 | 1300 | 0.3995 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.4525 | 1400 | 0.2976 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.4848 | 1500 | 0.2971 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.5171 | 1600 | 0.2716 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.5495 | 1700 | 0.2577 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.5818 | 1800 | 0.2370 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.5915 | 1830 | - | 0.2104 | 0.3406 | 0.1173 | 0.2414 | 0.2331 | - | - | - | - | - | - | - | - | - | - | | 0.6141 | 1900 | 0.2360 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.6464 | 2000 | 0.2238 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.6787 | 2100 | 0.2237 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.7111 | 2200 | 0.2162 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.7434 | 2300 | 0.2044 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.7757 | 2400 | 0.2202 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.7886 | 2440 | - | 0.1736 | 0.3932 | 0.1545 | 0.2717 | 0.2731 | - | - | - | - | - | - | - | - | - | - | | 0.8080 | 2500 | 0.1672 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.8403 | 2600 | 0.2122 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.8727 | 2700 | 0.1704 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.9050 | 2800 | 0.1870 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.9373 | 2900 | 0.1671 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.9696 | 3000 | 0.1386 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | **0.9858** | **3050** | **-** | **0.17** | **0.3714** | **0.1571** | **0.2925** | **0.2737** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | | 1.0 | 3094 | - | 0.1706 | 0.3675 | 0.1624 | 0.2892 | 0.2730 | - | - | - | - | - | - | - | - | - | - | | -1 | -1 | - | - | 0.3714 | 0.1571 | 0.2925 | 0.3459 | 0.2151 | 0.3792 | 0.5965 | 0.2237 | 0.5601 | 0.3147 | 0.2673 | 0.2567 | 0.4731 | 0.3890 | * The bold row denotes the saved checkpoint. ### Training Time - **Training**: 7.7 minutes - **Evaluation**: 2.0 minutes - **Total**: 9.7 minutes ### Framework Versions - Python: 3.12.3 - Sentence Transformers: 5.4.1 - Transformers: 5.5.4 - PyTorch: 2.11.0+cu130 - Accelerate: 1.13.0 - Datasets: 4.8.4 - Tokenizers: 0.22.2 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### SpladeLoss ```bibtex @misc{formal2022distillationhardnegativesampling, title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective}, author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant}, year={2022}, eprint={2205.04733}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2205.04733}, } ``` #### SparseMultipleNegativesRankingLoss ```bibtex @misc{oord2019representationlearningcontrastivepredictive, title={Representation Learning with Contrastive Predictive Coding}, author={Aaron van den Oord and Yazhe Li and Oriol Vinyals}, year={2019}, eprint={1807.03748}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/1807.03748}, } ``` #### FlopsLoss ```bibtex @article{paria2020minimizing, title={Minimizing flops to learn efficient sparse representations}, author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s}, journal={arXiv preprint arXiv:2004.05665}, year={2020} } ```