dgwon/e5-small-v2.nfcorpus.meta-llama-Llama-3.1-8B-Instruct

SentenceTransformer based on intfloat/e5-small-v2

This is a sentence-transformers model finetuned from intfloat/e5-small-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/e5-small-v2
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'query: What Concentration of IP6 Blocks Blood Vessel Growth?',
    'passsage: A significant anticancer activity of the naturally occurring carbohydrate inositol hexaphosphate (IP(6)) has been reported against numerous cancer models. Since tumors require angiogenesis for growth and metastasis, we hypothesize that IP(6) reduces tumor growth by inhibiting angiogenesis. Because angiogenesis depends on the interaction between endothelial and tumor cells, we investigated the effect of IP(6) on both. IP(6) inhibited the proliferation and induced the differentiation of endothelial cells in vitro; the growth of bovine aortic endothelial cells (BAECs) evaluated by MTT proliferation assay was inhibited in a dose-dependent manner (IC(50) = 0.74 mM). The combination of IP(6) and vasostatin, a calreticulin fragment with anti-angiogenic activity, was synergistically superior in growth inhibition than either compound. IP(6) inhibited human umbilical vein endothelial cell (HUVEC) tube formation (in vitro capillary differentiation) on a reconstituted extracellular matrix, Matrigel, and disrupted pre-formed tubes. IP(6) significantly reduced basic fibroblast growth factor (bFGF)-induced vessel formation (P < 0.01) in vivo in Matrigel plug assay. Exposure of HepG2, a human hepatoma cell line, to IP(6) for 8 h, resulted in a dose-dependent decrease in the mRNA levels of vascular endothelial growth factor (VEGF), as assessed by RT-PCR. IP(6) treatment of HepG2 cells for 24 h also significantly reduced the VEGF protein levels in conditioned medium, in a concentration-dependent manner (P = 0.012). Thus, IP(6) has an inhibitory effect on induced angiogenesis.',
    'passsage: Twenty three adults ingested 203Pb as lead acetate on the 12th hour of a 19 h fast. Retention measured 7 days later in a whole-body counter was 61% and whole-body turnover rates suggested that initial uptake had been considerably greater. Balanced meals eaten with 203Pb reduced lead uptake to 4% and the influence of the food lasted for up to 3 h. The effects of phytate, ethylene-diaminetetra acetate (EDTA), caffeine, alcohol, glucose, a liquid meal and a light snack were tested separately with intermediate results. The effect of a meal was probably largely due to its content of calcium and phosphate salts but lead uptake was probably further reduced by phytate which is plentiful in whole cereals and it was probably increased by a factor in milk. Uptake with skimmed milk was the same as with whole milk and we suggested that the factor was not fat. Comestibles with low mineral and phytate contents reduced lead uptake by intermediate amounts, possibly by stimulation of digestive secretions. The avid uptake of lead during a fast, the large reduction of lead uptake with meals and the likelihood of variations in gastric-emptying rates and dietary habits may be major causes of variation in body burdens of lead in the population.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.6574, -0.0130],
#         [ 0.6574,  1.0000, -0.1338],
#         [-0.0130, -0.1338,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 26,136 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 9 tokens mean: 13.48 tokens max: 25 tokens	min: 25 tokens mean: 336.45 tokens max: 512 tokens

• Samples:

  anchor	positive
  query: What Are the Common Causes of Dry Eyes?	passsage: The report of the Epidemiology Subcommittee of the 2007 Dry Eye WorkShop summarizes current knowledge on the epidemiology of dry eye disease, providing prevalence and incidence data from various populations. It stresses the need to expand epidemiological studies to additional geographic regions, to incorporate multiple races and ethnicities in future studies, and to build a consensus on dry eye diagnostic criteria for epidemiological studies. Recommendations are made regarding several characteristics of dry eye questionnaires that might be suitable for use in epidemiological studies and randomized controlled clinical trials. Risk factors for dry eye and morbidity of the disease are identified, and the impact of dry eye disease on quality of life and visual function are outlined. Suggestions are made for further prospective research that would lead to improvement of both eye and general public health.
  query: Do Community Gardens Improve Eating Habits?	passsage: Objectives. We considered the relationship between an urban adult population's fruit and vegetable consumption and several selected social and psychological processes, beneficial aesthetic experiences, and garden participation. Methods. We conducted a population-based survey representing 436 residents across 58 block groups in Denver, Colorado, from 2006 to 2007. We used multilevel statistical models to evaluate the survey data. Results. Neighborhood aesthetics, social involvement, and community garden participation were significantly associated with fruit and vegetable intake. Community gardeners consumed fruits and vegetables 5.7 times per day, compared with home gardeners (4.6 times per day) and nongardeners (3.9 times per day). Moreover, 56% of community gardeners met national recommendations to consume fruits and vegetables at least 5 times per day, compared with 37% of home gardeners and 25% of nongardeners. Conclusions. Our study results shed light on neighborhood proc...
  query: Does Eating Tomatoes Help Prevent Heart Attacks?	passsage: Cardiovascular disease (CVD) is the leading cause of death worldwide. Healthy eating is among its safeguards, especially the daily intake of fruits and vegetables. In this context it has been shown that tomato (Solanum lycopersicum) presents antiplatelet activity. In the present study, we evaluated in vitro antiplatelet activity of fresh hybrid tomato process (nine hybrids: Apt 410, H 9888, Bos 8066, Sun 6366, AB3, HMX 7883, H 9665, H 7709, and H 9997), paste and its by-product of industrial processes (pomace). We assessed antiplatelet activity ex vivo and bleeding time in rats that ingested 0.1 and 1.0 g/kg of pomace each day. In studies in vitro, no significant differences in antiplatelet activity was observed in fresh tomato hybrids. Furthermore, the agro-industrial process did not affect the antiplatelet activity of paste and pomace. Likewise, pomace intake of 1.0 g/kg per day prolonged bleeding time and reduced ex vivo platelet aggregation in rats. The data obtained indi...

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 16,
      "gather_across_devices": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 2,904 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 9 tokens mean: 13.5 tokens max: 25 tokens	min: 23 tokens mean: 336.25 tokens max: 512 tokens

• Samples:

  anchor	positive
  query: What's the Best Way to Clean a Head of Lettuce?	passsage: Much effort has been focused on sanitation of fresh produce at the commercial level; however, few options are available to the consumer. The purpose of this study was to determine the efficacy of different cleaning methods in reducing bacterial contamination on fresh produce in a home setting. Lettuce, broccoli, apples, and tomatoes were inoculated with Listeria innocua and then subjected to combinations of the following cleaning procedures: (i) soak for 2 min in tap water, Veggie Wash solution, 5% vinegar solution, or 13% lemon solution and (ii) rinse under running tap water, rinse and rub under running tap water, brush under running tap water, or wipe with wet/dry paper towel. Presoaking in water before rinsing significantly reduced bacteria in apples, tomatoes, and lettuce, but not in broccoli. Wiping apples and tomatoes with wet or dry paper towel showed lower bacterial reductions compared with soaking and rinsing procedures. Blossom ends of apples were more contaminated ...
  query: Does Eating Carbs Improve Premenstrual Depression?	passsage: We examined the occurrence and coincidence of depressed mood and excessive carbohydrate intake in 19 patients who claimed to suffer from severe premenstrual syndrome and in nine control subjects, all as inpatients, during the early follicular and late luteal phases of their menstrual cycles. Mood was assessed with the Hamilton Depression Scale and an addendum that evaluated fatigue, sociability, appetite, and carbohydrate craving. Calorie and nutrient intakes were measured directly. The subjects with premenstrual syndrome significantly increased calorie intake during the late luteal phase (from 1892 +/- 104 to 2395 +/- 93 kcal, mean +/- SEM); carbohydrate intake increased by 24% from meals and by 43% from snacks. Protein intake failed to change, whereas intake of fat, a fixed constituent of all of the test foods, rose in proportion to calorie intake. The Hamilton Depression Scale and addendum scores rose from 2.0 +/- 0.5 to 21.2 +/- 0.8 (Hamilton Scale) and from 0.5 +/- 0.5 t...
  query: Is Bacon Linked to Bladder Cancer?	passsage: BACKGROUND: Nitrosamines, which are known bladder carcinogens, or their precursors are found in certain meat items, and concentrations of these compounds are especially high in bacon. Only 3 cohort studies, all with <100 case subjects, have examined the relation between meat intake and bladder cancer, and few studies have examined the relation of different meat types with bladder cancer. OBJECTIVE: The aim was to examine the association between specific meat items and bladder cancer in 2 large prospective studies. DESIGN: We analyzed data from 2 cohorts with up to 22 y of follow-up and 808 incident bladder cancer cases. Detailed data on meat were obtained from multiple food-frequency questionnaires administered over time. Multivariate relative risks (RRs) and 95% CIs were estimated by using Cox proportional hazards models with control for potential confounders, including detailed smoking history. RESULTS: Men and women with a high intake of bacon (>/=5 servings/wk) had an ele...

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 16,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 30
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 30
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss
1.0	205	1.5499	0.0180
2.0	410	0.2469	0.0116
3.0	615	0.1838	0.0087
4.0	820	0.1409	0.0083
5.0	1025	0.1183	0.0074
6.0	1230	0.0996	0.0069
7.0	1435	0.085	0.0064
8.0	1640	0.0768	0.0066
9.0	1845	0.0722	0.0062
10.0	2050	0.0695	0.0062
11.0	2255	0.0628	0.0061
12.0	2460	0.0593	0.0069
13.0	2665	0.0562	0.0060
14.0	2870	0.0541	0.0063
15.0	3075	0.0539	0.0061
16.0	3280	0.0513	0.0058
17.0	3485	0.0479	0.0062
18.0	3690	0.0493	0.0068
19.0	3895	0.0464	0.0066

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.8.0+cu126
• Accelerate: 1.11.0
• Datasets: 4.0.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}