viethoangtranduong/gemma-embedding-300m-sleepqa

SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/embeddinggemma-300m
• Maximum Sequence Length: 2048 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("viethoangtranduong/gemma-attempt1")
# Run inference
queries = [
    "what does good sleep hygiene describe?",
]
documents = [
    'good sleep hygiene describes a set of habits that promote healthy sleep. incorporating these habits into your daily routine can help improve the quality of sleep you get each night. if stress is keeping you up, relaxation methods can help relieve your stress during the day. meditation and yoga can both reduce stress and anxiety. you can also incorporate relaxation techniques into your bedtime routine, such as progressive muscle relaxation, deep breathing exercises, or aromatherapy. if your anxiety is interfering with your quality of life, it may be beneficial to consult a mental health professional.',
    'researchers found later start times had the greatest impact on high school students, who reported start times after 8:30am led to an average of 43-52 minutes of additional sleep per night. the survey data showed that while weekday bedtimes remained largely consistent (with a shift of about 14 minutes later), weekday wake times were as much as 60 minutes later than prior to the start time change. on average, high schoolers gained nearly four additional hours of sleep per week, which also helped decrease weekend oversleep to make-up for the sleep loss during their week. middle school students also reported an increase in sleep duration, though not as dramatic as high schoolers.',
    'episodes typically last a couple of minutes and resolve on their own. narcolepsy is a sleep disorder characterized by excessive daytime sleepiness, sleep paralysis, hallucinations, and, in some cases, cataplexy. there are two major types of narcolepsy: type 1 and type 2, differentiated by whether or not a person experiences cataplexy. people diagnosed with type 1 narcolepsy experience episodes of cataplexy, while people with type 2 narcolepsy do not. for people with type 1 narcolepsy, episodes of cataplexy typically begin after the onset of excessive sleepiness. while type 1 and type 2 narcolepsy both have the name narcolepsy in it, the cause for type 1 narcolepsy is well understood (loss of a neurotransmitter, orexin), whereas the cause for type 2 narcolepsy is not well understood.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[     0.6632,     -0.0375,      0.0004]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 3,200 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 12.38 tokens max: 31 tokens	min: 77 tokens mean: 131.04 tokens max: 221 tokens	min: 79 tokens mean: 130.16 tokens max: 220 tokens

• Samples:

  anchor	positive	negative
  what two categories are the four stages of sleep broken up into?	both measures can provide insight into how well a person is sleeping, as a longer sleep latency can lead to a lower sleep efficiency. however, having a high sleep latency, or taking a long time to fall asleep, is not the only factor that impacts sleep efficiency. a person can also have a low sleep efficiency due to waking up throughout the night or early in the morning. each night, a person cycles through multiple stages of sleep. these four stages are broken up into two categories: rapid eye movement (rem) and non-rapid eye movement (nrem) sleep.	bipap machines are also sometimes prescribed to people who have sleep apnea as well as severe obesity or certain other health conditions, like chronic obstructive pulmonary disease and hypoventilation. automatic positive airway pressure machines, often called apap or auto-cpap machines, also work similarly to cpap machines in that they connect to a sleeper's face using a tube and mask, then push out air to keep the airway open. apap machines differ from cpap machines with their ability to automatically adjust the pressure of air they release. this ability can make the machine more comfortable than a cpap, since sleepers need different levels of pressure at different times of night, depending on their sleeping position, sleep stage, or amount of congestion.
  what do internal nasal dilators work better than?	internal nasal dilators may work better than nasal strips. also known as myofunctional therapy, certain mouth exercises are designed to improve the function of the mouth and upper respiratory tract to reduce snoring. for example, these movements may include pronouncing vowel sounds, swallowing and chewing motions, and moving the tongue against the teeth to strengthen the surrounding muscles. you can learn how to perform these exercises on your own using videos and apps. however, it is unclear if self-directed mouth exercises are as effective as those performed under the direction of a healthcare provider such as a speech therapist.	people with narcolepsy face an increased risk of cardiovascular and metabolic conditions such as high blood pressure, high cholesterol, obesity, and diabetes. one reason for the increased prevalence of these conditions in people with narcolepsy may be the multiple roles orexins play in the body. in addition to causing issues in maintaining sleep-wake cycles, a loss of orexin-producing neurons can also affect physical activity and weight gain, nighttime blood pressure, and the buildup of plaque in the arteries - all potential causes of heart disease. narcolepsy is also associated with psychiatric disorders, including attention deficit hyperactivity disorder (adhd), anxiety, eating disorders, depression, and schizophrenia. researchers hypothesize that lifestyle changes and impairment caused by the symptoms of narcolepsy may lead to the development of psychiatric disorders, or that both narcolepsy and psychiatric disorders share similar causes. while the impact of narcolepsy on mental hea...
  how many americans struggle with chronic sleep and wakefulness disorders?	between 50 and 70 million american adults struggle with chronic sleep and wakefulness disorders. dyssomnia refers to the collection of sleep disorders that negatively impact the quantity and quality of sleep. with dyssomnia, you might struggle to fall asleep at night, which is a symptom of insomnia, or feel the need to sleep excessively, which is called hypersomnolence. additionally, common symptoms of dyssomnia include excessive daytime sleepiness or feeling unrested when you wake up in the morning. more specific symptoms vary depending on the type of sleep disorder you are experiencing. unlike parasomnias, dyssomnias generally do not involve irregular physical actions while asleep.	sleep talking that is offensive, sexually explicit, or reveals secrets may cause embarrassment or shame; however, people are very rarely aware that they are talking in their sleep at the time and typically have no recollection of the episodes when they wake up. episodes of sleep talking tend to be short, and they rarely involve long or drawn-out conversations. the total episode may involve only a handful of words or a few sentences. research is mixed about the source of the content during episodes of sleep talking. speech may not have any clear connection to a person's life, recent events, or prior conversations.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 800 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 800 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 12.7 tokens max: 30 tokens	min: 79 tokens mean: 130.2 tokens max: 238 tokens	min: 79 tokens mean: 130.1 tokens max: 217 tokens

• Samples:

  anchor	positive	negative
  what do people with kls experience?	kleine-levin syndrome (kls) is a type of recurrent hypersomnia. people with kls experience episodes of excessive sleeping that last between a few days and a few weeks. they may need to sleep as many as 20 hours a day. these episodes of needing extra sleep can occur multiple times in a year but eventually subside over time. the disorder affects one to two people per million and primarily adolescent boys. around 70% of people with kls are male. kls shares characteristics similar to narcolepsy and can cause excessive sleepiness and hunger. the onset of kls is sudden and may at first present like the flu.	as bedtime approaches at the end of the day, our ability to regulate our behavior is already at its lowest. sleep-deprived people may have even less ability to exert self-control and adhere to their intended bedtimes. to add to this, people who procrastinate in other areas of life, such as school or work, are more likely to procrastinate at bedtime. another theory is that bedtime procrastination is related to chronotypes. chronotype is the biological inclination to sleep at a certain time of the day, such as being a night owl versus an early bird. evening types, or night owls, are more likely to procrastinate at bedtime during the week than other chronotypes.
  why are queen size mattresses slightly more expensive than full size models?	a full mattress - also known as a double - measures 54 inches wide by 75 inches long, whereas a queen measures 60 inches wide by 80 inches long. a full size mattress is the same length as a twin mattress, while the length of a queen is comparable to a twin xl or a king mattress. in terms of width, a full is just wide enough to accommodate two sleepers, though most partners tend to prefer a queen. because of their larger size, queen size mattresses are slightly more expensive than full size models. while both queen and full sizes cost more than twin mattresses, they have lower price-points than king sizes.	continuous positive airway pressure (cpap) therapy might be the gold standard for treating obstructive sleep apnea (osa). but a recent study out of la fe university and polytechnic hospital in spain says cpap treatment may not be as effective for patients ages 80 years and older. for this age range, the study looked at how cpap therapy affects sleep apnea symptoms and other factors, including brain function. researchers found that cpap therapy decreased patients' snoring and witnessed apneas, which is when a patient gasps for air and wakes from sleep. they also found that cpap therapy did not improve the patients' other sleep-related symptoms, such as frequent nighttime awakenings to use the bathroom, choking, or nightmares.
  how does the liver convert tryptophan into vitamin b3?	with the help of iron, riboflavin, and vitamin b6, the liver converts tryptophan into vitamin b3, also known as niacin. niacin plays a role in metabolizing energy and creating dna. tryptophan is also needed to produce serotonin, a neurotransmitter involved in sleep, appetite, pain, and mood. serotonin then works with enzymes to create melatonin, a hormone that regulates your sleep-wake cycle by making you feel tired at bedtime. this progression that begins with tryptophan consumption and ends with melatonin production is crucial for healthy sleep. there are two types of tryptophan, l-tryptophan and d-tryptophan. both serve the same general functions but differ slightly at the molecular level.	most knee pillows are smaller than regular pillows and are designed with a thinner center and flared ends. the middle section can be tucked under the knees when back sleeping or between the knees when side sleeping, while the flared ends help the pillow stay in place. because the middle is thinner than most regular pillows, knee pillows provide just the right amount of support for spinal alignment without overextending the knees. several types of pillows feature special designs to support the natural curvature of the neck. cervical pillows are higher under the neck and lower under the head area. u-shaped travel pillows adapt this concept for upright sleeping, with an open space for the head and a wraparound design that helps support the neck. pregnancy pillows come in many styles, but they typically feature some variation of a large u-shape or c-shape. they are designed to wrap around the body and support a baby bump to take pressure off the back and hips when you're lying on your side...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 10
• warmup_ratio: 0.1

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.8.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.4.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}