--- tags: - sentence-transformers - sentence-similarity - feature-extraction - dense - generated_from_trainer - dataset_size:12962 - loss:TripletLoss - dataset_size:21344 base_model: llm-semantic-router/mmbert-embed-32k-2d-matryoshka widget: - source_sentence: What is (are) Leigh's Disease ? sentences: - Leigh's disease is a rare inherited neurometabolic disorder that affects the central nervous system.This progressive disorder begins in infants between the ages of three months and two years.Rarely, it occurs in teenagers and adults.Leigh's disease can be caused by mutations in mitochondrial DNA or by deficiencies of an enzyme called pyruvate dehydrogenase.Symptoms of Leigh's disease usually progress rapidly.The earliest signs may be poor sucking ability,and the loss of head control and motor skills.These symptoms may be accompanied by loss of appetite, vomiting, irritability, continuous crying, and seizures. - Mutations in the ACE, AGT, AGTR1, or REN gene can cause renal tubular dysgenesis.These genes are involved in the renin-angiotensin system, which regulates blood pressure and the balance of fluids and salts in the body and plays a role in kidney development before birth. The renin-angiotensin system consists of several proteins that are involved in a series of steps to produce a protein called angiotensin II.In the first step, the renin protein (produced from the REN gene) converts a protein called angiotensinogen (produced from the AGT gene) to angiotensin I.In the next step, angiotensin-converting enzyme (produced from the ACE gene) converts angiotensin I to angiotensin II.Angiotensin II attaches (binds) to the angiotensin II receptor type 1 (AT1 receptor; produced from the AGTR1 gene), stimulating chemical signaling. By binding to the AT1 receptor, angiotensin II causes blood vessels to narrow (constrict), which results in increased blood pressure. - What causes Hashimoto's encephalitis? The exact cause of Hashimoto's encephalitis (HE) is unknown, but is thought to relate to autoimmune or other autoinflammatory processes. While it is associated with Hashimoto's thyroiditis, the exact nature of the relationship between the two conditions is unclear. It does not appear to be directly related to hypothyroidism or hyperthyroidism. - source_sentence: What causes Heart Palpitations ? sentences: - "These include decongestants (found in cough and cold medicines) and some herbal\ \ and nutritional supplements.\n \nCaffeine, nicotine (found in\ \ tobacco), alcohol, and illegal drugs (such as cocaine and amphetamines) also\ \ can cause palpitations.\n \nCauses Related to Heart Problems\n\ \ \nSome palpitations are symptoms of arrhythmias.Arrhythmias are\ \ problems with the rate or rhythm of the heartbeat.However, less than half of\ \ the people who have palpitations have arrhythmias.\n \nDuring\ \ an arrhythmia, the heart can beat too fast, too slow, or with an irregular rhythm." - "Arrhythmias are problems with the rate or rhythm of the heartbeat.\n \ \ \nSome arrhythmias are signs of heart conditions, such as heart attack,\ \ heart failure, heart valve disease, or heart muscle disease.However, less than\ \ half of the people who have palpitations have arrhythmias.\n \ \ \nYou can take steps to reduce or prevent palpitations.Try to avoid things that\ \ trigger them (such as stress and stimulants) and treat related medical conditions.\n\ \ \nOutlook\n \nPalpitations are very common.They\ \ usually aren't serious or harmful, but they can be bothersome.If you have them,\ \ your doctor can decide whether you need treatment or ongoing care." - Rheumatic fever is an inflammatory condition that may develop after infection with group A Streptococcus bacteria, such as strep throat or scarlet fever. It is primarily diagnosed in children between the ages of 6 and 16 and can affect the heart, joints, nervous system and/or skin. Early signs and symptoms include sore throat; swollen red tonsils; fever; headache; and/or muscle and joint aches. Some affected people develop rheumatic heart disease, which can lead to serious inflammation and scarring of the heart valves. It is not clear why some people who are infected with group A Streptococcus bacteria go on to develop rheumatic fever, while others do not; however, it appears that some families may have a genetic susceptibility to develop the condition. Treatment usually includes antibiotics and/or anti-inflammatory medications. - source_sentence: What are the symptoms of Colpocephaly ? sentences: - What are the symptoms of colpocephaly? Colpocephaly is characterized by a small head circumference and in many cases, intellectual disability. Other signs and symptoms may include movement abnormalities, muscle spasms, and seizures. Poor vision, speech and language difficulties, deafness, and chorioretinitis have been described in individual cases. Cases of people with colpocephaly and normal neurological and motor development have also been described. - The NINDS supports a broad program of research on disorders of the nervous system, including CIDP. Much of this research is aimed at increasing the understanding of these disorders and finding ways to prevent, treat, and cure them. - What are the signs and symptoms of pachygyria? Signs and symptoms of pachygyria vary among affected people and can depend on the extent of the abnormality. They often include poor muscle tone and motor function; seizures; developmental delays; intellectual disability; failure to grow and thrive; difficulties with feeding or swallowing; swelling in the extremities; and small head size (microcephaly). Most infants appear physically normal, but some conditions associated with pachygyria cause distinctive facial or skull characteristics. - source_sentence: How many people are affected by sialuria ? sentences: - PD is both chronic, meaning it persists over a long period of time, and progressive, meaning its symptoms grow worse over time. Although some people become severely disabled, others experience only minor motor disruptions. Tremor is the major symptom for some individuals, while for others tremor is only a minor complaint and other symptoms are more troublesome. It is currently not possible to predict which symptoms will affect an individual, and the intensity of the symptoms also varies from person to person. - Fewer than 10 people worldwide have been diagnosed with sialuria. There are probably more people with the disorder who have not been diagnosed, as sialuria can be difficult to detect because of its variable features. - Sialic acid storage disease is a very rare disorder. ISSD has been identified in only a few dozen infants worldwide. Salla disease occurs mainly in Finland and Sweden and has been reported in approximately 150 people. A few individuals have been identified as having intermediate severe Salla disease. - source_sentence: What is (are) Turner Syndrome ? sentences: - What are the signs and symptoms of Turner syndrome?There are various signs and symptoms of Turner syndrome, which can range from very mild to more severe.Short stature is the most common feature and usually becomes apparent by age 5.In early childhood, frequent middle ear infections are common and can lead to hearing loss in some cases.Most affected girls do not produce the necessary sex hormones for puberty, so they don't have a pubertal growth spurt, start their periods or develop breasts without hormone treatment.While most affected women are infertile, pregnancy is possible with egg donation and assisted reproductive technology.Intelligence is usually normal, but developmental delay, learning disabilities, and/or behavioral problems are sometimes present. - What are the signs and symptoms of Myotonic dystrophy?Signs and symptoms of myotonic dystrophy often begin in a person's 20s or 30s, but they can begin at any age.Symptoms often include progressive muscle weakness and wasting (particularly in the legs, hands, neck and face); stiffness and tightness of the muscles; cataracts; and cardiac conduction defects (irregular electrical control of the heartbeat).Some affected men also have hormonal changes that may cause balding or infertility.The severity of symptoms can vary widely among affected people.The signs and symptoms of type 1 and type 2 overlap, but type 2 is generally more mild than type 1.People who are born with the condition have congenital myotonic dystrophy, which is a variation of type 1.Congenital myotonic dystophy causes weakness of all muscles, in addition to breathing problems, developmental delays and intellectual disabilities. - 'Assisted reproduction techniques can help some women with Turner syndrome get pregnant. NIH: National Institute of Child Health and Human Development' pipeline_tag: sentence-similarity library_name: sentence-transformers --- # SentenceTransformer based on llm-semantic-router/mmbert-embed-32k-2d-matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [llm-semantic-router/mmbert-embed-32k-2d-matryoshka](https://huggingface.co/llm-semantic-router/mmbert-embed-32k-2d-matryoshka). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [llm-semantic-router/mmbert-embed-32k-2d-matryoshka](https://huggingface.co/llm-semantic-router/mmbert-embed-32k-2d-matryoshka) - **Maximum Sequence Length:** 32768 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'ModernBertModel'}) (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("sentence_transformers_model_id") # Run inference sentences = [ 'What is (are) Turner Syndrome ?', 'Assisted reproduction techniques can help some women with Turner syndrome get pregnant. NIH: National Institute of Child Health and Human Development', "What are the signs and symptoms of Turner syndrome?There are various signs and symptoms of Turner syndrome, which can range from very mild to more severe.Short stature is the most common feature and usually becomes apparent by age 5.In early childhood, frequent middle ear infections are common and can lead to hearing loss in some cases.Most affected girls do not produce the necessary sex hormones for puberty, so they don't have a pubertal growth spurt, start their periods or develop breasts without hormone treatment.While most affected women are infertile, pregnancy is possible with egg donation and assisted reproductive technology.Intelligence is usually normal, but developmental delay, learning disabilities, and/or behavioral problems are sometimes present.", ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities) # tensor([[1.0000, 0.7383, 0.4883], # [0.7383, 1.0000, 0.5508], # [0.4883, 0.5508, 1.0000]], dtype=torch.bfloat16) ``` ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 21,344 training samples * Columns: sentence_0, sentence_1, and sentence_2 * Approximate statistics based on the first 1000 samples: | | sentence_0 | sentence_1 | sentence_2 | |:--------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | sentence_0 | sentence_1 | sentence_2 | |:-------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | What is the outlook for Coffin Lowry Syndrome ? | The prognosis for individuals with Coffin-Lowry syndrome varies depending on the severity of symptoms. Early intervention may improve the outlook for patients. Life span is reduced in some individuals with Coffin-Lowry syndrome. | How might Coffin-Siris syndrome be treated? People with Coffin-Siris syndrome may benefit from occupational, physical, and speech therapy. Developmental pediatricians may be helpful in recommending and coordinating therapeutic and educational interventions. Additional specialty care may be needed depending on the symptoms in the individual, such as by gastrointestinal, eye, kidney, heart, and hearing specialists. | | What is (are) Sarcoidosis ? | Sarcoidosis is an inflammatory disease characterized by the development and growth of tiny lumps of cells called granulomas. If these tiny granulomas grow and clump together in an organ, they can affect how the organ works, leading to the symptoms of sarcoidosis. The granulomas can be found in almost any part of the body, but occur more commonly in the lungs, lymph nodes, eyes, skin, and liver. Although no one is sure what causes sarcoidosis, it is thought by most scientists to be a disorder of the immune system. The course of the disease varies from person to person. It often goes away on its own, but in some people symptoms of sarcoidosis may last a lifetime. For those who need treatment, anti-inflammatory medications and immunosuppressants can help. | Sarcoidosis is a disease that leads to inflammation, usually in your lungs, skin, or lymph nodes. It starts as tiny, grain-like lumps, called granulomas. Sarcoidosis can affect any organ in your body. No one is sure what causes sarcoidosis. It affects men and women of all ages and races. It occurs mostly in people ages 20 to 50, African Americans, especially women, and people of Northern European origin. Many people have no symptoms. If you have symptoms, they may include - Cough - Shortness of breath - Weight loss - Night sweats - Fatigue Tests to diagnose sarcoidosis include chest x-rays, lung function tests, and a biopsy. Not everyone who has the disease needs treatment. If you do, prednisone, a type of steroid, is the main treatment. NIH: National Heart, Lung, and Blood Institute | | What are the symptoms of High Blood Pressure ? | Because diagnosis is based on blood pressure readings, this condition can go undetected for years, as symptoms do not usually appear until the body is damaged from chronic high blood pressure.



Complications of High Blood Pressure

When blood pressure stays high over time, it can damage the body and cause complications.Some common complications and their signs and symptoms include:

Aneurysms:When an abnormal bulge forms in the wall of an artery.Aneurysms develop and grow for years without causing signs or symptoms until they rupture, grow large enough to press on nearby body parts, or block blood flow.The signs and symptoms that develop depend on the location of the aneurysm.

Chronic Kidney Disease: When blood vessels narrow in the kidneys, possibly causing kidney failure.

Cognitive Changes: Research shows that over time, higher blood pressure numbers can lead to cognitive changes. | High blood pressure is a common disease in which blood flows through blood vessels (arteries) at higher than normal pressures. There are two main types of high blood pressure: primary and secondary high blood pressure. Primary, or essential, high blood pressure is the most common type of high blood pressure. This type of high blood pressure tends to develop over years as a person ages. Secondary high blood pressure is caused by another medical condition or use of certain medicines. This type usually resolves after the cause is treated or removed. | * Loss: [TripletLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#tripletloss) with these parameters: ```json { "distance_metric": "TripletDistanceMetric.COSINE", "triplet_margin": 0.1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `num_train_epochs`: 2 - `multi_dataset_batch_sampler`: round_robin #### All Hyperparameters
Click to expand - `do_predict`: False - `eval_strategy`: no - `prediction_loss_only`: True - `per_device_train_batch_size`: 8 - `per_device_eval_batch_size`: 8 - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 5e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1 - `num_train_epochs`: 2 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: None - `warmup_ratio`: None - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `enable_jit_checkpoint`: False - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `use_cpu`: False - `seed`: 42 - `data_seed`: None - `bf16`: False - `fp16`: False - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: -1 - `ddp_backend`: None - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `group_by_length`: False - `length_column_name`: length - `project`: huggingface - `trackio_space_id`: trackio - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `hub_revision`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `auto_find_batch_size`: False - `full_determinism`: False - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_num_input_tokens_seen`: no - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `liger_kernel_config`: None - `eval_use_gather_object`: False - `average_tokens_across_devices`: True - `use_cache`: False - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: round_robin - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | |:------:|:----:|:-------------:| | 0.3085 | 500 | 0.0271 | | 0.6169 | 1000 | 0.0134 | | 0.9254 | 1500 | 0.0111 | | 1.2338 | 2000 | 0.0059 | | 1.5423 | 2500 | 0.0056 | | 1.8507 | 3000 | 0.0046 | | 0.1874 | 500 | 0.0184 | | 0.3748 | 1000 | 0.0161 | | 0.5622 | 1500 | 0.0143 | | 0.7496 | 2000 | 0.0127 | | 0.9370 | 2500 | 0.0125 | | 1.1244 | 3000 | 0.0090 | | 1.3118 | 3500 | 0.0064 | | 1.4993 | 4000 | 0.0059 | | 1.6867 | 4500 | 0.0062 | | 1.8741 | 5000 | 0.0054 | ### Framework Versions - Python: 3.12.3 - Sentence Transformers: 5.2.2 - Transformers: 5.0.0 - PyTorch: 2.10.0+cu128 - Accelerate: 1.12.0 - Datasets: 4.5.0 - Tokenizers: 0.22.2 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### TripletLoss ```bibtex @misc{hermans2017defense, title={In Defense of the Triplet Loss for Person Re-Identification}, author={Alexander Hermans and Lucas Beyer and Bastian Leibe}, year={2017}, eprint={1703.07737}, archivePrefix={arXiv}, primaryClass={cs.CV} } ```