swardiantara/bert-tiny-yelp-k1-fixed-cosine

SentenceTransformer based on google/bert_uncased_L-2_H-128_A-2

This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2. It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-2_H-128_A-2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-yelp-k1-fixed-cosine")
# Run inference
sentences = [
    "I went to a yelp event here, where we were able to make a fused glass piece and also watch some demonstrations and go to the gallery. Everything in the gallery was amazing. The demonstrations were neat, and the fused glass was something I would go back to do again.\\n\\nI really didn't know that the glass center was open most of the time, and if you go there on any random day, you can watch whatever is going on. I think that is pretty cool! They also have classes where you can blow glass or make beads or whatever you want... I can't wait to pick up my piece and come back for another class!",
    "First Watch on Black Canyon Hwy in Phoenix has amazing service and very good food! They're worth the drive. You won't be disappointed. Coincidentally, I went there b/c it was right next door to the Courtyard Marriott. I have been back several times and continue to receive excellent customer service and delicious food. \\n\\nMy favorite breakfast is their Tri-Athlete omelette; which I hesitatingly tried b/c it was a healthy choice but fell in love with it! Wonderfully roasted vegetables in an egg-white only omelette topped with salsa...muahhh, magnifico!!\\n\\nFor lunch I had their Pecan Dijon chicken salad...wow, another amazing dish full of flavors...good to the last drop!\\n\\nAnd lastly, their customer service...always SPOT ON! Once we had only 15 minutes. Informed the always-welcoming staff and VOILA, an excellent breakfast in exactly 15 minutes. I'll be back soon! \\n\\nThank you First Watch on Black Canyon Hwy!! You're awesome :)",
    "Mmmm....decaf latte. That's honestly all I ever had at this cafe, but man it's good. It comes with a pretty leaf design too! They offer Felix and Norton tiny cookies and also pastries. My friend had a muffin and a cappucino, both which were good according to her. It was actually very quiet when I went which is a plus. Nothing I hate more than overcrowded coffee shops. \\n\\nService is friendly. Oh! And they offer some salads and sandwiches/wraps as well. Located near UQAM, perfect for university students. I really recommend this place, it's a cute and quaint place.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9973, 0.9974],
#         [0.9973, 1.0000, 0.9991],
#         [0.9974, 0.9991, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 585,010 training samples
• Columns: text_a, text_b, and label
• Approximate statistics based on the first 100 samples:

  	text_a	text_b	label
  type	string	string	list
  modality	text	text
  details	min: 6 tokens mean: 93.98 tokens max: 128 tokens	min: 128 tokens mean: 128.0 tokens max: 128 tokens	size: 2 elements

• Samples:

  text_a	text_b	label
  dr. goldberg offers everything i look for in a general practitioner. he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first. really, what more do you need? i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.	First Watch on Black Canyon Hwy in Phoenix has amazing service and very good food! They're worth the drive. You won't be disappointed. Coincidentally, I went there b/c it was right next door to the Courtyard Marriott. I have been back several times and continue to receive excellent customer service and delicious food. \n\nMy favorite breakfast is their Tri-Athlete omelette; which I hesitatingly tried b/c it was a healthy choice but fell in love with it! Wonderfully roasted vegetables in an egg-white only omelette topped with salsa...muahhh, magnifico!!\n\nFor lunch I had their Pecan Dijon chicken salad...wow, another amazing dish full of flavors...good to the last drop!\n\nAnd lastly, their customer service...always SPOT ON! Once we had only 15 minutes. Informed the always-welcoming staff and VOILA, an excellent breakfast in exactly 15 minutes. I'll be back soon! \n\nThank you First Watch on Black Canyon Hwy!! You're awesome :)	[1.0, 0.0]
  Unfortunately, the frustration of being Dr. Goldberg's patient is a repeat of the experience I've had with so many other doctors in NYC -- good doctor, terrible staff. It seems that his staff simply never answers the phone. It usually takes 2 hours of repeated calling to get an answer. Who has time for that or wants to deal with it? I have run into this problem with many other doctors and I just don't get it. You have office workers, you have patients with medical needs, why isn't anyone answering the phone? It's incomprehensible and not work the aggravation. It's with regret that I feel that I have to give Dr. Goldberg 2 stars.	I had been scouting out the yelp reviews of this buffet for a couple of months, waiting for the next time my friend came into town so we could try it. We went on the weekend for dinner and payed $40each.\n\nI must say that it was a huge disappointment. I used to be a fan of the Rio seafood buffet, but it has gone way down in quality over the years so I was really looking forward to getting some King Crab legs at the Bellagio, even though it is chilled and not steamed. \n\nTo my dismay, the crab was not good. Both the King & Snow crab had no flavor and some of it actually smelled like it was bad. But I am not basing my review on the dismal crab.\n\nI had read many posts about how good the Kobe beef, and the Beef Wellington were, and numerous other items that were great, and they just weren't all that good. \n\nThe "Kobe" beef tasted like dry roast beef. The Beef Wellington was ok(never had it before), but it didn't knock my socks off. The Lamb chops were a bit gamey, and I've ...	[1.0, 0.0]
  Been going to Dr. Goldberg for over 10 years. I think I was one of his 1st patients when he started at MHMG. He's been great over the years and is really all about the big picture. It is because of him, not my now former gyn Dr. Markoff, that I found out I have fibroids. He explores all options with you and is very patient and understanding. He doesn't judge and asks all the right questions. Very thorough and wants to be kept in the loop on every aspect of your medical health and your life.	Mmmm....decaf latte. That's honestly all I ever had at this cafe, but man it's good. It comes with a pretty leaf design too! They offer Felix and Norton tiny cookies and also pastries. My friend had a muffin and a cappucino, both which were good according to her. It was actually very quiet when I went which is a plus. Nothing I hate more than overcrowded coffee shops. \n\nService is friendly. Oh! And they offer some salads and sandwiches/wraps as well. Located near UQAM, perfect for university students. I really recommend this place, it's a cute and quaint place.	[1.0, 0.0]

• Loss: main.OrdinalProxyContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1024
• num_train_epochs: 10
• learning_rate: 2e-05
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 1024
• num_train_epochs: 10
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.8741	500	0.0006
1.0	572	-
1.7483	1000	0.0000
2.0	1144	-
2.6224	1500	0.0000
3.0	1716	-

• The bold row denotes the saved checkpoint.

Training Time

• Training: 12.7 minutes
• Evaluation: 4.2 minutes
• Total: 16.8 minutes

Framework Versions

• Python: 3.12.4
• Sentence Transformers: 5.5.1
• Transformers: 5.11.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 2.21.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}