Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Paper • 1908.10084 • Published • 15
How to use ahmedHamdi/IR-fr-en-gemma_masked with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("ahmedHamdi/IR-fr-en-gemma_masked")
sentences = [
"Ever since a pet shop groomer took in a Persian cat belonging to a deceased client, she has been haunted by a little girl with feline eyes. The cat emits plaintive meows, and this apparition is linked to the deaths of people close to the groomer, who are being killed one by one. She sees herself as the next victim and, with the help of a police officer, decides to discover the identity of the girl with the feline eyes…",
" The Italian veline phenomenon is explained. We meet young mechanic PERSON who tries to become a TV star, but complains that it is more difficult for a man. TV agent PERSON admiringly says that PERSON resembles PERSON. PERSON takes embarrassing photographs of celebrities, and asks them for money to not publicize them. He explains that he is a new version of PERSON: he steals from the rich, but keeps the money to himself. When he was convicted for extortion, it made him a greater celebrity, and he is now cashing in on this. He is shown full-frontally naked taking a shower.",
"So-yeon is a kind woman who works as a groomer in a pet shop called ORG, but has claustrophobia due to childhood trauma. A woman comes to the pet shop to collect her Persian cat, GPE, one day, after one of GPE's lengthy checkups at the pet shop. The next day, the woman is found dead in an elevator, but GPE is unharmed. The police are unable to determine the cause of her death. So-yeon's friend PERSON, one of the police officers investigating the murder, gives her GPE to look after, since she has already had experience with animals. So-yeon starts to have nightmares of a young girl with cat-like eyes and is haunted by hallucinations. Jun-seok and his fellow officers watch ORG footage of the woman who died and it's concluded that she died of a panic attack. So-yeon's friend Bo-hee, who recently adopted a cat, dies of a similar cause. That night, So-yeon cuts her finger while preparing food for GPE. GPE licks the blood and becomes aggressive. So-yeon, terrified, takes the cat to the dead woman's husband, but he does not want him. He explains that his wife claimed to be haunted by a strange little girl after buying GPE. Disturbed, So-yeon leaves GPE in a park. At the animal shelter, a staff member cremates a dead cat, but is pulled inside the furnace by an unseen force and burns to death. Jun-seok and So-yeon go to the animal shelter, where they find dead cats and the charred remains of the staff member. They learn that some time ago, there was an infestation of stray cats in the boiler room of an apartment complex. The doors and windows were cemented shut and the cats were left to suffocate. Two weeks later, workers removed the dead cats. So-yeon remarks the similarity of the murder victims all being found dead in a small space. She is again approached by a confused old woman she had encountered before, who is looking for her granddaughter. Jun-seok discovers that the old woman reported her missing granddaughter nine months ago, but her son closed the case. So-yeon escorts the woman back to her apartment - in the same complex where the stray cats lived in the boiler room - and Jun-seok gives her a photo of the granddaughter, who looks exactly like the cat-eyed girl So-yeon had seen in her hallucinations, the same one the dead woman had complained of seeing. So-yeon sees the old woman's son beating his mother; he is then killed by a horde of cats. So-yeon goes to the complex's boiler room and is confronted by cats. While trying to fight off the hoarde, including GPE, So-yeon falls into a large canister. The cat-eyed girl appears and shows her how she died; she had played with the cats in the boiler room, and upon hearing of the plans to kill them, she attempted to hide them in the canister. While climbing out, she fell and was paralyzed, dying with the cats after the door and windows were cemented. The girl also has a special relationship with those who have cats, and it is actually the restless spirits of the cats who have murdered the others. The girl convinces the cats to stop and they let So-yeon go free. Having conquered her claustrophobia, So-yeon visits her father in a mental hospital, riding in an elevator for the first time without panicking. As she leaves, she and Ju-seok find a kitten underneath their car, and she kindly beckons toward it.",
" Teenage pranksters PERSON, PERSON, PERSON, and GPE, a.k.a. “PrankMonkey69”, successfully convince Mrs. PERSON that an intruder is in her house and her daughter is in danger. The video of the prank phone call goes viral and the boys become well known on YouTube. With PERSON alone at home for the weekend, PERSON comes over and the two of them make more prank phone calls. One of their pranks involves ordering a pizza for PERSON's neighbor PERSON. PERSON, who delivers the pizza from the shop where he works with PERSON's girlfriend ORG, becomes upset with his friends for involving him in the prank. A mysterious man who identifies himself as Mr. PERSON begins repeatedly calling the boys, insisting that they do not hang up on him. PERSON and PERSON are initially indifferent until Mr. PERSON reveals that he knows their names, addresses, and intimate details of their lives. Peyton delivers a pizza that PERSON and PERSON didn't order. Peyton reveals that PERSON disappeared after his earlier delivery. PERSON and PERSON have an uncomfortable conversation about their relationship before Peyton leaves. Mr. PERSON begins texting PERSON and PERSON in addition to calling. Video footage he sends them reveals that Mr. PERSON is holding PERSON's parents hostage at the Mannion residence. Mr. PERSON prevents the boys from calling 911 by hijacking their cell phone lines. He also takes control of the house's electrical power. A video feed shows the masked Mr. PERSON suffocating PERSON to death with a plastic bag tied around his head. PERSON finds PERSON similarly bound with a plastic bag and tied to the back door. PERSON dies soon after PERSON cuts him loose. While PERSON is away, Mr. PERSON makes PERSON an offer to kill PERSON in exchange for his parents' freedom. Mr. PERSON sends footage showing that he captured LOC. PERSON deduces that PERSON had a private conversation with Mr. PERSON, leading to a fight between the two. PERSON is knocked unconscious. Mr. PERSON shows PERSON live footage of LOC held prisoner. He promises to let LOC go if PERSON. PERSON restrains PERSON by securing his wrist to the banister with a zip tie to prevent him from leaving. A clue of a child's image and the name GPE leads PERSON to check a previously ignored social media friend request. The GPE account reveals that Mr. PERSON had been stalking the pranksters for over a year. PERSON also finds a video revealing that PERSON had been sleeping with LOC. PERSON beats PERSON again. PERSON notices from a clock in the background that the video of PERSON's parents is two hours behind the current time. PERSON frees PERSON. New video shows that Mr. PERSON already killed PERSON's parents. Believing that ORG is probably dead too, the boys plan to escape and contact authorities. However, once PERSON is outside, PERSON locks the front door and remains behind. PERSON finds a surveillance van outside in the rain. Inside the van, PERSON finds PERSON and rescues her. Back in the house, PERSON steps over PERSON's covered body to go out the back door. In the backyard, PERSON finds PERSON's real body. PERSON realizes that Mr. PERSON switched places with PERSON and hid in the house. Mr. PERSON catches PERSON off guard and attacks him. PERSON has PERSON go for help while he looks for PERSON. PERSON follows ORG's screams to the backyard. A masked man suddenly lunges at PERSON and PERSON stabs him. The man turns out to be PERSON, tied up and gagged to look like Mr. PERSON in a mask so that PERSON would kill him by mistake. With PERSON and PERSON defeated, Mr. PERSON shows up and explains his actions. When the four boys previously called Mrs. PERSON, she dropped her phone and didn't hear the revelation that she had been pranked. Still convinced that there was an intruder in the house, Mrs. PERSON grabbed a gun and accidentally shot her daughter GPE. Upon realizing what she had done, Mrs. PERSON killed herself. It's implied that Mrs. PERSON and the young girl were Mr. PERSON's wife and daughter, and he wanted to avenge their deaths. Mr. PERSON knocks PERSON unconscious. When PERSON wakes up, he finds Peyton shot in the head. Holding a knife in one hand and a gun in the other, PERSON is arrested by the police arriving at the scene; Mr. PERSON successfully having framed PERSON for killing PERSON and PERSON out of jealousy, as well as the other murders. In the epilogue, Mr. PERSON begins targeting a new teen prankster."
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]This is a sentence-transformers model finetuned from google/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("ahmedHamdi/IR-fr-en-gemma_masked")
# Run inference
sentences = [
"Tuscany, 1984. Mario and GPE find themselves stuck at a level crossing, and the two friends confide in each other about their lives. GPE, among other things, is worried about his sister PERSON, who has fallen into depression because of her relationship with an American. As the wait drags on, the two men decide to take a detour and end up stranded in the middle of the countryside. They take refuge in an inn for the night and wake up the next morning in a Tuscan village called PERSON, sometime between 1400 and 1500. Initially believing it to be a big joke, the two men find themselves having to confront the reality of the era. They find shelter with PERSON, the brother of the PERSON's guest who died the day before. They begin working at the ORG owned by PERSON's mother, PERSON, and while GPE adapts immediately to his new lifestyle, PERSON refuses to adjust. Soon, PERSON, the daughter of a wealthy family, with whom he begins a relationship. Meanwhile, PERSON is arrested, and GPE tries to write a letter to ORG to secure his release. GPE makes no secret of his jealousy of ORG relationship and convinces his friend to set off for GPE to join PERSON.\nDuring their journey to GPE, they encounter a beautiful ORG, PERSON, who frightens them by shooting an arrow at their chariot. The story then divides into two parts, depending on whether it is presented in the television or film version.",
'Tuscan countryside. The janitor PERSON and the teacher GPE are stopped at a level crossing, waiting for the train to pass. The two are friends and trust each other. Among other things, GPE is worried about his sister PERSON, who fell into depression for the failure of her relationship with an American boy. The wait continues and decide to travel a road in the fields. After a while they remain in cream with the car in the middle of the countryside. You do evening, it rains. The two find accommodation in an inn for the night, in a room that already hosts another person. The next morning as soon as they wake up they see, having fun, the guest urinating from the window, but their laughter are immediately truncated by the hiss of a spear that kills him. Mario and GPE see fonding people in the black cloak on horseback, rush to the ground floor and find other people, dressed in a very strange way. Increducts make themselves be said by a man where they find themselves and discover that they are fried, an imaginary Tuscan village, in 1492. considering it first a terrible joke, they must resign themselves to the harsh reality making themselves hosting from PERSON, the brother of the killed man, ORG, who tells them of a terrible feud with such a ORG, who is exterminating his family. Once in the village they know PERSON, a mother of PERSON and the late ORG and begin to work in their slaughter shop. In the ORG context of the village, the most disparate episodes happen. GPE immediately seems comfortable, while PERSON does not want to settle down; Soon, however, during a religious function, PERSON makes the acquaintance of Pia, a girl of a rich family, with whom he begins to see himself looking out from the surrounding wall of her house. In the meantime, PERSON is arrested and GPE and PERSON write a letter to ORG in vain to obtain his liberation. GPE (who continues to tell PERSON to ask PERSON if he has an friend to introduce him) does not hide a certain annoyance for the meetings between PERSON and the young PERSON and complains that he always remains and only him to work in the butcher shop. Driven by his political-intellectual ardor, he convinces his friend to travel to GPE, in order to reach PERSON and dissuade him from leaving for the ORG and discovering GPE, so that, in the future, his sister cannot meet the American boy who left her. In an unspecified place the two come across a beautiful ORG, PERSON, which intimidates them by throwing an arrow against their wagon. At this point the story differs depending on the version, standard or extended.',
"World-famous symphony conductor Sir PERSON returns from a visit to his native GPE and discovers that his rich brother-in-law, August Henshler, having misunderstood PERSON's casual instruction to watch over his much younger wife PERSON while he was away, hired a private detective named ORG to follow her. PERSON is livid, and when August hands him the detective's report he tears it to pieces. Despite himself, the idea of infidelity on ORG's part hounds PERSON's mind, and he acquires ORG's address from August, ostensibly so he can destroy any copies of the report which ORG may have kept on file. ORG summarizes the report for PERSON: his wife was spied late at night wearing only a nightgown as she went to the hotel room of PERSON's secretary, PERSON, a man closer in age to her own, where she stayed for thirty-eight minutes. Distressed by the news, PERSON quarrels with PERSON before proceeding to his concert, where he conducts three pieces of classical music, envisioning scenarios appropriate to each one: a perfect crime scenario in which he murders his wife with a straight razor and frames PERSON (to the ORG to PERSON's Semiramide), accepting the situation and giving PERSON a generous check and his blessing (to the Prelude to ORG's Tannhäuser), and a game of Russian roulette with a blubbering PERSON and PERSON, that ends in PERSON's death with the first trigger pull (to GPE's Francesca da Rimini). After the concert, PERSON tries to stage his murder fantasy, but makes a mess of their apartment while looking for the recording device which he fantasized using to lure PERSON to the crime scene, then accidentally records at the wrong speed. Daphne returns home and questions her husband about his ill temperament, but devotedly bandages his thumb when he cuts it testing the sharpness of his razor. He next tries the forgiveness fantasy, but spills ink all over his checkbook. Russian roulette also fizzles out when he cannot remember where he left the bullets. Without revealing his suspicions, he asks if she has ever been to PERSON's room. She admits it, saying she was searching for her sister, PERSON, August's wife, suspecting she was having an affair with PERSON; she became trapped there when she saw ORG spying on the room, fearful that if he saw her walk out she would be suspected of having an affair. Alfred begs PERSON's forgiveness for his irrational behavior, which she gladly gives, ascribing it to the creative temperament of a great artist.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000, 0.5453, -0.0518],
# [ 0.5453, 1.0000, 0.0562],
# [-0.0518, 0.0562, 1.0000]])
sentence_0 and sentence_1| sentence_0 | sentence_1 | |
|---|---|---|
| type | string | string |
| details |
|
|
| sentence_0 | sentence_1 |
|---|---|
General Overview |
As a war galley sails through a storm, two dragons fight above the clouds, ending in one's death. In the kingdom of GPE, the royal wizard ORG proclaims the dragons to be a sign of a loss of balance in the world. The king is dealing with disease sweeping across his kingdom, and the disappearance of his son, PERSON. In the middle of the night, ORG kills his father, steals his father's sword, and flees the castle. Arren travels through the desert and is rescued from wolves by the archmage Sparrowhawk. Together they travel to the city of LOC. When ORG explores the town alone, he rescues a young girl named ORG from slavers, but is later captured by the same slave master, GPE. His sword is dumped in the sea. Sparrowhawk rescues Arren from the slave caravan and takes him to a farm run by ORG's oldest and greatest friend PERSON, who lives with ORG. ORG's intervention against ORG's slave caravan angers Lord Cob, a powerful warlock and the ruler of Hort Town, who wants the archmage brought to th... |
Inspired by the true story of ORG (PERSON), Fighter in the PERSON tells the story of how a Korean immigrant became a legend in GPE by creating a new martial art: Taekkyon. |
At the tail-end of World War II, PERSON is a young Korean man who longs to be able to fly fighter planes. Stowing away to GPE in order to join their air force, Bae-dal's first experience of the country is when a con-man tries to steal his money. Bae-dal discovers that the man is a fellow Korean called ORG (PERSON), who has survived the harsh treatment of Koreans in GPE by turning to petty crime. With their different motives: Bae-dal driven by desire for action and Chun-bae needing to escape from some gangsters, the two Koreans stow away in a truck to the air force training camp. The commander in charge of the camp is a pompous imperialist called PERSON (PERSON). Having mistreated the two Koreans, he is amused by Bae-dal's fighting spirit and says that if Bae-dal can beat him with his inferior foreign fighting style (Taekkyon), he will release them. The two men fight with PERSON easily defeating Bae-dal, but an American attack on the airforce base allows Bae-dal and PERSON to escape. La... |
Ten-year-old PERSON is thrilled when his grandfather moves into his house. He's less than thrilled when he has to give up his bedroom and move into the attic. With the help of his friends, PERSON decides to declare war on his grandfather, who is much smarter than he looks... |
After accidentally stealing from a grocery store due to having trouble with the self-checkouts and causing a scene with the store manager, recently widowed PERSON (PERSON) is visited by his daughter PERSON (PERSON) who wants him to move in with her family. PERSON does not want to leave his house because he built it himself. PERSON convinces PERSON to move in with her and gives him her son PERSON's (PERSON) bedroom. PERSON is not happy about giving his room to his grandfather and being moved to the attic. PERSON is welcomed by PERSON's husband PERSON (PERSON) and two daughters, PERSON (PERSON) and PERSON (Poppy Gagnon). During his first day, PERSON spends most of his time in his new room, sitting in his chair and looking at the sky while still thinking about his late wife. PERSON then tells his friends PERSON (PERSON), PERSON (PERSON) and PERSON (PERSON) about his grandfather moving in with his family and living in his room. After a miserable first night in his new room, PERSON decides ... |
MultipleNegativesRankingLoss with these parameters:{
"scale": 20.0,
"similarity_fct": "cos_sim",
"gather_across_devices": false
}
per_device_train_batch_size: 4per_device_eval_batch_size: 4multi_dataset_batch_sampler: round_robinoverwrite_output_dir: Falsedo_predict: Falseeval_strategy: noprediction_loss_only: Trueper_device_train_batch_size: 4per_device_eval_batch_size: 4per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 1eval_accumulation_steps: Nonetorch_empty_cache_steps: Nonelearning_rate: 5e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1num_train_epochs: 3max_steps: -1lr_scheduler_type: linearlr_scheduler_kwargs: Nonewarmup_ratio: 0.0warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falsebf16: Falsefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Falseignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedeepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torch_fusedoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthproject: huggingfacetrackio_space_id: trackioddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Nonehub_always_push: Falsehub_revision: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseinclude_for_metrics: []eval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters: auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: noneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falseeval_on_start: Falseuse_liger_kernel: Falseliger_kernel_config: Noneeval_use_gather_object: Falseaverage_tokens_across_devices: Trueprompts: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: round_robinrouter_mapping: {}learning_rate_mapping: {}| Epoch | Step | Training Loss |
|---|---|---|
| 0.1229 | 500 | 0.0396 |
| 0.2458 | 1000 | 0.0699 |
| 0.3686 | 1500 | 0.1463 |
| 0.4915 | 2000 | 0.1551 |
| 0.6144 | 2500 | 0.1705 |
| 0.7373 | 3000 | 0.1281 |
| 0.8602 | 3500 | 0.1261 |
| 0.9830 | 4000 | 0.1362 |
| 1.1059 | 4500 | 0.09 |
| 1.2288 | 5000 | 0.0709 |
| 1.3517 | 5500 | 0.1023 |
| 1.4746 | 6000 | 0.0672 |
| 1.5974 | 6500 | 0.0847 |
| 1.7203 | 7000 | 0.0734 |
| 1.8432 | 7500 | 0.0687 |
| 1.9661 | 8000 | 0.0602 |
| 2.0890 | 8500 | 0.0431 |
| 2.2118 | 9000 | 0.0298 |
| 2.3347 | 9500 | 0.0317 |
| 2.4576 | 10000 | 0.0218 |
| 2.5805 | 10500 | 0.0178 |
| 2.7034 | 11000 | 0.0211 |
| 2.8262 | 11500 | 0.0171 |
| 2.9491 | 12000 | 0.0227 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Base model
google/embeddinggemma-300m