Integrate Sentence Transformers, prevent manual tokenizer EOS ac202b7
Tom Aarsen commited on
How to use facebook/drama-1b with Transformers:
# Load model directly
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("facebook/drama-1b", trust_remote_code=True)
model = AutoModel.from_pretrained("facebook/drama-1b", trust_remote_code=True)How to use facebook/drama-1b with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("facebook/drama-1b", trust_remote_code=True)
sentences = [
"هذا شخص سعيد",
"هذا كلب سعيد",
"هذا شخص سعيد جدا",
"اليوم هو يوم مشمس"
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]