Instructions to use Sleem247/chatbot_sentence-transformer-Q8_0-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use Sleem247/chatbot_sentence-transformer-Q8_0-GGUF with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("Sleem247/chatbot_sentence-transformer-Q8_0-GGUF") sentences = [ "Its personal! [SEP] I put you on uppers and you still yawned. Means its a symptom, of being a big fat liar. Yawning is a side effect of some antidepressants, apparently the ones youre on. Im not on antidepressants Im on SPEEEEEED! Well that means its a symptom of a cerebral tumour. You got six weeks to live. Mr. Welladjusted is as messed up as the rest of us. Whwhy would you keep that a secret? Are you ashamed of recognising how pathetic your life is? Its not a secret. House itsits... its personal! How long has it been personal? Yawnings recent so! either you just started or you changed prescription.", "Yawnings recent so! either you just started or you changed prescription.", "The High Sparrow has hundreds of Faith Militant surrounding him. Ser Gregor will can’t face them all. And he won’t have to. He’ll only have to face one.", "Whoa, whoa, whoa! Hey! Whoa!" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Sleem247/chatbot_sentence-transformer-Q8_0-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: ./llama-cli -hf Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: ./build/bin/llama-cli -hf Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0
Use Docker
docker model run hf.co/Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0
- LM Studio
- Jan
- Ollama
How to use Sleem247/chatbot_sentence-transformer-Q8_0-GGUF with Ollama:
ollama run hf.co/Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0
- Unsloth Studio
How to use Sleem247/chatbot_sentence-transformer-Q8_0-GGUF with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for Sleem247/chatbot_sentence-transformer-Q8_0-GGUF to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for Sleem247/chatbot_sentence-transformer-Q8_0-GGUF to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for Sleem247/chatbot_sentence-transformer-Q8_0-GGUF to start chatting
- Docker Model Runner
How to use Sleem247/chatbot_sentence-transformer-Q8_0-GGUF with Docker Model Runner:
docker model run hf.co/Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0
- Lemonade
How to use Sleem247/chatbot_sentence-transformer-Q8_0-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Sleem247/chatbot_sentence-transformer-Q8_0-GGUF:Q8_0
Run and chat with the model
lemonade run user.chatbot_sentence-transformer-Q8_0-GGUF-Q8_0
List all available models
lemonade list
- Atomic Chat
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:6284
- loss:TripletLoss
- llama-cpp
- gguf-my-repo
base_model: nikatonika/chatbot_sentence-transformer
widget:
- source_sentence: >-
Its personal! [SEP] I put you on uppers and you still yawned. Means its a
symptom, of being a big fat liar. Yawning is a side effect of some
antidepressants, apparently the ones youre on. Im not on antidepressants
Im on SPEEEEEED! Well that means its a symptom of a cerebral tumour. You
got six weeks to live. Mr. Welladjusted is as messed up as the rest of us.
Whwhy would you keep that a secret? Are you ashamed of recognising how
pathetic your life is? Its not a secret. House itsits... its personal! How
long has it been personal? Yawnings recent so! either you just started or
you changed prescription.
sentences:
- Yawnings recent so! either you just started or you changed prescription.
- >-
The High Sparrow has hundreds of Faith Militant surrounding him. Ser
Gregor will can’t face them all. And he won’t have to. He’ll only have
to face one.
- Whoa, whoa, whoa! Hey! Whoa!
- source_sentence: >-
Angio showed a clot in a branch of his middle cerebral artery. Started him
on streptokinase to break it up. Although! maybe we should have just
played a few rounds of Savagescape 2: The REvenge, because thats obviously
the best way to make someone feel better. [SEP] Hes referring to the pie.
Fourteen Patients with kidney biopsies. I tabulated them against their
urinalysis results. Six men, three diabetics, one LCDD, one kid, two
seniors, and by default, one! Cuddy. Uh! the biopsy was inconclusive. The
mass is near the center of her kidney. They couldnt get a readable sample.
Means theyll have to remove the mass to know if its cancer or not. The
file says theyre doing some imaging now, give the surgeon a better idea of
what hes dealing with. Can we order pizza? Theres nothing in the fridge.
At least not anymore. Whats happening with the Patient? You keep talking
like Wilson, your face will freeze like that.
sentences:
- You keep talking like Wilson, your face will freeze like that.
- >-
Of course he had brain cancer; Even oncologists dont screw up for 8
years.
- If you have any last words, now is the time.
- source_sentence: >-
And your life is simple? You went all the way up to the medical conference
to cozy up to Cuddy. Instead shes dating one of two people in the world
you think of as a friend. Theres no way thats not devastating. [SEP]
Cerebral vasculitis would explain both the eye and arm findings. Steroids
to treat, brain angiogram to confirm, EMG and nerve biopsy while youre at
it. Ill be at lunch. Id hire all four, but a fiveperson team seems
unwieldy. Who would you turn down? Ill have whatEver hes buying. Two
cheeseburgers and two large fries. There are a thousand people in the
world who want to be on your staff, but youre going after the four who
dont. They dont because their lives are irrelevantly and annoyingly
compliCated, which makes them confused, which makes them make poor
decisions. So I had an attraction of sorts.
sentences:
- Relax your bow arm.
- >-
Its a myth that fake hooters blow up at high altitude, shell be fine.
Just think of it as one giant rack for Mankind.
- So I had an attraction of sorts.
- source_sentence: >-
What about a glycogen storage disease like McArdles? It explains the pain.
Plus theres plenty of muscle cells in the wall of the intestine. [SEP]
Apparently he lied. Didnt think Id have to remind you of that remote
possibility. I have a DYFS inspection in less than 24 hours, so if you
cant control Nonmotor seizures. Sorry, I was thinking about the Patient.
What were you saying? Go on. The pain started in his abdomen near his
intestine. The first symptom has got to be key. Hes had multiple EEGs, all
of them cleaner and squeakier than Cuddys rubber nipples. Fourteen is
right. Go run an ischemic forearm test.
sentences:
- Fourteen is right. Go run an ischemic forearm test.
- >-
Do I look like a man without money? . Never trust looks. Until quite
recently, I was one of the richest men in the world. .
- >-
WhatEver test will prove there was an echovirusirusirus, and hes not
cured, obviously.
- source_sentence: >-
Like you are now? [SEP] That liver is going to somebody right now. Were
doing that surgery. If you do the surgery, youll be killing a mother of
four. Father of three. I was guessing. Naphthalene poisoning is the best
explanation we have for whats wrong with your son. It explains the
internal bleeding, the hemolytic anemia, the liver failure! it also
predicts whatll happen next. If you do the surgery hes gonna lay on that
table for fourteen hours while his body continues to burn fat and release
poison into his system. Either way, I did you a favor. Hes awake now,
youve got a chance to say goodbye.
sentences:
- >-
Spoken like a true Aussie. By the way, if you know where I can get me
the sheet music to Waltzing Matilda! Hey. Want some ice cream? Were
having a sundae bar.
- I know none of that. If I did, youd be the last to know.
- >-
If you do the surgery hes gonna lay on that table for fourteen hours
while his body continues to burn fat and release poison into
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- cosine_accuracy
model-index:
- name: SentenceTransformer based on distilbert/distilroberta-base
results:
- task:
type: triplet
name: Triplet
dataset:
name: dev evaluator
type: dev_evaluator
metrics:
- type: cosine_accuracy
value: 0.9809038639068604
name: Cosine Accuracy
Sleem247/chatbot_sentence-transformer-Q8_0-GGUF
This model was converted to GGUF format from nikatonika/chatbot_sentence-transformer using llama.cpp via the ggml.ai's GGUF-my-repo space.
Refer to the original model card for more details on the model.
Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)
brew install llama.cpp
Invoke the llama.cpp server or the CLI.
CLI:
llama-cli --hf-repo Sleem247/chatbot_sentence-transformer-Q8_0-GGUF --hf-file chatbot_sentence-transformer-q8_0.gguf -p "The meaning to life and the universe is"
Server:
llama-server --hf-repo Sleem247/chatbot_sentence-transformer-Q8_0-GGUF --hf-file chatbot_sentence-transformer-q8_0.gguf -c 2048
Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.
Step 1: Clone llama.cpp from GitHub.
git clone https://github.com/ggerganov/llama.cpp
Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
cd llama.cpp && LLAMA_CURL=1 make
Step 3: Run inference through the main binary.
./llama-cli --hf-repo Sleem247/chatbot_sentence-transformer-Q8_0-GGUF --hf-file chatbot_sentence-transformer-q8_0.gguf -p "The meaning to life and the universe is"
or
./llama-server --hf-repo Sleem247/chatbot_sentence-transformer-Q8_0-GGUF --hf-file chatbot_sentence-transformer-q8_0.gguf -c 2048