Instructions to use flyingfishinwater/good_and_small_models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use flyingfishinwater/good_and_small_models with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf flyingfishinwater/good_and_small_models:Q4_K_M # Run inference directly in the terminal: llama cli -hf flyingfishinwater/good_and_small_models:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf flyingfishinwater/good_and_small_models:Q4_K_M # Run inference directly in the terminal: llama cli -hf flyingfishinwater/good_and_small_models:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf flyingfishinwater/good_and_small_models:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf flyingfishinwater/good_and_small_models:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf flyingfishinwater/good_and_small_models:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf flyingfishinwater/good_and_small_models:Q4_K_M
Use Docker
docker model run hf.co/flyingfishinwater/good_and_small_models:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use flyingfishinwater/good_and_small_models with Ollama:
ollama run hf.co/flyingfishinwater/good_and_small_models:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use flyingfishinwater/good_and_small_models with Docker Model Runner:
docker model run hf.co/flyingfishinwater/good_and_small_models:Q4_K_M
- Lemonade
How to use flyingfishinwater/good_and_small_models with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull flyingfishinwater/good_and_small_models:Q4_K_M
Run and chat with the model
lemonade run user.good_and_small_models-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Update README.md
Browse files
README.md
CHANGED
|
@@ -22,10 +22,11 @@ Llama 3 is the latest and most advanced LLM trained over 15T tokens, which impro
|
|
| 22 |
|
| 23 |
**Prompt Format:**
|
| 24 |
```
|
| 25 |
-
<|
|
| 26 |
|
| 27 |
{{prompt}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
|
| 28 |
|
|
|
|
| 29 |
|
| 30 |
```
|
| 31 |
|
|
@@ -97,7 +98,7 @@ The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens.
|
|
| 97 |
|
| 98 |
**Prompt Format:**
|
| 99 |
```
|
| 100 |
-
<|
|
| 101 |
```
|
| 102 |
|
| 103 |
**Template Name:** TinyLlama
|
|
@@ -146,12 +147,12 @@ The Mistral-7B-v0.2 Large Language Model (LLM) is a pretrained generative text m
|
|
| 146 |
|
| 147 |
---
|
| 148 |
|
| 149 |
-
# OpenChat 3.5
|
| 150 |
OpenChat is an innovative library of open-source language models, fine-tuned with C-RLFT - a strategy inspired by offline reinforcement learning. Our models learn from mixed-quality data without preference labels, delivering exceptional performance on par with ChatGPT, even with a 7B model. Despite our simple approach, we are committed to developing a high-performance, commercially viable, open-source large language model, and we continue to make significant strides toward this vision.
|
| 151 |
|
| 152 |
**Model Intention:** It's a 7B large model and performs really good for Q&A. But it requires a high-end device to run.
|
| 153 |
|
| 154 |
-
**Model URL:** [https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/openchat-3.5-
|
| 155 |
|
| 156 |
**Model Info URL:** [https://huggingface.co/openchat/openchat_3.5](https://huggingface.co/openchat/openchat_3.5)
|
| 157 |
|
|
@@ -161,13 +162,13 @@ OpenChat is an innovative library of open-source language models, fine-tuned wit
|
|
| 161 |
|
| 162 |
**Developer:** [https://openchat.team/](https://openchat.team/)
|
| 163 |
|
| 164 |
-
**File Size:**
|
| 165 |
|
| 166 |
-
**Context Length:**
|
| 167 |
|
| 168 |
**Prompt Format:**
|
| 169 |
```
|
| 170 |
-
|
| 171 |
```
|
| 172 |
|
| 173 |
**Template Name:** Mistral
|
|
@@ -186,7 +187,7 @@ Phi-2 is a Transformer with 2.7 billion parameters. It was trained using the sam
|
|
| 186 |
|
| 187 |
**Model Intention:** It's a 2.7B model and is intended for QA, chat, and code purposes
|
| 188 |
|
| 189 |
-
**Model URL:** [https://huggingface.co/ggml-org/models/resolve/main/phi-2
|
| 190 |
|
| 191 |
**Model Info URL:** [https://huggingface.co/microsoft/phi-2](https://huggingface.co/microsoft/phi-2)
|
| 192 |
|
|
@@ -196,9 +197,9 @@ Phi-2 is a Transformer with 2.7 billion parameters. It was trained using the sam
|
|
| 196 |
|
| 197 |
**Developer:** [https://huggingface.co/microsoft/phi-2](https://huggingface.co/microsoft/phi-2)
|
| 198 |
|
| 199 |
-
**File Size:**
|
| 200 |
|
| 201 |
-
**Context Length:**
|
| 202 |
|
| 203 |
**Prompt Format:**
|
| 204 |
```
|
|
@@ -222,7 +223,7 @@ The Yi series models are the next generation of open-source large language model
|
|
| 222 |
|
| 223 |
**Model Intention:** It's a 6B model and can understand English and Chinese. It's good for QA and Chat
|
| 224 |
|
| 225 |
-
**Model URL:** [https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/yi-
|
| 226 |
|
| 227 |
**Model Info URL:** [https://huggingface.co/01-ai/Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat)
|
| 228 |
|
|
@@ -232,9 +233,9 @@ The Yi series models are the next generation of open-source large language model
|
|
| 232 |
|
| 233 |
**Developer:** [https://01.ai/](https://01.ai/)
|
| 234 |
|
| 235 |
-
**File Size:**
|
| 236 |
|
| 237 |
-
**Context Length:**
|
| 238 |
|
| 239 |
**Prompt Format:**
|
| 240 |
```
|
|
@@ -297,7 +298,7 @@ Gemma is a family of lightweight, state-of-the-art open models built from the sa
|
|
| 297 |
# StarCoder2 3B
|
| 298 |
StarCoder2-3B model is a 3B parameter model trained on 17 programming languages from The Stack v2, with opt-out requests excluded. The model uses Grouped Query Attention, a context window of 16,384 tokens with a sliding window attention of 4,096 tokens, and was trained using the Fill-in-the-Middle objective on 3+ trillion tokens
|
| 299 |
|
| 300 |
-
**Model Intention:** The model is good at 17 programming languages.
|
| 301 |
|
| 302 |
**Model URL:** [https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/starcoder2-3b-instruct-gguf_Q8_0.gguf?download=true](https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/starcoder2-3b-instruct-gguf_Q8_0.gguf?download=true)
|
| 303 |
|
|
@@ -311,12 +312,11 @@ StarCoder2-3B model is a 3B parameter model trained on 17 programming languages
|
|
| 311 |
|
| 312 |
**File Size:** 3220 MB
|
| 313 |
|
| 314 |
-
**Context Length:**
|
| 315 |
|
| 316 |
**Prompt Format:**
|
| 317 |
```
|
| 318 |
-
|
| 319 |
-
{{prompt}}### Response
|
| 320 |
|
| 321 |
```
|
| 322 |
|
|
@@ -368,6 +368,45 @@ Chinese Tiny LLM 2B 是首个以中文为中心的大型语言模型,主要在
|
|
| 368 |
**Parse Special Tokens:** Yes
|
| 369 |
|
| 370 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 371 |
---
|
| 372 |
|
| 373 |
# Dophin 2.8 Mistralv02 7B
|
|
@@ -387,11 +426,11 @@ This model is based on Mistral-7b-v0.2 with 16k context lengths. It's a uncensor
|
|
| 387 |
|
| 388 |
**File Size:** 2728 MB
|
| 389 |
|
| 390 |
-
**Context Length:**
|
| 391 |
|
| 392 |
**Prompt Format:**
|
| 393 |
```
|
| 394 |
-
<|im_start|>user
|
| 395 |
{{prompt}}
|
| 396 |
<|im_end|>
|
| 397 |
<|im_start|>assistant
|
|
@@ -444,4 +483,4 @@ ASSISTANT:
|
|
| 444 |
**Parse Special Tokens:** Yes
|
| 445 |
|
| 446 |
|
| 447 |
-
---
|
|
|
|
| 22 |
|
| 23 |
**Prompt Format:**
|
| 24 |
```
|
| 25 |
+
<|start_header_id|>user<|end_header_id|>
|
| 26 |
|
| 27 |
{{prompt}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
|
| 28 |
|
| 29 |
+
assistant
|
| 30 |
|
| 31 |
```
|
| 32 |
|
|
|
|
| 98 |
|
| 99 |
**Prompt Format:**
|
| 100 |
```
|
| 101 |
+
<|user|>{{prompt}}</s><|assistant|>
|
| 102 |
```
|
| 103 |
|
| 104 |
**Template Name:** TinyLlama
|
|
|
|
| 147 |
|
| 148 |
---
|
| 149 |
|
| 150 |
+
# OpenChat 3.5(0106)
|
| 151 |
OpenChat is an innovative library of open-source language models, fine-tuned with C-RLFT - a strategy inspired by offline reinforcement learning. Our models learn from mixed-quality data without preference labels, delivering exceptional performance on par with ChatGPT, even with a 7B model. Despite our simple approach, we are committed to developing a high-performance, commercially viable, open-source large language model, and we continue to make significant strides toward this vision.
|
| 152 |
|
| 153 |
**Model Intention:** It's a 7B large model and performs really good for Q&A. But it requires a high-end device to run.
|
| 154 |
|
| 155 |
+
**Model URL:** [https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/openchat-3.5-0106.Q3_K_M.gguf?download=true](https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/openchat-3.5-0106.Q3_K_M.gguf?download=true)
|
| 156 |
|
| 157 |
**Model Info URL:** [https://huggingface.co/openchat/openchat_3.5](https://huggingface.co/openchat/openchat_3.5)
|
| 158 |
|
|
|
|
| 162 |
|
| 163 |
**Developer:** [https://openchat.team/](https://openchat.team/)
|
| 164 |
|
| 165 |
+
**File Size:** 3520 MB
|
| 166 |
|
| 167 |
+
**Context Length:** 8192 tokens
|
| 168 |
|
| 169 |
**Prompt Format:**
|
| 170 |
```
|
| 171 |
+
GPT4 Correct User: {{prompt}}<|end_of_turn|>GPT4 Correct Assistant:
|
| 172 |
```
|
| 173 |
|
| 174 |
**Template Name:** Mistral
|
|
|
|
| 187 |
|
| 188 |
**Model Intention:** It's a 2.7B model and is intended for QA, chat, and code purposes
|
| 189 |
|
| 190 |
+
**Model URL:** [https://huggingface.co/ggml-org/models/resolve/main/phi-2.Q5_K_M.gguf?download=true](https://huggingface.co/ggml-org/models/resolve/main/phi-2.Q5_K_M.gguf?download=true)
|
| 191 |
|
| 192 |
**Model Info URL:** [https://huggingface.co/microsoft/phi-2](https://huggingface.co/microsoft/phi-2)
|
| 193 |
|
|
|
|
| 197 |
|
| 198 |
**Developer:** [https://huggingface.co/microsoft/phi-2](https://huggingface.co/microsoft/phi-2)
|
| 199 |
|
| 200 |
+
**File Size:** 2070 MB
|
| 201 |
|
| 202 |
+
**Context Length:** 2048 tokens
|
| 203 |
|
| 204 |
**Prompt Format:**
|
| 205 |
```
|
|
|
|
| 223 |
|
| 224 |
**Model Intention:** It's a 6B model and can understand English and Chinese. It's good for QA and Chat
|
| 225 |
|
| 226 |
+
**Model URL:** [https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/yi-chat-6b.Q4_K_M.gguf?download=true](https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/yi-chat-6b.Q4_K_M.gguf?download=true)
|
| 227 |
|
| 228 |
**Model Info URL:** [https://huggingface.co/01-ai/Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat)
|
| 229 |
|
|
|
|
| 233 |
|
| 234 |
**Developer:** [https://01.ai/](https://01.ai/)
|
| 235 |
|
| 236 |
+
**File Size:** 3670 MB
|
| 237 |
|
| 238 |
+
**Context Length:** 4096 tokens
|
| 239 |
|
| 240 |
**Prompt Format:**
|
| 241 |
```
|
|
|
|
| 298 |
# StarCoder2 3B
|
| 299 |
StarCoder2-3B model is a 3B parameter model trained on 17 programming languages from The Stack v2, with opt-out requests excluded. The model uses Grouped Query Attention, a context window of 16,384 tokens with a sliding window attention of 4,096 tokens, and was trained using the Fill-in-the-Middle objective on 3+ trillion tokens
|
| 300 |
|
| 301 |
+
**Model Intention:** The model is good at 17 programming languages. By just start with your codes, the model will finish it.
|
| 302 |
|
| 303 |
**Model URL:** [https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/starcoder2-3b-instruct-gguf_Q8_0.gguf?download=true](https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/starcoder2-3b-instruct-gguf_Q8_0.gguf?download=true)
|
| 304 |
|
|
|
|
| 312 |
|
| 313 |
**File Size:** 3220 MB
|
| 314 |
|
| 315 |
+
**Context Length:** 16384 tokens
|
| 316 |
|
| 317 |
**Prompt Format:**
|
| 318 |
```
|
| 319 |
+
{{prompt}}
|
|
|
|
| 320 |
|
| 321 |
```
|
| 322 |
|
|
|
|
| 368 |
**Parse Special Tokens:** Yes
|
| 369 |
|
| 370 |
|
| 371 |
+
---
|
| 372 |
+
|
| 373 |
+
# Qwen1.5 4B Chat
|
| 374 |
+
Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. It supports both Chinese and English. 通义千问是阿里巴巴公司开发的大大预言模型,支持中英文双语。
|
| 375 |
+
|
| 376 |
+
**Model Intention:** It's one of the best LLM that supports Chinese and English. 这是支持中英双语的最佳的大语言模型之一。
|
| 377 |
+
|
| 378 |
+
**Model URL:** [https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/qwen1_5-4b-chat-q4_k_m.gguf?download=true](https://huggingface.co/flyingfishinwater/goodmodels/resolve/main/qwen1_5-4b-chat-q4_k_m.gguf?download=true)
|
| 379 |
+
|
| 380 |
+
**Model Info URL:** [https://huggingface.co/Qwen/Qwen1.5-4B-Chat-GGUF](https://huggingface.co/Qwen/Qwen1.5-4B-Chat-GGUF)
|
| 381 |
+
|
| 382 |
+
**Model License:** [License Info](https://huggingface.co/Qwen/Qwen1.5-4B-Chat/raw/main/LICENSE)
|
| 383 |
+
|
| 384 |
+
**Model Description:** Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. It supports both Chinese and English. 通义千问是阿里巴巴公司开发的大大预言模型,支持中英文双语。
|
| 385 |
+
|
| 386 |
+
**Developer:** [https://qwenlm.github.io/](https://qwenlm.github.io/)
|
| 387 |
+
|
| 388 |
+
**File Size:** 2460 MB
|
| 389 |
+
|
| 390 |
+
**Context Length:** 32768 tokens
|
| 391 |
+
|
| 392 |
+
**Prompt Format:**
|
| 393 |
+
```
|
| 394 |
+
<|im_start|>user
|
| 395 |
+
{{prompt}}
|
| 396 |
+
<|im_end|>
|
| 397 |
+
<|im_start|>assistant
|
| 398 |
+
|
| 399 |
+
```
|
| 400 |
+
|
| 401 |
+
**Template Name:** chatml
|
| 402 |
+
|
| 403 |
+
**Add BOS Token:** Yes
|
| 404 |
+
|
| 405 |
+
**Add EOS Token:** No
|
| 406 |
+
|
| 407 |
+
**Parse Special Tokens:** Yes
|
| 408 |
+
|
| 409 |
+
|
| 410 |
---
|
| 411 |
|
| 412 |
# Dophin 2.8 Mistralv02 7B
|
|
|
|
| 426 |
|
| 427 |
**File Size:** 2728 MB
|
| 428 |
|
| 429 |
+
**Context Length:** 32768 tokens
|
| 430 |
|
| 431 |
**Prompt Format:**
|
| 432 |
```
|
| 433 |
+
<s><|im_start|>user
|
| 434 |
{{prompt}}
|
| 435 |
<|im_end|>
|
| 436 |
<|im_start|>assistant
|
|
|
|
| 483 |
**Parse Special Tokens:** Yes
|
| 484 |
|
| 485 |
|
| 486 |
+
---
|