Instructions to use mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF with Ollama:
```
ollama run hf.co/mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M
```

Unsloth Studio

How to use mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF to start chatting

Docker Model Runner
How to use mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/TinyMixtral-4x248M-MoE-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.TinyMixtral-4x248M-MoE-i1-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

mradermacher commited on Jan 7, 2025

Commit

1f046c9

verified ·

1 Parent(s): 3761247

uploaded from rain

Browse files

Files changed (25) hide show

.gitattributes +24 -0
TinyMixtral-4x248M-MoE.i1-IQ1_M.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ1_S.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ2_M.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ2_S.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ2_XS.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ2_XXS.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ3_M.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ3_S.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ3_XS.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ3_XXS.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ4_NL.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-IQ4_XS.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q2_K.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q2_K_S.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q3_K_L.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q3_K_M.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q3_K_S.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q4_0.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q4_1.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q4_K_M.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q4_K_S.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q5_K_M.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q5_K_S.gguf +3 -0
TinyMixtral-4x248M-MoE.i1-Q6_K.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imatrix.dat filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imatrix.dat filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+TinyMixtral-4x248M-MoE.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

TinyMixtral-4x248M-MoE.i1-IQ1_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1976aa6c4c5debd3c6e2f8f20b595a1744f58ce8ec7b972005e8e22a9e67bcce
+size 176719104

TinyMixtral-4x248M-MoE.i1-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:941c5edda072c170def056bbd5c0095692c36afcd4248bb90d1a1c49001dbf61
+size 162587904

TinyMixtral-4x248M-MoE.i1-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e0a3c6971d068b15dc3b41411f9ed972deb0274bc7ba08a753debe032db051a6
+size 245149248

TinyMixtral-4x248M-MoE.i1-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:598019b25ec55c9cf30fc6ef45d0e52f77a6ec81bbdd39346a461bb3431685a6
+size 226307648

TinyMixtral-4x248M-MoE.i1-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:038c1429e30a6b932711babb304ba46d71c251fe8acd6ab3419f142859053818
+size 219505920

TinyMixtral-4x248M-MoE.i1-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:47cdf3ff44ca411d90c1dad82ef028b432f69fc83c4bd857af0b6182405a57f4
+size 200271104

TinyMixtral-4x248M-MoE.i1-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2610b4d72a698f0949eaf2db2c1d86c627f561d4df45ada0a35b9f71aab7d420
+size 319346656

TinyMixtral-4x248M-MoE.i1-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4289bfd4d78d04d1c8a36361a4e797235219b332ecf13117bebdd768199bc57e
+size 315447264

TinyMixtral-4x248M-MoE.i1-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ee76c003a317bd0aa8bd43558498a834932f2f3df0e34f781ff8b030d544c404
+size 300554208

TinyMixtral-4x248M-MoE.i1-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:74f6a07d86977beb9406d275f370aed3b7300587e297168d26c381252cd06612
+size 281062976

TinyMixtral-4x248M-MoE.i1-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dc22a7d24763409de27ad8b7c764d75c20a6ac92ce94c6d4189a66affd62bb96
+size 404169056

TinyMixtral-4x248M-MoE.i1-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:51807cf14c852a2f80855f69b75c92bc6f678b3eff1968c441fd23a242592a13
+size 383385856

TinyMixtral-4x248M-MoE.i1-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5119793d004009898087509c23633050b5b325b2ddc31165946d1c414d1e7109
+size 269627040

TinyMixtral-4x248M-MoE.i1-Q2_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d32f820e20fac73a760d305d4e655b099dc3666bd916a9cde14c2e1098f44a84
+size 251834016

TinyMixtral-4x248M-MoE.i1-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3463fe8568b3313c995cd88675cffb96494e6194faaafecf304ed55ef9b0a24c
+size 370989024

TinyMixtral-4x248M-MoE.i1-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a2ba67c7efb2ba133193033a49322ccc28428032867e3c419d6623aa19dfeba4
+size 343922656

TinyMixtral-4x248M-MoE.i1-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a035c09d85cbf7f11d353a7b36f100d93d81ed671670e4052eed0a9f5ad4606a
+size 315029472

TinyMixtral-4x248M-MoE.i1-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:df32eb0c695f1d96a426fc7462451bd7d61c4445e2d9d2c7c5bad5c00fa384e1
+size 404824416

TinyMixtral-4x248M-MoE.i1-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:eebaefe1d08d70673bf98de543959f24468de28bf3af87234a89c5c7d6c8aba0
+size 445538848

TinyMixtral-4x248M-MoE.i1-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b90e0951bbb19b809e92eeeea99955770625ac2db56ad40379bced7fce2ea62a
+size 430133600

TinyMixtral-4x248M-MoE.i1-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f0867b3a339906532f090578b6dd9b862a1e2e4109341edce3e78970bbaaeb76
+size 406004064

TinyMixtral-4x248M-MoE.i1-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c0ce483e66bf4e4dbd3e8e6be94a836bcfbfa3254eec60093c1d03ccb19193bf
+size 500880096

TinyMixtral-4x248M-MoE.i1-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:21816772295d30a401c2a86b69ab20af26e1d8e5864f1a253ad84c5ac2722e13
+size 487301856

TinyMixtral-4x248M-MoE.i1-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:20c62fe0981474b36791d94a82bddd6dab62210e0fb33fc9e8c16192ab0893af
+size 576048256