Instructions to use mradermacher/granite-3.0-1b-a400m-base-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/granite-3.0-1b-a400m-base-i1-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/granite-3.0-1b-a400m-base-i1-GGUF", dtype="auto")

llama-cpp-python

How to use mradermacher/granite-3.0-1b-a400m-base-i1-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="mradermacher/granite-3.0-1b-a400m-base-i1-GGUF",
	filename="granite-3.0-1b-a400m-base.i1-IQ1_M.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/granite-3.0-1b-a400m-base-i1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/granite-3.0-1b-a400m-base-i1-GGUF with Ollama:
```
ollama run hf.co/mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M
```

Unsloth Studio

How to use mradermacher/granite-3.0-1b-a400m-base-i1-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/granite-3.0-1b-a400m-base-i1-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/granite-3.0-1b-a400m-base-i1-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for mradermacher/granite-3.0-1b-a400m-base-i1-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use mradermacher/granite-3.0-1b-a400m-base-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/granite-3.0-1b-a400m-base-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/granite-3.0-1b-a400m-base-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.granite-3.0-1b-a400m-base-i1-GGUF-Q4_K_M

List all available models

lemonade list

mradermacher commited on Jan 15, 2025

Commit

29c2f43

verified ·

1 Parent(s): 906e041

uploaded from kaos

Browse files

Files changed (25) hide show

.gitattributes +24 -0
granite-3.0-1b-a400m-base.i1-IQ1_M.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ1_S.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ2_M.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ2_S.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ2_XS.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ2_XXS.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ3_M.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ3_S.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ3_XS.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ3_XXS.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ4_NL.gguf +3 -0
granite-3.0-1b-a400m-base.i1-IQ4_XS.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q2_K.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q2_K_S.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q3_K_L.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q3_K_M.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q3_K_S.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q4_0.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q4_1.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q4_K_M.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q4_K_S.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q5_K_M.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q5_K_S.gguf +3 -0
granite-3.0-1b-a400m-base.i1-Q6_K.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imatrix.dat filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imatrix.dat filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+granite-3.0-1b-a400m-base.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

granite-3.0-1b-a400m-base.i1-IQ1_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b207cd24ec9bebb811fdd49ab14d1a963da7631f88369b2a38be2f3512c4ff32
+size 347770592

granite-3.0-1b-a400m-base.i1-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9c781bab20f4abc80f73520bfc7176a38d06e343d99cd646d0342850fb0fd143
+size 319753952

granite-3.0-1b-a400m-base.i1-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a07d2d1ae73accf74891004da5b8f9e487c4c48a8fe1cdc4f4b25971c0f8f806
+size 483725024

granite-3.0-1b-a400m-base.i1-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:af1214afb624cff87ed36b34c7992824f9b2e95847fbf1ef8a4242c11cf55338
+size 446369504

granite-3.0-1b-a400m-base.i1-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:76efa317a5cf35ff206d003ebd08cc01a4747934cdd37be953799c2a937425c9
+size 432606944

granite-3.0-1b-a400m-base.i1-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fdfcd69422710d67180218f3d613eb9f7ba03a1f047dc577e19fd61e4d55f8c3
+size 394464992

granite-3.0-1b-a400m-base.i1-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cc311c7b6f64cdb30f27414f18ea4478d7673b4682925fb5ea5ab404d1ebfb8c
+size 631181024

granite-3.0-1b-a400m-base.i1-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:989c5671a153f5943849f641a0fd15f0102c4daac6f018b396c6c7883b2cd1b4
+size 619482848

granite-3.0-1b-a400m-base.i1-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ea603101846fccded928c9d548331f0d97254286f61f750377d29c0958ad6341
+size 589401824

granite-3.0-1b-a400m-base.i1-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c527a0ed3ef6cc4cd97ca47eac4dd489df9c0500639d42f87914e1cade3276c6
+size 551456480

granite-3.0-1b-a400m-base.i1-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2cf8131c233ae5cb042a0df1a7633f06a75f624ad2efcc7cda3082d4878b24c0
+size 796626656

granite-3.0-1b-a400m-base.i1-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:42545087e510333987f14c54724b988d934d37c94610ddcbc7775b4c42990d2a
+size 754945760

granite-3.0-1b-a400m-base.i1-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:28bfb5c22005a4ba96a0ce9ef01737d22d9f79432d17f44ae7924e4edb365b21
+size 528748256

granite-3.0-1b-a400m-base.i1-Q2_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a8480ec44e3450203efd6d49247cfa02cc4ea0b93c770ec9e1e1c6b432ff080e
+size 495816416

granite-3.0-1b-a400m-base.i1-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f440e918815112b680a2fea563918f8eb599b75a468fa8904785baab8c8f56dc
+size 733023968

granite-3.0-1b-a400m-base.i1-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:be9606fa6537a658563a2d6cd4c9d8cb1b92e0c00d0badb3bef2ede060bb3f37
+size 680201952

granite-3.0-1b-a400m-base.i1-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fed8fdcea7e60520b0900f69a761a16f87cb5bab90ebde9585f8e9946200a3e1
+size 619482848

granite-3.0-1b-a400m-base.i1-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:296633c54916246efbbea0e9a2337e1858a018857755431070cad5d05f21c656
+size 799772384

granite-3.0-1b-a400m-base.i1-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:199a11ade2db34b6f8a03fcd8023ae8d9e09a2e5430ccafff2cfbd97effa5818
+size 879988448

granite-3.0-1b-a400m-base.i1-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:984558b62c6a9cde846d019c3a91001f7fccb0f02b7274a2868e3530698aeca1
+size 850153184

granite-3.0-1b-a400m-base.i1-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b240bcd60eb4ebc5f4e255cfcabc69f21cf4598df7ecc6956f1c4002c97432df
+size 803180256

granite-3.0-1b-a400m-base.i1-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8f357ca16dedb6ba956fa014cfc3f9f22ab2dae0367bd6739763b48b5d103da2
+size 990924512

granite-3.0-1b-a400m-base.i1-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3a49b3183399804e24c6969f958c56cb973fd665412696984da6bf5d1f61de45
+size 963350240

granite-3.0-1b-a400m-base.i1-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f29866ea78208944c5799d0f609b537554d7189bfca33de3ba038b755744eca9
+size 1140494048