How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ProCreations/grug-3b-gguf:# Run inference directly in the terminal:
llama cli -hf ProCreations/grug-3b-gguf:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ProCreations/grug-3b-gguf:# Run inference directly in the terminal:
./llama-cli -hf ProCreations/grug-3b-gguf:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ProCreations/grug-3b-gguf:# Run inference directly in the terminal:
./build/bin/llama-cli -hf ProCreations/grug-3b-gguf:Use Docker
docker model run hf.co/ProCreations/grug-3b-gguf:Quick Links
grug-3b-gguf
gguf of ProCreations/grug-3b.
grug think in grug inside <think>, then answer normal english. think short for
small question (13 token median), long for hard one (737 token median).
| file | size |
|---|---|
| grug-3b-f16.gguf | 8.34 GB |
| grug-3b-Q8_0.gguf | 4.43 GB |
| grug-3b-Q6_K.gguf | 3.42 GB |
| grug-3b-Q5_K_M.gguf | 2.99 GB |
| grug-3b-Q4_K_M.gguf | 2.57 GB |
| grug-3b-Q3_K_M.gguf | 2.17 GB |
want q4 that hold up better? grug also train one q4-aware: ProCreations/grug-3b-qat-q4-gguf.
llama.cpp support
Nanbeige4.2 not in upstream llama.cpp yet (issue
#26086). Nanbeige team PR
#25994 add it - weight-shared
depth loop, num_loops=2. until merge, build from that branch:
git clone --depth 1 --branch nanbeige42 https://github.com/Nanbeige/llama.cpp
cd llama.cpp && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bin/llama-cli -m grug-3b-Q4_K_M.gguf -p "What is 12 times 12?"
these gguf converted and load-probed with that branch.
- Downloads last month
- 1,435
Hardware compatibility
Log In to add your hardware
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
Model tree for ProCreations/grug-3b-gguf
Base model
Nanbeige/Nanbeige4.2-3B-Base Finetuned
Nanbeige/Nanbeige4.2-3B Finetuned
ProCreations/grug-3b
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf ProCreations/grug-3b-gguf:# Run inference directly in the terminal: llama cli -hf ProCreations/grug-3b-gguf: