GGUF
English
llama.cpp
gemma4
gemma
deckard
iq4_nl
q8_0
imatrix
creative-writing
storymaker
roleplay
voice
uncensored
english
conversational
Instructions to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL # Run inference directly in the terminal: llama cli -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL # Run inference directly in the terminal: llama cli -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL # Run inference directly in the terminal: ./llama-cli -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL # Run inference directly in the terminal: ./build/bin/llama-cli -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Use Docker
docker model run hf.co/jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
- LM Studio
- Jan
- Ollama
How to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with Ollama:
ollama run hf.co/jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
- Unsloth Studio
How to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker to start chatting
- Pi
How to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Run Hermes
hermes
- Atomic Chat new
- OpenClaw new
How to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Docker Model Runner
How to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with Docker Model Runner:
docker model run hf.co/jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
- Lemonade
How to use jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jackasda211233/Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker:IQ4_NL
Run and chat with the model
lemonade run user.Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker-IQ4_NL
List all available models
lemonade list
Revise release card benchmark notes
Browse files
README.md
CHANGED
|
@@ -28,6 +28,10 @@ This is a normal `llama.cpp` GGUF quant of DavidAU's Deckard Gemma model:
|
|
| 28 |
|
| 29 |
The goal was a small, practical creative/voice/writing model that keeps more coherence than a plain Q4-style quant while still fitting comfortably on modest GPUs.
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
## What Is Special
|
| 32 |
|
| 33 |
This is not a plain one-shot Q4 quant.
|
|
@@ -123,54 +127,33 @@ Local file sizes from the experiment:
|
|
| 123 |
| IQ4_NL + attnQ8 last12 | 5,283,415,040 bytes |
|
| 124 |
| Q6_K reference | 6,184,790,688 bytes |
|
| 125 |
|
| 126 |
-
The `last6` mixed quant is only about 45 MB larger than the comparable b8763 IQ4_NL build
|
| 127 |
|
| 128 |
-
##
|
| 129 |
|
| 130 |
-
These are
|
| 131 |
|
| 132 |
-
|
| 133 |
-
-
|
| 134 |
-
-
|
|
|
|
| 135 |
|
| 136 |
-
|
| 137 |
|
| 138 |
```text
|
| 139 |
-
docs/dynamic_q8_mini_20260429_112232_report.md
|
| 140 |
-
```
|
| 141 |
-
|
| 142 |
-
Summary from that run:
|
| 143 |
-
|
| 144 |
-
| Model | Proxy score | Coverage | Worst 4-gram repeat | Decode tok/s | Prompt tok/s |
|
| 145 |
-
|---|---:|---:|---:|---:|---:|
|
| 146 |
-
| attnQ8_last6 | 76.407 | 0.489 | 1 | 86.579 | 1354.519 |
|
| 147 |
-
| Q6_K | 73.778 | 0.422 | 1 | 72.340 | 1480.222 |
|
| 148 |
-
| Q8_0 | 73.278 | 0.411 | 1 | 61.660 | 1502.476 |
|
| 149 |
-
| IQ4_NL b8763 | 70.185 | 0.356 | 1 | 87.449 | 1300.931 |
|
| 150 |
-
| attnQ8_last12 | 68.426 | 0.300 | 1 | 85.448 | 1372.260 |
|
| 151 |
-
|
| 152 |
-
Interpretation: `attnQ8_last6` was the best quality/size tradeoff in this test. The broader `attnQ8_last12` variant was worse, so simply adding more Q8 was not helpful.
|
| 153 |
-
|
| 154 |
-
## 5060 Ti Runtime Probe
|
| 155 |
-
|
| 156 |
-
On one RTX 5060 Ti, using standard `llama.cpp` with f32 KV and `-c 120000`:
|
| 157 |
-
|
| 158 |
-
```text
|
| 159 |
-
offloaded 43/43 layers to GPU
|
| 160 |
CUDA0 model buffer size = 2775.92 MiB
|
| 161 |
non-SWA KV buffer size = 3752.00 MiB
|
| 162 |
SWA KV buffer size = 60.00 MiB
|
| 163 |
compute buffer size = 572.00 MiB
|
| 164 |
-
observed VRAM = about 7796 MiB
|
| 165 |
-
prompt = 2816.56 tok/s on a 1352-token prompt
|
| 166 |
-
decode = 73.54 tok/s for 256 generated tokens
|
| 167 |
```
|
| 168 |
|
| 169 |
-
|
| 170 |
|
| 171 |
```text
|
| 172 |
-
|
| 173 |
-
|
|
|
|
|
|
|
| 174 |
```
|
| 175 |
|
| 176 |
## Intended Use
|
|
@@ -186,11 +169,11 @@ Because the source model is an uncensored/aliterated model, apply your own safet
|
|
| 186 |
3. Built an English creative/voice/logic imatrix calibration set.
|
| 187 |
4. Quantized with `IQ4_NL`.
|
| 188 |
5. Re-quantized selected late attention tensors in blocks `36..41` to `Q8_0`.
|
| 189 |
-
6.
|
| 190 |
|
| 191 |
## Limitations
|
| 192 |
|
| 193 |
-
-
|
|
|
|
| 194 |
- The no-thinking template is recommended for creative use because it reduces visible planning/scaffold output.
|
| 195 |
- This is a GGUF release for `llama.cpp`; it is not a safetensors checkpoint.
|
| 196 |
-
|
|
|
|
| 28 |
|
| 29 |
The goal was a small, practical creative/voice/writing model that keeps more coherence than a plain Q4-style quant while still fitting comfortably on modest GPUs.
|
| 30 |
|
| 31 |
+
This was an unplanned release. The author was testing models for creative writing use, found a quantized GGUF of this model that showed severe looping and quality degradation, and built this version to check whether the issue came from the core finetune or from the quantization. The result was unexpectedly useful: creative, able to reason, good at following instructions, strong for roleplay, and expressive in a way that made it enjoyable to use. Because of that, it was uploaded for anyone else interested in this niche.
|
| 32 |
+
|
| 33 |
+
Because this was exploratory rather than a planned release, it has not gone through the author's usual comprehensive release process. A gender-affirming finetune based on the DavidAU model may come later for people interested in that direction.
|
| 34 |
+
|
| 35 |
## What Is Special
|
| 36 |
|
| 37 |
This is not a plain one-shot Q4 quant.
|
|
|
|
| 127 |
| IQ4_NL + attnQ8 last12 | 5,283,415,040 bytes |
|
| 128 |
| Q6_K reference | 6,184,790,688 bytes |
|
| 129 |
|
| 130 |
+
The `last6` mixed quant is only about 45 MB larger than the comparable b8763 IQ4_NL build.
|
| 131 |
|
| 132 |
+
## Runtime Tests
|
| 133 |
|
| 134 |
+
These are direct runtime probes for this exact released GGUF using standard `llama.cpp`. No custom fork was used.
|
| 135 |
|
| 136 |
+
| GPU | Runtime | Context / KV | Prompt workload | Decode workload | Prompt tok/s | Decode tok/s | VRAM note |
|
| 137 |
+
|---|---|---|---:|---:|---:|---:|---|
|
| 138 |
+
| RTX 5060 Ti | `llama.cpp` b8763 server | `-c 120000`, f32 K/V, all layers offloaded | 1352 tokens | 256 tokens | 2816.56 | 73.54 | about 7796 MiB observed |
|
| 139 |
+
| RTX 5090 | `llama.cpp` b8763 server | `-c 120000`, f32 K/V, all layers offloaded | 1122 tokens | 40 tokens | 4251.93 | 138.82 | 7159 MiB llama.cpp self allocation |
|
| 140 |
|
| 141 |
+
5060 Ti log summary:
|
| 142 |
|
| 143 |
```text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
CUDA0 model buffer size = 2775.92 MiB
|
| 145 |
non-SWA KV buffer size = 3752.00 MiB
|
| 146 |
SWA KV buffer size = 60.00 MiB
|
| 147 |
compute buffer size = 572.00 MiB
|
|
|
|
|
|
|
|
|
|
| 148 |
```
|
| 149 |
|
| 150 |
+
5090 log summary:
|
| 151 |
|
| 152 |
```text
|
| 153 |
+
CUDA0 self allocation = 7159 MiB
|
| 154 |
+
model buffer = 2775 MiB
|
| 155 |
+
context buffer = 3812 MiB
|
| 156 |
+
compute buffer = 572 MiB
|
| 157 |
```
|
| 158 |
|
| 159 |
## Intended Use
|
|
|
|
| 169 |
3. Built an English creative/voice/logic imatrix calibration set.
|
| 170 |
4. Quantized with `IQ4_NL`.
|
| 171 |
5. Re-quantized selected late attention tensors in blocks `36..41` to `Q8_0`.
|
| 172 |
+
6. Runtime-tested the released GGUF on RTX 5060 Ti and RTX 5090 with standard `llama.cpp`.
|
| 173 |
|
| 174 |
## Limitations
|
| 175 |
|
| 176 |
+
- This was an unplanned exploratory release, not a broad benchmarked model release.
|
| 177 |
+
- Public runtime numbers are limited to the two direct tests above.
|
| 178 |
- The no-thinking template is recommended for creative use because it reduces visible planning/scaffold output.
|
| 179 |
- This is a GGUF release for `llama.cpp`; it is not a safetensors checkpoint.
|
|
|