README.md

---
license: apache-2.0
language:
- en
library_name: llama.cpp
base_model:
- DavidAU/gemma-4-E4B-it-The-DECKARD-Claude-Opus-Expresso-Universe-HERETIC-UNCENSORED-Thinking
tags:
- gguf
- llama.cpp
- gemma4
- gemma
- deckard
- iq4_nl
- q8_0
- imatrix
- creative-writing
- voice
- uncensored
- english
---

# Gemma 4 E4B Deckard AttnQ8 Last6 IQ4_NL GGUF

This is a normal `llama.cpp` GGUF quant of DavidAU's Deckard Gemma model:

<https://huggingface.co/DavidAU/gemma-4-E4B-it-The-DECKARD-Claude-Opus-Expresso-Universe-HERETIC-UNCENSORED-Thinking>

The goal was a small, practical creative/voice/writing model that keeps more coherence than a plain Q4-style quant while still fitting comfortably on modest GPUs.

This was an unplanned release. The author was testing models for creative writing use, found a quantized GGUF of this model that showed severe looping and quality degradation, and built this version to check whether the issue came from the core finetune or from the quantization. The result was unexpectedly useful: creative, able to reason, good at following instructions, strong for roleplay, and expressive in a way that made it enjoyable to use. Because of that, it was uploaded for anyone else interested in this niche.

Because this was exploratory rather than a planned release, it has not gone through the author's usual comprehensive release process. A gender-affirming finetune based on the DavidAU model may come later for people interested in that direction.

## What Is Special

This is not a plain one-shot Q4 quant.

The released GGUF is:

- base quant: `IQ4_NL`
- calibration: English creative writing, general logic, emotional understanding, and voice/vocal-concept prompts
- imatrix: 1000 calibration chunks at ctx512, 342 importance-matrix entries
- protected tensors: attention `q`, `k`, `v`, and output projections in blocks `36..41`
- protected quant type: `Q8_0`

In other words, most of the model stays Q4-sized, while the final six attention blocks are kept at higher precision. The intent is to spend a small amount of extra size on the late attention path, where coherence, style continuity, and repetition stability are more likely to matter for writing-style use.

## Files

Main model:

```text
Gemma-4-E4B-Deckard-AttnQ8-Last6-IQ4_NL-Imatrix-CreativeVoice.gguf
```

Optional template used in our deployment:

```text
gemma4_deckard_no_think.jinja
```

The GGUF works with standard `llama.cpp`. No custom fork is required.

## Suggested llama.cpp Command

Example server command:

```bash
llama-server \
  -m Gemma-4-E4B-Deckard-AttnQ8-Last6-IQ4_NL-Imatrix-CreativeVoice.gguf \
  -ngl 999 \
  -fa on \
  -sm none \
  -ctk f32 \
  -ctv f32 \
  --kv-offload \
  -c 120000 \
  -np 1 \
  -b 512 \
  -ub 128 \
  --jinja \
  --chat-template-file gemma4_deckard_no_think.jinja \
  --reasoning off \
  --reasoning-format deepseek \
  --temp 0.9 \
  --top-p 0.92 \
  --min-p 0.03 \
  --repeat-penalty 1.15 \
  --repeat-last-n 256 \
  --presence-penalty 0.10 \
  --frequency-penalty 0.05 \
  --dry-multiplier 0.60
```

The included no-thinking template is optional, but it is the one we used for the creative deployment. The upstream model is a thinking model, and in our creative tests the no-thinking template reduced scaffold-style answers.

## Quantization Details

The Q8 override rule was:

```text
^blk\.(3[6-9]|4[0-1])\.attn_(q|k|v|output)\.weight=q8_0
```

That means 24 attention tensors were manually promoted from `IQ4_NL` to `Q8_0`:

- `blk.36..41.attn_q.weight`
- `blk.36..41.attn_k.weight`
- `blk.36..41.attn_v.weight`
- `blk.36..41.attn_output.weight`

Quantizer log summary:

```text
model size = 14340.66 MiB, 16.00 BPW
quant size = 4980.91 MiB, 5.56 BPW
```

Local file sizes from the experiment:

| Variant | Size |
|---|---:|
| IQ4_NL imatrix baseline | 4,427,727,872 bytes |
| IQ4_NL llama.cpp b8763 baseline | 5,193,958,400 bytes |
| IQ4_NL + attnQ8 last6 | 5,238,686,720 bytes |
| IQ4_NL + attnQ8 last12 | 5,283,415,040 bytes |
| Q6_K reference | 6,184,790,688 bytes |

The `last6` mixed quant is only about 45 MB larger than the comparable b8763 IQ4_NL build.

## Runtime Tests

These are direct runtime probes for this exact released GGUF using standard `llama.cpp`. No custom fork was used.

| GPU | Runtime | Context / KV | Prompt workload | Decode workload | Prompt tok/s | Decode tok/s | VRAM note |
|---|---|---|---:|---:|---:|---:|---|
| RTX 5060 Ti | `llama.cpp` b8763 server | `-c 120000`, f32 K/V, all layers offloaded | 1352 tokens | 256 tokens | 2816.56 | 73.54 | about 7796 MiB observed |
| RTX 5090 | `llama.cpp` b8763 server | `-c 120000`, f32 K/V, all layers offloaded | 1122 tokens | 40 tokens | 4251.93 | 138.82 | 7159 MiB llama.cpp self allocation |

5060 Ti log summary:

```text
CUDA0 model buffer size = 2775.92 MiB
non-SWA KV buffer size = 3752.00 MiB
SWA KV buffer size = 60.00 MiB
compute buffer size = 572.00 MiB
```

5090 log summary:

```text
CUDA0 self allocation = 7159 MiB
model buffer = 2775 MiB
context buffer = 3812 MiB
compute buffer = 572 MiB
```

## Intended Use

This quant was tuned for English creative writing, expressive prose, conceptual writing, emotional/voice-adjacent discussion, and general reasoning. It was not built as a coding specialist.

Because the source model is an uncensored/aliterated model, apply your own safety, policy, and deployment filters where needed.

## Lineage

1. Source model: `DavidAU/gemma-4-E4B-it-The-DECKARD-Claude-Opus-Expresso-Universe-HERETIC-UNCENSORED-Thinking`
2. Converted to BF16 GGUF locally.
3. Built an English creative/voice/logic imatrix calibration set.
4. Quantized with `IQ4_NL`.
5. Re-quantized selected late attention tensors in blocks `36..41` to `Q8_0`.
6. Runtime-tested the released GGUF on RTX 5060 Ti and RTX 5090 with standard `llama.cpp`.

## Limitations

- This was an unplanned exploratory release, not a broad benchmarked model release.
- Public runtime numbers are limited to the two direct tests above.
- The no-thinking template is recommended for creative use because it reduces visible planning/scaffold output.
- This is a GGUF release for `llama.cpp`; it is not a safetensors checkpoint.