Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker

This is a normal llama.cpp GGUF quant of DavidAU's Deckard Gemma model:

https://huggingface.co/DavidAU/gemma-4-E4B-it-The-DECKARD-Claude-Opus-Expresso-Universe-HERETIC-UNCENSORED-Thinking

The goal was a small, practical creative/voice/writing model that keeps more coherence than a plain Q4-style quant while still fitting comfortably on modest GPUs. The storyMaker name reflects the actual niche where it stood out: expressive writing, roleplay, voice-adjacent prose, and creative instruction following.

This was an unplanned release. The author was testing models for creative writing use, found a quantized GGUF of this model that showed severe looping and quality degradation, and built this version to check whether the issue came from the core finetune or from the quantization. The result was unexpectedly useful: creative, able to reason, good at following instructions, strong for roleplay, and expressive in a way that made it enjoyable to use. Because of that, it was uploaded for anyone else interested in this niche.

Because this was exploratory rather than a planned release, it has not gone through the author's usual comprehensive release process. A gender-affirming finetune based on the DavidAU model may come later for people interested in that direction.

What Is Special

This is not a plain one-shot Q4 quant.

The released GGUF is:

• base quant: IQ4_NL
• calibration: English creative writing, general logic, emotional understanding, and voice/vocal-concept prompts
• imatrix: 1000 calibration chunks at ctx512, 342 importance-matrix entries
• protected tensors: attention q, k, v, and output projections in blocks 36..41
• protected quant type: Q8_0

In other words, most of the model stays Q4-sized, while the final six attention blocks are kept at higher precision. The intent is to spend a small amount of extra size on the late attention path, where coherence, style continuity, and repetition stability are more likely to matter for writing-style use.

Files

Main model:

Gemma-4-E4B-Deckard-AttnQ8-Last6-IQ4_NL-Imatrix-CreativeVoice.gguf

Optional template used in our deployment:

gemma4_deckard_no_think.jinja

The GGUF works with standard llama.cpp. No custom fork is required.

Suggested llama.cpp Command

Example server command:

llama-server \
  -m Gemma-4-E4B-Deckard-AttnQ8-Last6-IQ4_NL-Imatrix-CreativeVoice.gguf \
  -ngl 999 \
  -fa on \
  -sm none \
  -ctk f32 \
  -ctv f32 \
  --kv-offload \
  -c 120000 \
  -np 1 \
  -b 512 \
  -ub 128 \
  --jinja \
  --chat-template-file gemma4_deckard_no_think.jinja \
  --reasoning off \
  --reasoning-format deepseek \
  --temp 0.9 \
  --top-p 0.92 \
  --min-p 0.03 \
  --repeat-penalty 1.15 \
  --repeat-last-n 256 \
  --presence-penalty 0.10 \
  --frequency-penalty 0.05 \
  --dry-multiplier 0.60

The included no-thinking template is optional, but it is the one we used for the creative deployment. The upstream model is a thinking model, and in our creative tests the no-thinking template reduced scaffold-style answers.

Quantization Details

The Q8 override rule was:

^blk\.(3[6-9]|4[0-1])\.attn_(q|k|v|output)\.weight=q8_0

That means 24 attention tensors were manually promoted from IQ4_NL to Q8_0:

• blk.36..41.attn_q.weight
• blk.36..41.attn_k.weight
• blk.36..41.attn_v.weight
• blk.36..41.attn_output.weight

Quantizer log summary:

model size = 14340.66 MiB, 16.00 BPW
quant size = 4980.91 MiB, 5.56 BPW

Released GGUF size:

5,238,686,720 bytes

Runtime Tests

These are direct runtime probes for this exact released GGUF using standard llama.cpp. No custom fork was used.

GPU	Runtime	Context / KV	Prompt workload	Decode workload	Prompt tok/s	Decode tok/s	VRAM note
RTX 5060 Ti	llama.cpp b8763 server	-c 120000, f32 K/V, all layers offloaded	1352 tokens	256 tokens	2816.56	73.54	about 7796 MiB observed
RTX 5090	llama.cpp b8763 server	-c 120000, f32 K/V, all layers offloaded	1122 tokens	40 tokens	4251.93	138.82	7159 MiB llama.cpp self allocation

5060 Ti log summary:

CUDA0 model buffer size = 2775.92 MiB
non-SWA KV buffer size = 3752.00 MiB
SWA KV buffer size = 60.00 MiB
compute buffer size = 572.00 MiB

5090 log summary:

CUDA0 self allocation = 7159 MiB
model buffer = 2775 MiB
context buffer = 3812 MiB
compute buffer = 572 MiB

Intended Use

This quant was tuned for English creative writing, expressive prose, conceptual writing, emotional/voice-adjacent discussion, and general reasoning. It was not built as a coding specialist.

Because the source model is an uncensored/aliterated model, apply your own safety, policy, and deployment filters where needed.

Lineage

1. Source model: DavidAU/gemma-4-E4B-it-The-DECKARD-Claude-Opus-Expresso-Universe-HERETIC-UNCENSORED-Thinking
2. Converted to BF16 GGUF locally.
3. Built an English creative/voice/logic imatrix calibration set.
4. Quantized with IQ4_NL.
5. Re-quantized selected late attention tensors in blocks 36..41 to Q8_0.
6. Runtime-tested the released GGUF on RTX 5060 Ti and RTX 5090 with standard llama.cpp.

Limitations

• This was an unplanned exploratory release, not a broad benchmarked model release.
• Public runtime numbers are limited to the two direct tests above.
• The no-thinking template is recommended for creative use because it reduces visible planning/scaffold output.
• This is a GGUF release for llama.cpp; it is not a safetensors checkpoint.