--- license: apache-2.0 language: - en library_name: llama.cpp base_model: - DavidAU/gemma-4-E4B-it-The-DECKARD-Claude-Opus-Expresso-Universe-HERETIC-UNCENSORED-Thinking tags: - gguf - llama.cpp - gemma4 - gemma - deckard - iq4_nl - q8_0 - imatrix - creative-writing - storymaker - roleplay - voice - uncensored - english --- # Gemma-4-E4B-Deckard-AttnQ8-IQ4_NL-storyMaker This is a normal `llama.cpp` GGUF quant of DavidAU's Deckard Gemma model: The goal was a small, practical creative/voice/writing model that keeps more coherence than a plain Q4-style quant while still fitting comfortably on modest GPUs. The `storyMaker` name reflects the actual niche where it stood out: expressive writing, roleplay, voice-adjacent prose, and creative instruction following. This was an unplanned release. The author was testing models for creative writing use, found a quantized GGUF of this model that showed severe looping and quality degradation, and built this version to check whether the issue came from the core finetune or from the quantization. The result was unexpectedly useful: creative, able to reason, good at following instructions, strong for roleplay, and expressive in a way that made it enjoyable to use. Because of that, it was uploaded for anyone else interested in this niche. Because this was exploratory rather than a planned release, it has not gone through the author's usual comprehensive release process. A gender-affirming finetune based on the DavidAU model may come later for people interested in that direction. ## What Is Special This is not a plain one-shot Q4 quant. The released GGUF is: - base quant: `IQ4_NL` - calibration: English creative writing, general logic, emotional understanding, and voice/vocal-concept prompts - imatrix: 1000 calibration chunks at ctx512, 342 importance-matrix entries - protected tensors: attention `q`, `k`, `v`, and output projections in blocks `36..41` - protected quant type: `Q8_0` In other words, most of the model stays Q4-sized, while the final six attention blocks are kept at higher precision. The intent is to spend a small amount of extra size on the late attention path, where coherence, style continuity, and repetition stability are more likely to matter for writing-style use. ## Files Main model: ```text Gemma-4-E4B-Deckard-AttnQ8-Last6-IQ4_NL-Imatrix-CreativeVoice.gguf ``` Optional template used in our deployment: ```text gemma4_deckard_no_think.jinja ``` The GGUF works with standard `llama.cpp`. No custom fork is required. ## Suggested llama.cpp Command Example server command: ```bash llama-server \ -m Gemma-4-E4B-Deckard-AttnQ8-Last6-IQ4_NL-Imatrix-CreativeVoice.gguf \ -ngl 999 \ -fa on \ -sm none \ -ctk f32 \ -ctv f32 \ --kv-offload \ -c 120000 \ -np 1 \ -b 512 \ -ub 128 \ --jinja \ --chat-template-file gemma4_deckard_no_think.jinja \ --reasoning off \ --reasoning-format deepseek \ --temp 0.9 \ --top-p 0.92 \ --min-p 0.03 \ --repeat-penalty 1.15 \ --repeat-last-n 256 \ --presence-penalty 0.10 \ --frequency-penalty 0.05 \ --dry-multiplier 0.60 ``` The included no-thinking template is optional, but it is the one we used for the creative deployment. The upstream model is a thinking model, and in our creative tests the no-thinking template reduced scaffold-style answers. ## Quantization Details The Q8 override rule was: ```text ^blk\.(3[6-9]|4[0-1])\.attn_(q|k|v|output)\.weight=q8_0 ``` That means 24 attention tensors were manually promoted from `IQ4_NL` to `Q8_0`: - `blk.36..41.attn_q.weight` - `blk.36..41.attn_k.weight` - `blk.36..41.attn_v.weight` - `blk.36..41.attn_output.weight` Quantizer log summary: ```text model size = 14340.66 MiB, 16.00 BPW quant size = 4980.91 MiB, 5.56 BPW ``` Released GGUF size: ```text 5,238,686,720 bytes ``` ## Runtime Tests These are direct runtime probes for this exact released GGUF using standard `llama.cpp`. No custom fork was used. | GPU | Runtime | Context / KV | Prompt workload | Decode workload | Prompt tok/s | Decode tok/s | VRAM note | |---|---|---|---:|---:|---:|---:|---| | RTX 5060 Ti | `llama.cpp` b8763 server | `-c 120000`, f32 K/V, all layers offloaded | 1352 tokens | 256 tokens | 2816.56 | 73.54 | about 7796 MiB observed | | RTX 5090 | `llama.cpp` b8763 server | `-c 120000`, f32 K/V, all layers offloaded | 1122 tokens | 40 tokens | 4251.93 | 138.82 | 7159 MiB llama.cpp self allocation | 5060 Ti log summary: ```text CUDA0 model buffer size = 2775.92 MiB non-SWA KV buffer size = 3752.00 MiB SWA KV buffer size = 60.00 MiB compute buffer size = 572.00 MiB ``` 5090 log summary: ```text CUDA0 self allocation = 7159 MiB model buffer = 2775 MiB context buffer = 3812 MiB compute buffer = 572 MiB ``` ## Intended Use This quant was tuned for English creative writing, expressive prose, conceptual writing, emotional/voice-adjacent discussion, and general reasoning. It was not built as a coding specialist. Because the source model is an uncensored/aliterated model, apply your own safety, policy, and deployment filters where needed. ## Lineage 1. Source model: `DavidAU/gemma-4-E4B-it-The-DECKARD-Claude-Opus-Expresso-Universe-HERETIC-UNCENSORED-Thinking` 2. Converted to BF16 GGUF locally. 3. Built an English creative/voice/logic imatrix calibration set. 4. Quantized with `IQ4_NL`. 5. Re-quantized selected late attention tensors in blocks `36..41` to `Q8_0`. 6. Runtime-tested the released GGUF on RTX 5060 Ti and RTX 5090 with standard `llama.cpp`. ## Limitations - This was an unplanned exploratory release, not a broad benchmarked model release. - Public runtime numbers are limited to the two direct tests above. - The no-thinking template is recommended for creative use because it reduces visible planning/scaffold output. - This is a GGUF release for `llama.cpp`; it is not a safetensors checkpoint.