jackasda211233 commited on
Commit
2fa748f
·
verified ·
1 Parent(s): ead0a14

Revise release card benchmark notes

Browse files
Files changed (1) hide show
  1. README.md +20 -37
README.md CHANGED
@@ -28,6 +28,10 @@ This is a normal `llama.cpp` GGUF quant of DavidAU's Deckard Gemma model:
28
 
29
  The goal was a small, practical creative/voice/writing model that keeps more coherence than a plain Q4-style quant while still fitting comfortably on modest GPUs.
30
 
 
 
 
 
31
  ## What Is Special
32
 
33
  This is not a plain one-shot Q4 quant.
@@ -123,54 +127,33 @@ Local file sizes from the experiment:
123
  | IQ4_NL + attnQ8 last12 | 5,283,415,040 bytes |
124
  | Q6_K reference | 6,184,790,688 bytes |
125
 
126
- The `last6` mixed quant is only about 45 MB larger than the comparable b8763 IQ4_NL build, while testing better in our small creative/voice/coherence probe.
127
 
128
- ## Test Results
129
 
130
- These are local smoke/proxy tests, not a formal benchmark suite. The prompts targeted:
131
 
132
- - short creative scene writing
133
- - transfeminine voice-practice instruction quality
134
- - coherence/repetition stress with port-city micro-scenes
 
135
 
136
- Best mini-eval run:
137
 
138
  ```text
139
- docs/dynamic_q8_mini_20260429_112232_report.md
140
- ```
141
-
142
- Summary from that run:
143
-
144
- | Model | Proxy score | Coverage | Worst 4-gram repeat | Decode tok/s | Prompt tok/s |
145
- |---|---:|---:|---:|---:|---:|
146
- | attnQ8_last6 | 76.407 | 0.489 | 1 | 86.579 | 1354.519 |
147
- | Q6_K | 73.778 | 0.422 | 1 | 72.340 | 1480.222 |
148
- | Q8_0 | 73.278 | 0.411 | 1 | 61.660 | 1502.476 |
149
- | IQ4_NL b8763 | 70.185 | 0.356 | 1 | 87.449 | 1300.931 |
150
- | attnQ8_last12 | 68.426 | 0.300 | 1 | 85.448 | 1372.260 |
151
-
152
- Interpretation: `attnQ8_last6` was the best quality/size tradeoff in this test. The broader `attnQ8_last12` variant was worse, so simply adding more Q8 was not helpful.
153
-
154
- ## 5060 Ti Runtime Probe
155
-
156
- On one RTX 5060 Ti, using standard `llama.cpp` with f32 KV and `-c 120000`:
157
-
158
- ```text
159
- offloaded 43/43 layers to GPU
160
  CUDA0 model buffer size = 2775.92 MiB
161
  non-SWA KV buffer size = 3752.00 MiB
162
  SWA KV buffer size = 60.00 MiB
163
  compute buffer size = 572.00 MiB
164
- observed VRAM = about 7796 MiB
165
- prompt = 2816.56 tok/s on a 1352-token prompt
166
- decode = 73.54 tok/s for 256 generated tokens
167
  ```
168
 
169
- Small-context kernel probe on the same GPU:
170
 
171
  ```text
172
- prompt = 5558.54 tok/s
173
- decode = 105.80 tok/s
 
 
174
  ```
175
 
176
  ## Intended Use
@@ -186,11 +169,11 @@ Because the source model is an uncensored/aliterated model, apply your own safet
186
  3. Built an English creative/voice/logic imatrix calibration set.
187
  4. Quantized with `IQ4_NL`.
188
  5. Re-quantized selected late attention tensors in blocks `36..41` to `Q8_0`.
189
- 6. Tested against IQ4_NL, Q6_K, Q8_0, and a wider `attnQ8_last12` candidate.
190
 
191
  ## Limitations
192
 
193
- - The reported eval is a small automatic proxy eval plus user testing, not a broad benchmark.
 
194
  - The no-thinking template is recommended for creative use because it reduces visible planning/scaffold output.
195
  - This is a GGUF release for `llama.cpp`; it is not a safetensors checkpoint.
196
-
 
28
 
29
  The goal was a small, practical creative/voice/writing model that keeps more coherence than a plain Q4-style quant while still fitting comfortably on modest GPUs.
30
 
31
+ This was an unplanned release. The author was testing models for creative writing use, found a quantized GGUF of this model that showed severe looping and quality degradation, and built this version to check whether the issue came from the core finetune or from the quantization. The result was unexpectedly useful: creative, able to reason, good at following instructions, strong for roleplay, and expressive in a way that made it enjoyable to use. Because of that, it was uploaded for anyone else interested in this niche.
32
+
33
+ Because this was exploratory rather than a planned release, it has not gone through the author's usual comprehensive release process. A gender-affirming finetune based on the DavidAU model may come later for people interested in that direction.
34
+
35
  ## What Is Special
36
 
37
  This is not a plain one-shot Q4 quant.
 
127
  | IQ4_NL + attnQ8 last12 | 5,283,415,040 bytes |
128
  | Q6_K reference | 6,184,790,688 bytes |
129
 
130
+ The `last6` mixed quant is only about 45 MB larger than the comparable b8763 IQ4_NL build.
131
 
132
+ ## Runtime Tests
133
 
134
+ These are direct runtime probes for this exact released GGUF using standard `llama.cpp`. No custom fork was used.
135
 
136
+ | GPU | Runtime | Context / KV | Prompt workload | Decode workload | Prompt tok/s | Decode tok/s | VRAM note |
137
+ |---|---|---|---:|---:|---:|---:|---|
138
+ | RTX 5060 Ti | `llama.cpp` b8763 server | `-c 120000`, f32 K/V, all layers offloaded | 1352 tokens | 256 tokens | 2816.56 | 73.54 | about 7796 MiB observed |
139
+ | RTX 5090 | `llama.cpp` b8763 server | `-c 120000`, f32 K/V, all layers offloaded | 1122 tokens | 40 tokens | 4251.93 | 138.82 | 7159 MiB llama.cpp self allocation |
140
 
141
+ 5060 Ti log summary:
142
 
143
  ```text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
144
  CUDA0 model buffer size = 2775.92 MiB
145
  non-SWA KV buffer size = 3752.00 MiB
146
  SWA KV buffer size = 60.00 MiB
147
  compute buffer size = 572.00 MiB
 
 
 
148
  ```
149
 
150
+ 5090 log summary:
151
 
152
  ```text
153
+ CUDA0 self allocation = 7159 MiB
154
+ model buffer = 2775 MiB
155
+ context buffer = 3812 MiB
156
+ compute buffer = 572 MiB
157
  ```
158
 
159
  ## Intended Use
 
169
  3. Built an English creative/voice/logic imatrix calibration set.
170
  4. Quantized with `IQ4_NL`.
171
  5. Re-quantized selected late attention tensors in blocks `36..41` to `Q8_0`.
172
+ 6. Runtime-tested the released GGUF on RTX 5060 Ti and RTX 5090 with standard `llama.cpp`.
173
 
174
  ## Limitations
175
 
176
+ - This was an unplanned exploratory release, not a broad benchmarked model release.
177
+ - Public runtime numbers are limited to the two direct tests above.
178
  - The no-thinking template is recommended for creative use because it reduces visible planning/scaffold output.
179
  - This is a GGUF release for `llama.cpp`; it is not a safetensors checkpoint.