Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1,11 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tar.gz filter=lfs diff=lfs merge=lfs -text

Gemma-4-31B-NVFP4.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce8e40ba5ce80b3925a7c1e2dbf7177f2fc328284f0f66e2eb81da19a3be07aa
+size 22132161664

README.md

@@ -1,3 +1,387 @@
 ---
 license: apache-2.0
+base_model:
+  - RedHatAI/gemma-4-31B-it-NVFP4
+  - google/gemma-4-31B-it
+base_model_relation: quantized
+library_name: llama.cpp
+tags:
+  - gguf
+  - llama.cpp
+  - gemma4
+  - gemma
+  - dense
+  - nvfp4
+  - fp4
+  - llm-compressor
+  - compressed-tensors
+  - mtp
+  - speculative-decoding
+  - assistant-model
+pipeline_tag: image-text-to-text
 ---
+
+# gemma-4-31B-it-NVFP4-GGUF
+
+GGUF conversion of **RedHatAI/gemma-4-31B-it-NVFP4** for use with **llama.cpp**.
+
+> **This is an NVFP4-quantized standalone Gemma 4 target model.**
+>
+> It can run by itself. The speculative-decoding benchmark below pairs it with a compatible **Gemma 4 31B Assistant/MTP draft model**.
+
+## Model Details
+
+* **Source model:** `RedHatAI/gemma-4-31B-it-NVFP4`
+* **Base model:** `google/gemma-4-31B-it`
+* **Format:** GGUF
+* **Quantization:** NVIDIA NVFP4 / FP4
+* **Quantization framework:** Red Hat AI LLM Compressor / compressed-tensors
+* **Architecture:** Gemma 4 31B dense multimodal instruction model
+* **Input:** Text and image
+* **Output:** Text
+* **Purpose:** Local inference and Assistant/MTP speculative decoding with llama.cpp
+
+The upstream checkpoint quantizes the weights and activations of linear operators inside the transformer blocks to NVFP4. The vision tower, embedding layers, and output head remain at their original precision.
+
+The upstream model card specifies:
+
+* FP4 weight quantization with `group_size = 16`
+* FP4 activation quantization with local per-group scaling
+* Approximately 75% lower disk and GPU-memory requirements than the unquantized 16-bit checkpoint
+* Release date: 2026-04-04
+* Model version: 1.0
+
+This repository contains the complete target model. The compatible Assistant/MTP draft model used for speculative decoding is separate.
+
+## Compatibility
+
+A recent version of **llama.cpp** with Gemma 4, NVFP4, multimodal, and Assistant/MTP support is required.
+
+Tested with:
+
+* Windows
+* NVIDIA GeForce RTX 5070 Ti 16 GB
+* NVIDIA GeForce RTX 5060 Ti 16 GB
+* llama.cpp CUDA backend
+* A compatible Gemma 4 31B Assistant GGUF draft model
+* `--spec-type draft-mtp`
+
+Older llama.cpp builds may fail to recognize the `nvfp4` tensor type, load the Assistant architecture, or initialize the target and draft KV caches correctly.
+
+Performance varies with the llama.cpp build, target and draft quantizations, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.
+
+> The benchmark below is text-only. Image input requires a compatible multimodal projector and llama.cpp multimodal support.
+
+## Usage
+
+Replace the Assistant filename below with your compatible Gemma 4 31B Assistant GGUF.
+
+### llama-server
+
+```bash
+llama-server \
+  -m gemma-4-31B-it-NVFP4.gguf \
+  -md gemma-4-31B-it-assistant.gguf \
+  --spec-type draft-mtp \
+  --spec-draft-n-max 5 \
+  --flash-attn on \
+  -ngl 99
+```
+
+### llama-cli
+
+```bash
+llama-cli \
+  -m gemma-4-31B-it-NVFP4.gguf \
+  -md gemma-4-31B-it-assistant.gguf \
+  --spec-type draft-mtp \
+  --spec-draft-n-max 5 \
+  --flash-attn on \
+  -ngl 99
+```
+
+### Multi-GPU example
+
+```bash
+llama-server \
+  -m gemma-4-31B-it-NVFP4.gguf \
+  -md gemma-4-31B-it-assistant.gguf \
+  --spec-type draft-mtp \
+  --spec-draft-n-max 5 \
+  --split-mode layer \
+  --tensor-split 1,1 \
+  --flash-attn on \
+  -ngl 99
+```
+
+The target and Assistant models must use compatible tokenizer, vocabulary, and Gemma 4 architecture assumptions.
+
+An even tensor split is only a starting point. The best split depends on GPU speed, available VRAM, PCIe topology, context length, and placement of the target model, Assistant model, and KV caches.
+
+## Suggested Settings
+
+Based on the mixed-task benchmark below:
+
+* **`n_max = 5`** gave the highest aggregate throughput of the tested settings.
+* **`n_max = 2`** gave the highest aggregate acceptance rate and remained the best choice for creative writing and other high-entropy output.
+* **`n_max = 3`** was a strong balanced setting with substantially less draft work than `n_max = 5`.
+* **`n_max = 4`** improved most structured tasks and remained a reasonable general-purpose compromise.
+* **`n_max = 5`** was best for code, JSON, repeated patterns, mathematical reasoning, and long reasoning.
+* Creative writing degraded steadily as draft depth increased.
+
+Recommended maximum-throughput setting:
+
+```text
+--spec-draft-n-max 5
+```
+
+Recommended balanced setting:
+
+```text
+--spec-draft-n-max 3
+```
+
+For role-play, creative writing, open-ended conversation, or other high-entropy generation:
+
+```text
+--spec-draft-n-max 2
+```
+
+## Conversion
+
+Converted from the original Hugging Face compressed-tensors checkpoint using a recent `convert_hf_to_gguf.py` from llama.cpp:
+
+```bash
+python convert_hf_to_gguf.py \
+  /path/to/gemma-4-31B-it-NVFP4 \
+  --outfile gemma-4-31B-it-NVFP4.gguf \
+  --outtype auto \
+  --verbose
+```
+
+Do not pass a conventional GGUF quantization type such as `q4_k_m` when the goal is to preserve the original NVFP4 tensors.
+
+The conversion repackages the compressed-tensors NVFP4 checkpoint into a GGUF-compatible representation rather than requantizing it into a K-quant.
+
+The compatible Assistant model must be converted separately. No target-model weights were retrained by this conversion.
+
+## File Size
+
+The converted GGUF may differ in size from the original Hugging Face checkpoint.
+
+The conversion does not copy the original safetensors byte-for-byte. NVFP4 tensors are repacked into GGML-compatible layouts, and GGUF stores the scale metadata required by llama.cpp. The vision tower, embeddings, output head, and other tensors that remain at higher precision also contribute to the file size.
+
+A larger GGUF does not mean the model was converted to FP16 or to a conventional Q4 format.
+
+Windows reports file sizes using binary units even though File Explorer labels them as `GB`. The same file can therefore appear smaller in Windows than on Hugging Face.
+
+File size is not the same as runtime VRAM usage. Additional memory is needed for:
+
+* the Assistant draft model
+* target and draft KV caches
+* CUDA context
+* compute buffers
+* graph workspace
+* multimodal projector, when used
+
+## Verifying the GGUF
+
+The converted target can be checked with `llama-gguf`:
+
+```powershell
+.\llama-gguf.exe E:\HF_MODELS\gemma-4-31B-it-NVFP4.gguf r
+```
+
+A successful conversion should show NVFP4 transformer tensors and their associated scale tensors. The vision tower, embeddings, and output head may remain in higher-precision formats.
+
+Example:
+
+```text
+tensor: name = blk.0.ffn_down.weight,       type = nvfp4
+tensor: name = blk.0.ffn_down.scale,        type = f32
+tensor: name = blk.0.ffn_down.input_scale,  type = f32
+```
+
+This confirms that the checkpoint was not converted into a conventional Q4 type and is not an empty metadata-only GGUF.
+
+## Benchmark
+
+Benchmarked using **mtp-bench** with a compatible Gemma 4 31B Assistant model and:
+
+```text
+--spec-type draft-mtp
+```
+
+### Aggregate Results — Assistant/MTP
+
+| `n_max` | Draft acceptance | Predicted tokens | Draft tokens | Accepted tokens | Wall time | Effective batch throughput |
+| ------: | ---------------: | ---------------: | -----------: | --------------: | --------: | -------------------------: |
+|       2 |        **78.2%** |            1,979 |        1,534 |           1,199 |   37.27 s |                  53.10 tok/s |
+|       3 |            71.6% |            1,979 |        1,874 |           1,341 |   32.96 s |                  60.04 tok/s |
+|       4 |            65.6% |            2,016 |        2,206 |           1,447 |   31.35 s |                  64.31 tok/s |
+|       5 |            60.2% |            2,016 |        2,495 |           1,501 | **30.18 s** |              **66.80 tok/s** |
+
+Effective batch throughput is calculated as total predicted tokens divided by total wall time.
+
+Per-request `tok/s` can be higher because aggregate wall time may include prompt processing, request transitions, and benchmark overhead.
+
+The predicted-token totals differ slightly because some generated responses terminated at different lengths. The `n_max = 4` and `n_max = 5` runs both generated 2,016 tokens and can be compared directly.
+
+These tests do not include a non-speculative baseline and therefore should not be interpreted as direct speedup ratios over standard decoding.
+
+## Detailed Assistant Results
+
+### `n_max = 2`
+
+```text
+code_python        pred= 192 draft= 138 acc= 121 rate=0.877 tok/s=61.6
+code_cpp           pred= 192 draft= 147 acc= 117 rate=0.796 tok/s=58.4
+explain_concept    pred= 192 draft= 170 acc= 105 rate=0.618 tok/s=50.2
+summarize          pred=  48 draft=  38 acc=  30 rate=0.789 tok/s=56.5
+qa_factual         pred= 172 draft= 146 acc=  99 rate=0.678 tok/s=53.2
+translation        pred=  16 draft=  12 acc=  10 rate=0.833 tok/s=57.5
+creative_short     pred=  34 draft=  38 acc=  14 rate=0.368 tok/s=39.7
+stepwise_math      pred= 192 draft= 138 acc= 122 rate=0.884 tok/s=62.8
+json_output        pred= 192 draft= 129 acc= 126 rate=0.977 tok/s=65.7
+long_reasoning     pred= 192 draft= 147 acc= 117 rate=0.796 tok/s=58.3
+repeat_pattern     pred= 192 draft= 127 acc= 127 rate=1.000 tok/s=67.7
+code_completion    pred= 173 draft= 126 acc= 110 rate=0.873 tok/s=61.7
+long_code_review   pred= 192 draft= 178 acc= 101 rate=0.567 tok/s=46.4
+
+Aggregate:
+  requests:            13
+  predicted tokens:    1979
+  draft tokens:        1534
+  accepted tokens:     1199
+  acceptance rate:     0.7816
+  total wall time:     37.27 s
+```
+
+### `n_max = 3`
+
+```text
+code_python        pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=71.7
+code_cpp           pred= 192 draft= 178 acc= 131 rate=0.736 tok/s=67.3
+explain_concept    pred= 192 draft= 225 acc= 116 rate=0.516 tok/s=53.7
+summarize          pred=  48 draft=  48 acc=  33 rate=0.688 tok/s=62.6
+qa_factual         pred= 172 draft= 171 acc= 114 rate=0.667 tok/s=63.3
+translation        pred=  16 draft=  15 acc=  11 rate=0.733 tok/s=64.1
+creative_short     pred=  34 draft=  54 acc=  16 rate=0.296 tok/s=39.7
+stepwise_math      pred= 192 draft= 168 acc= 134 rate=0.798 tok/s=70.5
+json_output        pred= 192 draft= 144 acc= 142 rate=0.986 tok/s=80.7
+long_reasoning     pred= 192 draft= 177 acc= 131 rate=0.740 tok/s=65.8
+repeat_pattern     pred= 192 draft= 143 acc= 143 rate=1.000 tok/s=83.8
+code_completion    pred= 173 draft= 147 acc= 124 rate=0.844 tok/s=73.5
+long_code_review   pred= 192 draft= 236 acc= 111 rate=0.470 tok/s=48.6
+
+Aggregate:
+  requests:            13
+  predicted tokens:    1979
+  draft tokens:        1874
+  accepted tokens:     1341
+  acceptance rate:     0.7156
+  total wall time:     32.96 s
+```
+
+### `n_max = 4`
+
+```text
+code_python        pred= 192 draft= 199 acc= 141 rate=0.709 tok/s=77.7
+code_cpp           pred= 192 draft= 201 acc= 140 rate=0.697 tok/s=77.3
+explain_concept    pred= 192 draft= 273 acc= 122 rate=0.447 tok/s=57.5
+summarize          pred=  48 draft=  56 acc=  33 rate=0.589 tok/s=70.0
+qa_factual         pred= 192 draft= 229 acc= 133 rate=0.581 tok/s=66.8
+translation        pred=  16 draft=  16 acc=  11 rate=0.688 tok/s=71.3
+creative_short     pred=  32 draft=  64 acc=  16 rate=0.250 tok/s=39.0
+stepwise_math      pred= 192 draft= 189 acc= 143 rate=0.757 tok/s=77.1
+json_output        pred= 192 draft= 157 acc= 151 rate=0.962 tok/s=91.2
+long_reasoning     pred= 192 draft= 213 acc= 137 rate=0.643 tok/s=67.9
+repeat_pattern     pred= 192 draft= 155 acc= 152 rate=0.981 tok/s=94.2
+code_completion    pred= 192 draft= 168 acc= 149 rate=0.887 tok/s=86.5
+long_code_review   pred= 192 draft= 286 acc= 119 rate=0.416 tok/s=49.0
+
+Aggregate:
+  requests:            13
+  predicted tokens:    2016
+  draft tokens:        2206
+  accepted tokens:     1447
+  acceptance rate:     0.6559
+  total wall time:     31.35 s
+```
+
+### `n_max = 5`
+
+```text
+code_python        pred= 192 draft= 219 acc= 147 rate=0.671 tok/s=83.2
+code_cpp           pred= 192 draft= 235 acc= 144 rate=0.613 tok/s=78.4
+explain_concept    pred= 192 draft= 331 acc= 124 rate=0.375 tok/s=53.2
+summarize          pred=  48 draft=  60 acc=  37 rate=0.617 tok/s=76.0
+qa_factual         pred= 192 draft= 270 acc= 136 rate=0.504 tok/s=63.2
+translation        pred=  16 draft=  20 acc=  12 rate=0.600 tok/s=66.7
+creative_short     pred=  32 draft=  80 acc=  16 rate=0.200 tok/s=35.8
+stepwise_math      pred= 192 draft= 200 acc= 150 rate=0.750 tok/s=83.1
+json_output        pred= 192 draft= 169 acc= 157 rate=0.929 tok/s=100.6
+long_reasoning     pred= 192 draft= 238 acc= 142 rate=0.597 tok/s=74.8
+repeat_pattern     pred= 192 draft= 165 acc= 158 rate=0.958 tok/s=104.6
+code_completion    pred= 192 draft= 183 acc= 154 rate=0.842 tok/s=97.9
+long_code_review   pred= 192 draft= 325 acc= 124 rate=0.382 tok/s=52.8
+
+Aggregate:
+  requests:            13
+  predicted tokens:    2016
+  draft tokens:        2495
+  accepted tokens:     1501
+  acceptance rate:     0.6016
+  total wall time:     30.18 s
+```
+
+## Observations
+
+* `n_max = 5` delivered the highest effective aggregate throughput at approximately **66.80 tok/s**.
+* Relative to `n_max = 2`, `n_max = 5` improved aggregate throughput by approximately **25.8%**.
+* Relative to the previously tested lower-quality Assistant pairing at `n_max = 2`, the final Assistant pairing and `n_max = 5` improved throughput from approximately **48.60 tok/s** to **66.80 tok/s**.
+* `n_max = 5` reached **104.6 tok/s** on repeated-pattern generation.
+* JSON output reached **100.6 tok/s**.
+* Code completion reached **97.9 tok/s**.
+* Code and stepwise mathematics exceeded **78 tok/s**.
+* Creative writing did not benefit from deeper drafting and declined from **39.7 tok/s** at `n_max = 2` to **35.8 tok/s** at `n_max = 5`.
+* Open-ended explanation, factual QA, and translation showed diminishing returns or regressions at `n_max = 5`.
+* Increasing from `n_max = 4` to `n_max = 5` improved aggregate throughput by approximately **3.9%**, while aggregate acceptance fell from **65.6%** to **60.2%**.
+
+The benchmark shows a clear task-dependent trade-off: structured and predictable output benefits from deeper drafting, while high-entropy natural-language generation generally favors a shorter draft.
+
+## Upstream Accuracy
+
+The upstream Red Hat AI model card reports strong recovery relative to the unquantized `google/gemma-4-31B-it`, including approximately:
+
+* 99.0–99.6% recovery on IFEval
+* 99.9% on GSM8K Platinum
+* 98.9% on MMLU-Pro
+* 95.2% on MATH-500
+* 98.7% on AIME 2025
+* 98.9% on GPQA Diamond
+* 98.9% on LiveCodeBench v6
+
+These are upstream vLLM evaluation results and are separate from the local llama.cpp throughput benchmark above.
+
+## Notes
+
+* This GGUF preserves the NVFP4 transformer tensors from Red Hat AI's compressed-tensors checkpoint; it is not equivalent to `Q4_K_M`, `Q4_K_S`, or an Unsloth dynamic quant.
+* Mixed tensor types are expected because the vision tower, embeddings, and output head are not quantized to NVFP4.
+* Runtime memory requirements depend on the full target model, the separate Assistant model, both KV caches, multimodal components, and compute buffers.
+* The upstream checkpoint targets vLLM. llama.cpp support is a separate community implementation and may behave differently.
+* Assistant/MTP speculative decoding accelerates token generation but does not provide the same benefit to prompt prefill.
+* Very short generations, especially the translation test in these runs, are sensitive to run-to-run variance.
+* Results are specific to the tested hardware, model files, llama.cpp build, prompts, sampling configuration, and context settings.
+
+## Credits
+
+* **Google DeepMind** — Gemma 4 and `google/gemma-4-31B-it`
+* **Red Hat AI** — NVFP4 quantization, LLM Compressor, and `RedHatAI/gemma-4-31B-it-NVFP4`
+* **ggml-org** — llama.cpp, GGUF, NVFP4 inference support, and Gemma 4 Assistant/MTP support
+
+## License
+
+The source checkpoint is distributed under the **Apache License 2.0**.
+
+Users should review the upstream `RedHatAI/gemma-4-31B-it-NVFP4` and `google/gemma-4-31B-it` model cards and applicable terms before redistribution or commercial use.