--- tags: - vllm - llm-compressor - compressed-tensors - nvfp4a16 library_name: transformers license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license pipeline_tag: image-text-to-text base_model: coder3101/gemma-4-E4B-it-heretic provider: xdavxd name: xdavxd/gemma-4-E4B-it-heretic-NVFP4A16 description: NVFP4 variant of gemma-4-E4B-it-heretic. readme: https://huggingface.co/xdavxd/gemma-4-E4B-it-heretic-NVFP4A16/edit/main/README.md tool_calling_supported: true required_cli_args: - '--reasoning-parser gemma4' - '--enable-prefix-caching' default-chat-template-kwargs: '{"enable_thinking": true}' chat_template_file_name: None chat_template_path: None tool_call_parser: gemma4 validated_tasks: - tool-calling tasks: - text-to-text - text-generation - tool-calling --- # gemma-4-E4B-it-heretic-NVFP4A16 ## Model Overview - **Model Architecture:** Gemma 4 - **Input:** Text / Image / Audio - **Output:** Text - **Model Optimizations:** - **Weight quantization:** FP4 - **Activation quantization:** None (16-bit) - **Release Date:** 2026-07-03 - **Version:** 1.0 - **Quantized by:** xdavxd - **Base Model:** [coder3101/gemma-4-E4B-it-heretic](https://huggingface.co/coder3101/gemma-4-E4B-it-heretic) - **Original Model:** [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) This model is a quantized version of [coder3101/gemma-4-E4B-it-heretic](https://huggingface.co/coder3101/gemma-4-E4B-it-heretic). It was evaluated on several tasks to assess its quality in comparison to the original model. ### Model Optimizations This model was obtained by quantizing the weights and activations of [coder3101/gemma-4-E4B-it-heretic](https://huggingface.co/coder3101/gemma-4-E4B-it-heretic) to FP4 data type using the NVFP4 format, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 35%. Weights are quantized with FP4 (group_size=16) using local per-group scaling. Only the weights of the linear operators within transformer blocks are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor). Vision, audio, embedding, and output head layers are kept in their original precision. ## Deployment ### Use with vLLM This model can be deployed using [vLLM](https://docs.vllm.ai/en/latest/). For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the [Gemma 4 vLLM usage guide](https://recipes.vllm.ai/Google/gemma-4-E4B-it). 1. Start the vLLM server: ``` vllm serve xdavxd/gemma-4-E4B-it-heretic-NVFP4A16 \ --max-model-len 32768 \ --gpu-memory-utilization 0.50 ``` To enable thinking/reasoning and tool calling: ``` vllm serve xdavxd/gemma-4-E4B-it-heretic-NVFP4A16 \ --max-model-len 32768 \ --gpu-memory-utilization 0.50 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ --async-scheduling ``` > **Tip:** For text-only workloads, pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip vision encoder memory allocation and free up GPU memory for a longer context window. 2. Send requests to the server: ```python from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://:8001/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) model = "xdavxd/gemma-4-E4B-it-heretic-NVFP4A16" messages = [ {"role": "user", "content": "Explain quantum mechanics clearly and concisely."}, ] outputs = client.chat.completions.create( model=model, messages=messages, ) generated_text = outputs.choices[0].message.content print(generated_text) ``` ## Creation This model was created by applying NVFP4 quantization with [LLM Compressor](https://github.com/vllm-project/llm-compressor), as presented in the code snippet below.
```python from transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier MODEL_ID = "./gemma-4-E4B-it-heretic" SAVE_DIR = "gemma-4-E4B-it-heretic-NVFP4A16" model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype="auto") tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) recipe = QuantizationModifier( targets="Linear", scheme="NVFP4A16", ignore=["lm_head", "re:.*embed.*", "re:.*vision_tower.*", "re:.*audio_tower.*", "re:.*per_layer.*"], ) oneshot( model=model, tokenizer=tokenizer, recipe=recipe, ) model.save_pretrained(SAVE_DIR, save_compressed=True) tokenizer.save_pretrained(SAVE_DIR) ```
## Evaluation This model was evaluated on GSM8K Platinum, IFEval, and MATH-500, using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and [lighteval](https://github.com/huggingface/lighteval), served with [vLLM](https://docs.vllm.ai/en/latest/) (OpenAI-compatible API). All evaluations were performed with **thinking enabled**. ### Performance (as reported by coder3101) | Metric | This model | Original model ([google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it)) | | :----- | :--------: | :---------------------------: | | **KL divergence** | 0.0058 | 0 *(by definition)* | | **Refusals** | 3/100 | 99/100 | ### Accuracy
Category Benchmark google/gemma-4-E4B-it xdavxd/gemma-4-E4B-it-heretic-NVFP4A16 Recovery
Instruction Following IFEval (0-shot, prompt-level strict) - - -
IFEval (0-shot, inst-level strict) - - -
Reasoning GSM8K Platinum (flexible-extract) - - -
GSM8K Platinum (strict-match) - - -
MATH-500 (0-shot, pass@1) - - -
### Reproduction The results were obtained using the following commands:
Each benchmark was run 3 times with different judgmental seeds (1234, 2345, 3456) and the scores were averaged; AIME 2025 used 8 seeds. Ran using a docker image, you will need to tweak these for your own environment. Kept parameters similar to the RedHatAI runs. **vLLM server (all benchmarks):** ``` sudo docker run --rm -it \ --gpus all \ --ipc=host \ --network host \ -v ~/quant-workspace/gemma-4-E4B-it-heretic-NVFP4A16:/models/heretic:ro \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e VLLM_USE_V2_MODEL_RUNNER=1 \ ghcr.io/timothystewart6/vllm-gb10:latest \ vllm serve /models/heretic \ --host 0.0.0.0 --port 8001 \ --served-model-name heretic \ --max-model-len 69632 \ --gpu-memory-utilization 0.50 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --limit-mm-per-prompt '{"image":0,"audio":0}' \ --kv-cache-dtype fp8 \ --max-num-seqs 32 \ --async-scheduling ``` #### GSM8K Platinum (lm-eval, 0-shot, 3 repetitions) ``` for SEED in 1234 2345 3456; do lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=heretic,max_length=36096,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \ --num_fewshot 0 \ --apply_chat_template \ --output_path "results_gsm8k_seed${SEED}.json" \ --seed $SEED \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=${SEED}" echo "Seed $SEED complete" done ``` #### IFEval (lm-eval, 0-shot, 3 repetitions) ``` for SEED in 1234 2345 3456; do lm_eval --model local-chat-completions \ --tasks ifeval \ --model_args "model=heretic,max_length=36096,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \ --num_fewshot 0 \ --apply_chat_template \ --output_path "results_ifeval_seed${SEED}.json" \ --seed $SEED \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=${SEED}" echo "IFEval seed $SEED complete" done ``` #### MATH-500, (lighteval, 3 repetitions) **litellm_config.yaml:** ```yaml model_parameters: provider: hosted_vllm model_name: hosted_vllm/heretic base_url: http://0.0.0.0:8001/v1 api_key: '' timeout: 3600 concurrent_requests: 32 generation_parameters: temperature: 1.0 max_new_tokens: 65536 top_p: 0.95 top_k: 64 seed: 1234 ``` ``` # MATH-500 (3 seeds) for SEED in 1234 2345 3456; do sed -i "s/seed: .*/seed: $SEED/" litellm_config.yaml lighteval endpoint litellm litellm_config.yaml 'math_500|0' \ --output-dir "results_math500_seed${SEED}/" --save-details echo "MATH-500 seed $SEED complete" done ```