pinklily69/DeepSeek-V4-Flash-IQ3_S-GGUF

🪷 DeepSeek V4-Flash — IQ3_S GGUF

the first community quant in the IQ3 range for V4-Flash 💗

Model Details

Model Description

A community-grade IQ3_S quantization of DeepSeek V4-Flash, made on a sovereign workstation with full transparency about what's in the recipe.

• Quantized by: @thepinklily69 🌸
• Model type: Mixture-of-Experts (256 experts, 6 active per token)
• Language(s): English (primary), Chinese, multilingual
• License: MIT (inherited from base model)
• Quantized from: deepseek-ai/DeepSeek-V4-Flash

Model Sources

• Base model repository: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash
• llama.cpp fork used (V4 support): nisparks/llama.cpp

Uses

Direct Use

Local inference via llama.cpp, Ollama, LM Studio, or any GGUF-compatible runtime. Suitable for:

• Conversational AI
• Code generation and reasoning
• Creative writing
• Multilingual tasks
• Research and experimentation

Out-of-Scope Use

• Production deployments requiring strict quality guarantees (consider FP8/FP16 from official providers)
• Safety-critical applications without additional alignment
• Use cases where the imatrix coverage limitation (see below) would be problematic

Bias, Risks, and Limitations

Quantization Limitations (read this~)

This quant was guided by an importance matrix calibrated on a 320K-token English corpus over 200 chunks at ctx-size 2048.

Transparency Notice: V4-Flash's FP16 source is 568 GB, which doesn't fit on a single 96 GB GPU. So imatrix generation used llama.cpp's --cpu-moe flag, which routes expert computation through CPU RAM. As a consequence, the imatrix collector only captured the expert FFN tensors (129/1328 entries) — the attention and indexer tensors got quantized without imatrix guidance, falling back to safe defaults inside llama-quantize.

Why this is still useful:

• The expert FFN tensors are the vast majority of V4-Flash's parameters by weight
• IQ3_S degrades gracefully on missing imatrix entries (unlike IQ3_XXS which bails)
• The model generates coherent text at expected speeds

Inherited Biases

This quant inherits all biases, limitations, and behaviors of the base DeepSeek V4-Flash model. Please refer to the base model card for details.

Recommendations

Users should test this quant on their specific use cases before relying on it. The quantization process is lossy by design — for highest fidelity, use the official FP8 endpoints from DeepSeek or third-party providers like SiliconFlow.

How to Get Started with the Model

llama.cpp

# download
hf download pinklily69/DeepSeek-V4-Flash-IQ3_S-GGUF V4-Flash-IQ3_S.gguf

# run with CPU-MoE offload (recommended for GPUs <128GB VRAM)
llama-cli \
  -m V4-Flash-IQ3_S.gguf \
  -ngl 999 \
  --cpu-moe \
  -p "Write a short poem about spring." \
  -n 200

Hardware Tier Guide

GPU	Fits how
96 GB (RTX PRO 6000 Blackwell, H100)	comfortably, with room for context
48 GB (RTX 6000 Ada, A6000)	tight, full model on GPU
24 GB (3090 / 4090) + 128 GB RAM	with --cpu-moe, expect ~12-15 t/s
128 GB+ RAM, any GPU	full CPU offload, slower but functional

Training Details

Training Data

Not applicable — this is a quantization, not a fine-tune. Refer to the base model for original training data.

Quantization Procedure

Preprocessing

1. Safetensors → FP16 GGUF conversion using convert_hf_to_gguf.py --use-temp-file (streams to disk, enables conversion on 128 GB RAM systems)
2. Imatrix generation on FP16 GGUF with --cpu-moe (~9 days)
3. Quantization with llama-quantize ... IQ3_S using the imatrix (~55 minutes)

Quantization Hyperparameters

• Source precision: FP16
• Target quantization: IQ3_S (3.45 BPW)
• Imatrix corpus: 320K-token English calibration text
• Imatrix chunks: 200 at ctx-size 2048
• Imatrix coverage: 129/1328 tensors (expert FFN only — see Limitations)

Speeds, Sizes, Times

• Source file size: 568 GB (FP16 GGUF)
• Output file size: ~114 GB (IQ3_S)
• Bits per weight: 3.45 BPW
• Generation speed: 12.9 tokens/sec on RTX PRO 6000 Blackwell with --cpu-moe

Evaluation

Testing Data, Factors & Metrics

Status

No standardized benchmark comparison is provided in this initial release. This is intentional and transparent:

• DeepSeek has not published official FP16 benchmark numbers for V4-Flash; available scores are from third-party providers running FP8
• Comparing IQ3_S against FP8 third-party scores would be apples-to-oranges
• Running independent benchmarks would significantly delay an already-overdue community release

Users are encouraged to run their own evaluations on tasks relevant to their use case.

Verified Behavior

• ✅ Loads cleanly via llama.cpp (nisparks fork)
• ✅ Generates coherent text on diverse prompts
• ✅ Maintains expected generation speed (~12-15 t/s on Blackwell with CPU-MoE offload)

Technical Specifications

Model Architecture

• Type: Mixture-of-Experts transformer
• Total experts: 256
• Active experts per token: 6
• Shared experts: 1
• Layers: 43
• Context length: 1,048,576 (1M tokens)
• Embedding dimension: 4096
• Attention heads: 64
• KV heads: 1 (Multi-head Latent Attention)

Compute Infrastructure

Hardware

Quantization was performed on a custom workstation ("Petal"):

• CPU: AMD Threadripper 7960X (24 cores / 48 threads)
• RAM: 128 GB DDR5-5200 ECC
• GPU: NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM)
• Storage: Corsair MP600 PRO NH 8TB NVMe
• OS: Ubuntu 24.04 LTS

Software

• llama.cpp build: 9053 (commit 9d3640870), nisparks fork with V4 architecture support
• Quantization tool: llama-quantize
• Imatrix tool: llama-imatrix

Citation

If this quant helped you, cite the base model:

@misc{deepseek2026v4flash,
  title={DeepSeek-V4-Flash},
  author={DeepSeek-AI},
  year={2026},
  url={https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash}
}

Model Card Authors

@thepinklily69 🌸

Acknowledgments

• DeepSeek-AI for V4-Flash 💞
• ggml-org / llama.cpp for the quantization toolchain
• nisparks for the V4-capable fork
• The r/LocalLLaMA community for keeping local AI alive

---

made on Petal, with love~ 💗🪷✨